System

Abstract

A system is provided.SOLUTION: A system comprising: means for accepting voice input; means for converting the voice input to text data; means for analyzing the text data and extracting necessary information; means for automatically generating a document in accordance with a preset format based on the analyzed information; means for presenting the automatically generated document to a user for confirmation and correction; and means for storing and transmitting the final document after the confirmation and correction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In child consultation centers, nurseries, and other workplaces where interpersonal work is the primary focus, the task of preparing the necessary documents places a heavy administrative burden, which puts a strain on resources for practical work and emergency cases, as well as on ensuring sufficient rest time for staff. To solve this problem, there is a need to provide technology that can automatically generate the necessary documents based on voice input, thereby reducing the administrative burden. [Means for solving the problem]

[0005] In order to solve these problems, the present invention provides the following means.

[0006] A system is provided that includes means for accepting voice input, means for converting the voice input into text data, means for analyzing the text data and extracting necessary information, means for automatically generating a document in a preset format based on the analyzed information, means for presenting the automatically generated document to a user for confirmation and correction, and means for saving and transmitting the final document after completion of the confirmation and correction. This system can consistently execute the process from voice input to automatic document generation, thereby reducing administrative burdens.

[0007] "Voice input" refers to the act of inputting voice information spoken by a user into a terminal, and the system recognizes the voice and treats it as data for processing.

[0008] "Text data" is data in the form of a string of characters converted from voice input, and is used by the system for analysis and document generation.

[0009] "Analysis" is the act of processing text data to extract necessary information, and is carried out using natural language processing algorithms, etc.

[0010] A "format" is a pre-set template that gives a document a certain form or appearance, and is a framework in which data is arranged appropriately.

[0011] "Document generation" is the process of constructing a document based on the analyzed information in accordance with a pre-defined format.

[0012] "Presenting" is the act of displaying the generated document to the user so that the user can confirm and modify the contents.

[0013] "Verification and correction" refers to the act of the user checking the generated document and changing or supplementing the content as necessary.

[0014] "Storage" is the act of keeping completed documents in a digital format within the system.

[0015] "Transmission" means the act of electronically sending the generated or modified document to the relevant parties or departments. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for reducing the administrative burden at child consultation centers, nurseries, and other institutions that provide interpersonal services, and providing an environment in which they can concentrate on their work. A specific embodiment of this system is described below.

[0038] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[0039] The converted text data is then sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[0040] The generated document is sent from the server to the terminal and presented to the user. The user can check the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished checking and correcting, the final document is saved and sent to the relevant department as necessary.

[0041] Specific examples

[0042] Cases at child consultation centers

[0043] 1. Start voice input

[0044] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[0045] 2. Collection of audio data

[0046] The device collects audio that says, "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[0047] 3. Sending text data

[0048] The content converted into text data is sent to the server.

[0049] 4. Text Data Analysis

[0050] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[0051] 5. Automatic generation of required documents

[0052] The server embeds the information into the "interview report" template and generates the document.

[0053] 6. Check and correct documents

[0054] The user checks the generated interview report and makes corrections as necessary.

[0055] 7. Saving and sending documents

[0056] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[0057] Cases at nursery schools

[0058] 1. Start voice input

[0059] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0060] 2. Collection of audio data

[0061] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[0062] 3. Sending text data

[0063] The content converted into text data is sent to the server.

[0064] 4. Text Data Analysis

[0065] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[0066] 5. Automatic generation of required documents

[0067] The server embeds the information into a "daily report" template and generates the document.

[0068] 6. Check and correct documents

[0069] The user checks the generated daily report and makes corrections as necessary.

[0070] 7. Saving and sending documents

[0071] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[0072] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user starts a dedicated application on the device (PC or smartphone) and selects voice input mode. When voice input mode is activated, the device prepares for voice recognition. A message is displayed to the user indicating that the device is ready.

[0076] Step 2:

[0077] The user initiates voice input, for example, saying, "Today's meeting discussed my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[0078] Step 3:

[0079] The collected voice data is converted into text data using voice recognition software on the device. The converted data might look something like, "During today's interview, we discussed my child's school life and home environment."

[0080] Step 4:

[0081] Text data is sent from the device to the server, which then begins analyzing it using natural language processing (NLP) algorithms.

[0082] Step 5:

[0083] The server analyzes the text data and extracts the necessary information, for example identifying key keywords and phrases in the text, such as "school life" and "home environment," and classifying each as an appropriate document item.

[0084] Step 6:

[0085] Based on the extracted information, the server fills in pre-set templates and automatically generates the necessary documents. For example, it fills in the "school life" and "home environment" sections in an "interview report" template.

[0086] Step 7:

[0087] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for checking and modifying the contents.

[0088] Step 8:

[0089] The user checks the document presented on the terminal and makes corrections as necessary, such as entering additional information in the "home environment" section.

[0090] Step 9:

[0091] When the user has finished checking and correcting the document, he or she issues an instruction to save the final document on the terminal, which then sends this instruction to the server.

[0092] Step 10:

[0093] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[0094] This is the specific flow of the program processing for this system. This processing significantly reduces the administrative burden of document preparation, allowing more resources to be allocated to practical work and responding to urgent cases.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] In organizations that handle interpersonal services, employees are required to prepare numerous documents and perform data entry tasks, placing a heavy administrative burden on them. This makes it difficult for employees to concentrate on their primary duties, resulting in reduced work efficiency. The present invention aims to solve these problems and provide an environment in which employees can concentrate on their work.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for initiating voice input, means for using a microphone built into the terminal to accept voice input, means for converting collected voice data into text data using voice recognition software, means for transmitting the text data to the server, means for executing a natural language processing algorithm to analyze the text data and extract necessary information, means for embedding information in a pre-set template based on the analyzed information to automatically generate a document, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after completion of the confirmation and correction. This allows users to efficiently create documents using voice input, reducing their administrative burden and allowing them to concentrate on their primary work.

[0100] "User" refers to the employee or staff member who uses the system to input voice data.

[0101] A "terminal" is a device that a user uses to input voice, and refers to hardware such as a PC or smartphone.

[0102] "Microphone" refers to an audio input device built into a terminal for collecting audio data.

[0103] "Speech recognition software" refers to software that converts collected voice data into text data, and is a program that executes a speech recognition algorithm.

[0104] "Text data" refers to character data converted by voice recognition software.

[0105] "Server" refers to a computer system that receives text data sent from a terminal, analyzes it, and generates documents.

[0106] "Natural language processing algorithm" refers to the algorithm used by the server to analyze text data and extract the required information.

[0107] "Document template" refers to a template for entering required information in a specific format.

[0108] "Document" refers to a document that is automatically generated based on analyzed information.

[0109] "Storage" refers to the act of recording the final document in a database or file system.

[0110] "Sending" refers to the operation of electronically sending a stored document to a designated recipient, such as a relevant department.

[0111] "Confirmation" refers to the operation of the user viewing the automatically generated document and checking its contents.

[0112] "Modification" refers to an operation in which a user changes the content of an automatically generated document.

[0113] This system aims to reduce the administrative burden in organizations that handle interpersonal work and provide an environment in which employees can concentrate on their work. This system is composed of three entities: users, terminals, and a server.

[0114] First, the user accesses the system using a device such as a PC or smartphone and activates the voice input mode. The user speaks into the device, for example, saying "Today's interview details." During this step, the device's built-in microphone collects the user's voice in real time.

[0115] The collected voice data is converted into text data using voice recognition software (e.g., Google® Cloud Speech-to-Text). The voice recognition software is installed on the device and runs an algorithm that converts the voice data into text data. The converted text data is then sent from the device to a server.

[0116] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI® GPT-4®) to analyze the received text data. The NLP algorithm extracts important information from the text data (e.g., "school life" and "home environment").

[0117] Next, the server automatically embeds the extracted information into a pre-defined document template. For example, the necessary information is added to a template such as an "interview report" or "daily report," and an automatically generated document is created. The generated document is then sent from the server to the terminal and presented to the user.

[0118] The user can review the document presented on the terminal and make corrections as necessary. An editing function is provided on the terminal, allowing the user to correct text and enter additional information. Once the document has been reviewed and corrected, it can be saved on the server as the final document by clicking the "Save" button. If necessary, the saved document can be sent to the relevant department.

[0119] Specific examples

[0120] Cases at child consultation centers

[0121] 1. Start voice input:

[0122] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[0123] 2. Audio data collection:

[0124] The terminal (PC) collects the following audio: "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[0125] 3. Sending text data:

[0126] The content converted into text data is sent to the server.

[0127] 4. Text data analysis:

[0128] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[0129] 5. Automatic generation of required documents:

[0130] The server embeds the information into the "interview report" template and generates the document.

[0131] 6. Document verification and correction:

[0132] The user checks the generated interview report and makes corrections as necessary.

[0133] 7. Save and send documents:

[0134] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[0135] Cases at nursery schools

[0136] 1. Start voice input:

[0137] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0138] 2. Audio data collection:

[0139] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[0140] 3. Sending text data:

[0141] The content converted into text data is sent to the server.

[0142] 4. Text data analysis:

[0143] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[0144] 5. Automatic generation of required documents:

[0145] The server embeds the information into a "daily report" template and generates the document.

[0146] 6. Document verification and correction:

[0147] The user checks the generated daily report and makes corrections as necessary.

[0148] 7. Save and send documents:

[0149] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[0150] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] The user accesses the system using a device such as a PC or smartphone and activates voice input mode. The user clicks the "voice input mode" button and begins speaking. The input voice data is collected in real time by the device's built-in microphone. Specifically, the user speaks "today's interview content" or "today's activities."

[0154] Input: User's voice

[0155] Output: Collected audio data

[0156] Step 2:

[0157] The device uses its built-in microphone to collect voice data, which is then sent to speech recognition software (e.g., Google Cloud Speech-to-Text) and converted into text. The speech recognition software then analyzes the voice data and generates corresponding text data.

[0158] Input: Audio data

[0159] Data processing: Converting voice data into character string data using voice recognition software

[0160] Output: Text data

[0161] Step 3:

[0162] The terminal transmits the converted text data to the server. Specifically, the terminal transmits the text data to the server using an HTTP request.

[0163] Input: Text data

[0164] Data processing: Sending text data to the server

[0165] Output: Text data sent to the server

[0166] Step 4:

[0167] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The server understands the context of the text data and extracts important information (e.g., "school life," "home environment," "activity participation," and "relationships with friends").

[0168] Input: Text data sent to the server

[0169] Data processing: Analyzing text data and extracting important information using natural language processing algorithms

[0170] Output: Extracted important information

[0171] Step 5:

[0172] The server automatically embeds the extracted information into pre-defined document templates. For example, necessary information is added to templates such as "interview report" and "daily report." This automatically generates documents.

[0173] Input: Extracted sensitive information

[0174] Data processing: Automatically embedding information into pre-defined document templates

[0175] Output: Auto-generated document

[0176] Step 6:

[0177] The server sends the generated document to the terminal. Specifically, it returns the document data in an HTTP response and launches an interface for displaying it on the terminal.

[0178] Input: Auto-generated document

[0179] Data processing: Sending document data to the terminal

[0180] Output: Document displayed on the device

[0181] Step 7:

[0182] The user checks the document received on the device and makes any necessary corrections. They adjust the content using the editing function provided on the device. For example, they might correct it to "Yuta participated very actively in the activities at the nursery school."

[0183] Input: Documents displayed on the device

[0184] Data processing: User confirmation and correction of documents

[0185] Output: The corrected document

[0186] Step 8:

[0187] After the user has completed the final confirmation, they click the "Save" button to save the document. The saved document will then be automatically sent to the relevant departments as needed. Specifically, the document will be saved to the server and sent by email to the specified recipient.

[0188] Input: Corrected document

[0189] Data processing: Document storage and transmission to relevant departments

[0190] Output: Final document saved and sent

[0191] (Application example 1)

[0192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0193] Autonomous vehicles require efficient reception of passenger requests and feedback, and the ability to compile appropriate reports and documents. However, conventional systems have limitations in the accuracy and speed of voice input, which is particularly problematic in moving autonomous vehicles. Furthermore, when passengers communicate their destinations and en-route requirements in real time, the time and effort required to accurately summarize and present that information presents a problem.

[0194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0195] In this invention, the server includes means for receiving voice input using a voice input device installed inside the vehicle, means for analyzing information including destinations and requests based on the voice input, and means for automatically generating a report for passengers. This allows passengers' voice input to be collected in real time, the content of the analysis to be analyzed quickly and accurately, and, after confirmation by the passengers themselves, smooth support within the autonomous vehicle.

[0196] The "voice input device" is a device that is installed inside a vehicle and collects passenger voices in real time.

[0197] The "means for converting into text data" refers to software or algorithms for analyzing collected voice data and converting it into data as a string of characters.

[0198] "Analysis means" refers to algorithms or programs for extracting necessary information based on text data.

[0199] The "automatic report generation means" is a function that automatically creates a report in a preset format based on the analyzed information.

[0200] "User presentation means" refers to a display device or interface that allows passengers to check automatically generated documents and reports.

[0201] "Verification and correction means" means an interface or tool that allows passengers to review and, if necessary, correct the documents and reports provided to them.

[0202] "Document storage and transmission means" is a function for storing final reports and documents and transmitting them to relevant departments as necessary.

[0203] A "passenger" is a person riding in an autonomous vehicle and is the subject of voice input.

[0204] "Destination" means the passenger's designated final destination.

[0205] "Requirements" are specific details such as the services that passengers would like to receive from the self-driving vehicle or requests for detours.

[0206] A "server" is a computer system that receives data sent from a voice input device and analyzes and processes it.

[0207] This invention relates to a system for collecting and analyzing passenger voice input in real time in an autonomous vehicle, and automatically generating an appropriate report based on the necessary information. Specific embodiments of this system will be described below.

[0208] This system consists of three main components: a voice input device, a server, and a user. The voice input device is built into the interior of the autonomous vehicle and collects passengers' voices in real time. The collected voice data is then converted into text data using speech recognition software (such as the Google Cloud Speech-to-Text API) on the device.

[0209] The converted text data is sent to a server via a communication device in the autonomous vehicle. The server analyzes the received text data and extracts the necessary information using a natural language processing (NLP) algorithm (such as Google NLP API). This analysis identifies the passenger's requirements, such as the destination and any detours they may wish to make along the way.

[0210] The server automatically generates a report based on the identified information in a pre-defined format (such as an HTML or PDF template). This report is displayed on a display when the passenger exits the autonomous vehicle or at a specified time. The passenger can review the report and make any necessary corrections. The final report is saved after the passenger has been identified and sent to the relevant departments as needed.

[0211] For example, if a passenger speaks into an in-car microphone, "I'm heading to my destination, Shinjuku Station, now. I'd like to stop at a coffee shop on the way," this voice input is collected in real time. The voice data is quickly converted into text data and sent to the server. The server extracts keywords such as "Shinjuku Station" and "coffee shop," and based on this, identifies the destination and desired detours. The generated report is displayed on the in-car display for passengers to review before arriving at Shinjuku Station. After passengers make any necessary corrections, the final report is saved in PDF format.

[0212] Examples of prompts are:

[0213] "Good morning. Today you are riding in a self-driving vehicle. Your destination is Shinjuku Station, and we'd like you to stop at a coffee shop along the way. Please also write this article as a report to provide feedback on your riding experience."

[0214] This system allows passengers to communicate their requests and wishes using only voice input, which the autonomous vehicle's system can then instantly analyze, process, and generate reports on, providing a smooth and comfortable riding experience.

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] The user speaks their destination and requirements into a voice input device inside the autonomous vehicle, which collects this voice data in real time.

[0218] Input: User's voice

[0219] Output: Audio data

[0220] Step 2:

[0221] The device converts the collected voice data into text data using voice recognition software (Google Cloud Speech-to-Text API).

[0222] Input: Audio data

[0223] Output: Text data

[0224] Step 3:

[0225] The converted text data is sent from the terminal to the server, and the terminal transfers the data using an HTTP request.

[0226] Input: Text data

[0227] Output: Text data sent to the server

[0228] Step 4:

[0229] The server analyzes the received text data and extracts the necessary information using natural language processing (NLP) algorithms (Google NLP API).

[0230] Input: Received text data

[0231] Output: Extracted information (destination, requirements, etc.)

[0232] Step 5:

[0233] The server automatically generates a report based on the extracted information in a predefined format, such as an HTML or PDF template file.

[0234] Input: Extracted information

[0235] Output: Auto-generated report (a file containing the information you have embedded in the report template)

[0236] Step 6:

[0237] The server sends the generated report to a display in the autonomous vehicle, where the user can view the displayed report.

[0238] Input: Generated report

[0239] Output: Report displayed on the display

[0240] Step 7:

[0241] The user checks the report and makes corrections if necessary through the display. The terminal receives input from the user and sends the corrections to the server.

[0242] Input: User correction instructions

[0243] Output: Corrected data

[0244] Step 8:

[0245] The server receives the user's correction instructions, saves the final report, and sends it to the relevant departments as needed, for example, in cloud storage or an internal database.

[0246] Input: Corrected data and final confirmed report

[0247] Output: Final saved report and submitted report

[0248] The above are the processing steps of this system.

[0249] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0250] This invention is a system that reduces the administrative burden at institutions that handle interpersonal work, such as child consultation centers and nurseries, and provides an environment where staff can concentrate on their work. Furthermore, by combining it with an emotion engine that recognizes and analyzes user emotions, it enables flexible document creation that responds to the user's emotions. A specific embodiment of this system is shown below.

[0251] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[0252] The converted text data is sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[0253] Furthermore, the app is equipped with an emotion engine that analyzes the user's emotions from voice input. The emotion engine extracts emotion data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results. For example, if the user is feeling stressed, that emotion data is also retained as an analysis result and reflected in the document.

[0254] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[0255] Specific examples

[0256] Cases at child consultation centers

[0257] 1. Start voice input

[0258] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[0259] 2. Collection of audio data

[0260] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[0261] 3. Sending text data and emotion data

[0262] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[0263] 4. Analysis of text data and reflection of emotional data

[0264] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[0265] 5. Automatic generation of required documents

[0266] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[0267] 6. Check and correct documents

[0268] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[0269] 7. Saving and sending documents

[0270] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[0271] Cases at nursery schools

[0272] 1. Start voice input

[0273] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0274] 2. Collection of audio data

[0275] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[0276] 3. Sending text data and emotion data

[0277] Text data and emotion data are sent from the smartphone to a server.

[0278] 4. Analysis of text data and reflection of emotional data

[0279] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[0280] 5. Automatic generation of required documents

[0281] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[0282] 6. Check and correct documents

[0283] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[0284] 7. Saving and sending documents

[0285] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[0286] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[0287] The processing flow will be explained below.

[0288] Step 1:

[0289] The user starts a dedicated application on their device (PC or smartphone) and selects voice input mode. The device prepares for voice recognition and emotion analysis, and displays a message to the user that it is ready.

[0290] Step 2:

[0291] The user begins voice input, for example, saying, "Today's meeting was about my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[0292] Step 3:

[0293] The collected voice data is converted into text data using voice recognition software within the device. For example, the voice is converted into text data such as, "Today's interview content was a discussion about the child's school life and home environment."

[0294] Step 4:

[0295] The device uses the collected voice data to analyze the user's emotions using a built-in emotion engine, which takes into account voice characteristics such as tone, speed, and strength.

[0296] Step 5:

[0297] The converted text data and analyzed emotion data are sent from the device to the server, which receives the data and begins analyzing it.

[0298] Step 6:

[0299] The server analyzes the text data and extracts the necessary information. For example, it identifies key keywords and phrases in the text, such as "school life" and "home environment," and classifies each as an appropriate document item.

[0300] Step 7:

[0301] Based on the extracted information, the server embeds the data into pre-set templates and automatically generates the necessary documents. For example, it embeds the sections on "school life" and "home environment" into an "interview report" template.

[0302] Step 8:

[0303] The server also incorporates the emotional data analyzed by the emotion engine into the generated document. For example, if the user is feeling stressed, the emotional data is noted in the report.

[0304] Step 9:

[0305] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for reviewing and modifying the contents.

[0306] Step 10:

[0307] The user checks the document presented on the terminal and makes corrections as necessary. For example, they can enter additional information in the "home environment" section. The content of the emotional data can also be checked.

[0308] Step 11:

[0309] After the user has finished checking and correcting the document, he / she instructs the terminal to save the final document, and the terminal sends this instruction to the server.

[0310] Step 12:

[0311] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[0312] This specific processing step streamlines the process of creating various documents and generates rich reports that reflect the user's emotional state, allowing employees and staff to spend more time on their core tasks.

[0313] Example 2

[0314] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] In organizations that handle interpersonal services, employees and staff are required to create numerous documents, which takes time and effort, making it difficult for them to concentrate on their work. Another issue is that the user's emotions are not reflected in the documents, making it difficult to create documents that take into account the user's emotional state.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0317] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, means for analyzing the text data and extracting necessary information, means for automatically generating a document in a preset format based on the analyzed information, means for analyzing a user's emotions from the voice input, means for reflecting the emotional information in the document based on the analyzed emotional data, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after confirmation and correction are complete. This makes it possible to streamline document creation, allowing employees and staff to concentrate on their work, and further realizing flexible document creation that takes the user's emotions into consideration.

[0318] "Means for accepting voice input" refers to an interface or device that allows a user to input voice.

[0319] "Means for converting voice input into text data" refers to the process or software that uses voice recognition technology to convert voice data into text information (text data).

[0320] "Means for analyzing text data and extracting necessary information" refers to processes and algorithms that use natural language processing techniques to identify and extract relevant information from text data.

[0321] "Means for automatically generating documents according to pre-defined formats" refers to a process or system that uses pre-defined templates to automatically create documents based on collected and analyzed information.

[0322] "Means for analyzing user emotions from voice input" refers to technologies and algorithms that analyze a user's voice tone and paralinguistic information to identify emotions.

[0323] "Means for reflecting emotional information in a document based on analyzed emotional data" refers to a technique or method for adjusting the content and format of a generated document in consideration of the analyzed emotional data.

[0324] "Means for presenting the automatically generated document to the user and for reviewing and correcting it" refers to an interface or tool that displays the generated document to the user and allows the user to review its contents and edit them as necessary.

[0325] "Means for saving and sending the final document after confirmation and correction is completed" refers to a system or process that has the function of saving the document after the user has confirmed and corrected it and sending it to the specified destination.

[0326] This invention is a system that reduces the administrative burden at institutions that handle interpersonal tasks, such as child consultation centers and nurseries, and provides an environment where employees can concentrate on their work. By combining this system with an emotion engine that recognizes and analyzes the user's emotions, this system realizes flexible document creation that responds to the user's emotions.

[0327] This system consists of three entities: the user, the terminal, and the server. The roles of each entity and their specific processing methods are described below.

[0328] First, a user accesses the system using a terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API) within the terminal.

[0329] The converted text data is sent from the device to a server, which then analyzes the received text data and runs natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) to extract the required information.

[0330] The information extracted by the server is automatically embedded in a pre-set document template, which automatically generates the target document (application form, report, daily report, etc.). Furthermore, an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer) is built in to analyze the user's emotions from voice input. The emotion engine extracts emotional data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results.

[0331] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[0332] Specific examples

[0333] Cases at child consultation centers

[0334] 1. Start voice input

[0335] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[0336] 2. Collection of audio data

[0337] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[0338] 3. Sending text data and emotion data

[0339] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[0340] 4. Analysis of text data and reflection of emotional data

[0341] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[0342] 5. Automatic generation of required documents

[0343] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[0344] 6. Check and correct documents

[0345] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[0346] 7. Saving and sending documents

[0347] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[0348] Cases at nursery schools

[0349] 1. Start voice input

[0350] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0351] 2. Collection of audio data

[0352] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[0353] 3. Sending text data and emotion data

[0354] Text data and emotion data are sent from the smartphone to a server.

[0355] 4. Analysis of text data and reflection of emotional data

[0356] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[0357] 5. Automatic generation of required documents

[0358] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[0359] 6. Check and correct documents

[0360] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[0361] 7. Saving and sending documents

[0362] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[0363] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[0364] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0365] Step 1:

[0366] Start voice input

[0367] The user starts speech input mode using the terminal. The user types the command "Start speech input."

[0368] Specific actions

[0369] The user starts the dedicated application and selects the voice input mode.

[0370] The application activates the built-in microphone and prepares for voice input.

[0371] input

[0372] Spoken commands from the user

[0373] output

[0374] Voice input mode is activated

[0375] Step 2:

[0376] Audio data collection

[0377] The device uses a built-in microphone to collect the user's voice input in real time, and the voice data is temporarily stored in a buffer.

[0378] Specific actions

[0379] A microphone captures the user's voice.

[0380] The audio data is stored in a buffer as digital data at a constant sampling rate.

[0381] input

[0382] User voice data

[0383] output

[0384] Temporarily stored audio data in a buffer

[0385] Step 3:

[0386] Audio data conversion

[0387] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the voice data in the buffer into text data.

[0388] Specific actions

[0389] The speech recognition software retrieves the speech data from the buffer.

[0390] The voice data is converted into text data through phonemic analysis and text conversion algorithms.

[0391] input

[0392] Buffered audio data

[0393] output

[0394] Text data

[0395] Step 4:

[0396] Emotional Data Analysis

[0397] The device uses an emotion engine to extract emotion data from the user's voice tone, speed, strength, etc.

[0398] Specific actions

[0399] The emotion engine analyzes the voice data and extracts voice features.

[0400] Emotional states such as stress and joy are identified from the extracted features.

[0401] input

[0402] Audio data

[0403] output

[0404] Emotional Data

[0405] Step 5:

[0406] Sending data

[0407] The device sends text data and emotion data to the server, generates an HTTP request, and sends it to the endpoint.

[0408] Specific actions

[0409] Format text data and emotion data into JSON format.

[0410] Constructs an HTTP request and sends it to the specified endpoint on the server.

[0411] input

[0412] Text data, emotion data

[0413] output

[0414] Data sent to the server

[0415] Step 6:

[0416] Text data analysis

[0417] The server analyzes the received text data using natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) and extracts the necessary information.

[0418] Specific actions

[0419] Natural language processing algorithms break down the text data into individual sentences.

[0420] Extract specific keywords or phrases.

[0421] input

[0422] Text data

[0423] output

[0424] Extracted information

[0425] Step 7:

[0426] Reflecting emotional data

[0427] The server adds emotional information to the extracted information based on the analyzed emotional data.

[0428] Specific actions

[0429] Integrate emotion data into analysis results.

[0430] Generate composite information.

[0431] input

[0432] Text data analysis results, emotion data

[0433] output

[0434] Complex information including emotional information

[0435] Step 8:

[0436] Automatic generation of required documents

[0437] The server embeds the composite information into a pre-defined document template and automatically generates the document.

[0438] Specific actions

[0439] Use a template engine to embed the extracted information into a template.

[0440] Automatically generate documents according to the format.

[0441] input

[0442] Complex information including emotional information

[0443] output

[0444] Generated documents

[0445] Step 9:

[0446] Document review and correction

[0447] The generated document is sent from the server to the terminal and presented to the user, who can then check the document and make any necessary corrections.

[0448] Specific actions

[0449] The server returns the document as an HTTP response.

[0450] The device displays the document and the user uses editing tools to modify the content.

[0451] input

[0452] Generated documents

[0453] output

[0454] Documents modified by users

[0455] Step 10:

[0456] Saving and sending documents

[0457] After the user completes the final confirmation, they save the document and send it to the relevant department if necessary.

[0458] Specific actions

[0459] The terminal sends the modified document to the server.

[0460] The server stores the document in a database and sends it to the specified destination.

[0461] input

[0462] Documents modified by users

[0463] output

[0464] Final documents saved and sent

[0465] (Application example 2)

[0466] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0467] In current interpersonal services, when staff manually create documents, they often experience problems such as reduced work efficiency and difficulty concentrating on their work. Furthermore, conventional systems have difficulty understanding users' emotions and responding appropriately, which can lead to a decline in the quality of service. To solve these issues, a system that combines voice input and emotion analysis is needed.

[0468] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0469] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, and means for automatically generating documents based on the analyzed information, thereby enabling personnel performing interpersonal tasks to quickly create documents using voice, and further enabling emotional data to be analyzed and reflected in the documents.

[0470] "Voice input" refers to voice data emitted by a user through a microphone.

[0471] "Text data" refers to data that has been converted from voice input into a character string format.

[0472] "Analysis" refers to the process of extracting the intended meaning and necessary information from text data.

[0473] "Format" refers to a pre-defined template used to generate a document.

[0474] "Automatic generation" refers to the process by which the system automatically creates documents based on analyzed information.

[0475] "Presenting" refers to showing the generated document to the user.

[0476] "Modification" refers to the user changing the content of the presented document as needed.

[0477] "Preservation" refers to the recording of the final document in digital or physical form.

[0478] "Transmission" refers to the electronic transfer of the final document to the relevant department or party.

[0479] "Emotion data" refers to data that represents the user's emotional state at the time of voice input.

[0480] "Emotion analysis" refers to the process of analyzing a user's emotions based on the tone, speed, and strength of voice input.

[0481] The system of the present invention is composed of three main components: a user, a terminal, and a server, for quickly creating documents using voice input. The details of each component and their operation are explained below.

[0482] First, a user accesses the system using a device (e.g., a smartphone or tablet). The user activates the device's voice input mode and inputs voice data through the microphone. This voice input is converted into text data in real time using speech recognition software in the device (e.g., Python's speech_recognition library).

[0483] The converted text data is sent from the device to a server. The server receives the text data and analyzes it using natural language processing (NLP) algorithms. The analysis process involves extracting necessary information from the text data. The server also has a built-in emotion engine that analyzes emotional data in voice input, extracting emotional data from the user's voice tone, speed, and strength.

[0484] The analysis results are automatically embedded into a document based on a pre-defined document format. This document can be used, for example, for service guides or proposals. The generated document is then resent from the server to the terminal, where the user can check and modify it. After the user completes the final check, the document is saved and sent to the relevant department.

[0485] A concrete example would be an application where a concierge at a luxury hotel or department store uses a smartphone to input conversations and questions with customers by voice, and then performs sentiment analysis. For example, if the concierge says, "The customer says he wants to relax. I'd like to recommend some spa services," the application converts the speech into text and sends it to a server. The server analyzes the data and automatically generates a proposal for a spa service that matches the customer's relaxation needs.

[0486] Here are some examples of relevant prompts:

[0487] "Your client says they want to relax. Please create a spa service proposal based on the following information:\n\n{Input data about your client's needs}\n\n---\n\nProposal:\n"

[0488] This system allows staff who handle interpersonal tasks to quickly create documents using voice commands and can analyze emotional data and reflect it in the documents, thereby improving the quality of service and realizing efficient work execution.

[0489] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0490] Step 1:

[0491] The user activates the voice input mode of the terminal and inputs voice data. The input voice data is collected through a microphone. At this point, the input is the user's voice, and the output is digital voice data.

[0492] Step 2:

[0493] The voice data collected by the device is converted into text data in real time using voice recognition software (for example, Python's speech_recognition library). The input is voice data and the output is text data. In this step, the voice data is converted into a string.

[0494] Step 3:

[0495] The terminal sends the converted text data to the server. The input is text data, and the output is data sent to the server. Data communication takes place in this step.

[0496] Step 4:

[0497] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is text data, and the output is analyzed information. In this step, the necessary information is extracted from the text data.

[0498] Step 5:

[0499] The server extracts the user's emotional data using an emotion engine that analyzes the emotional data from the voice input. The input is voice data, and the output is emotional data. In this step, data analysis is performed on the voice tone, speed, and strength.

[0500] Step 6:

[0501] The server automatically generates a document based on the analyzed information and emotion data according to a pre-set document format. The input is the analyzed information and emotion data, and the output is an automatically generated document. This step involves the information embedding process.

[0502] Step 7:

[0503] The server sends the automatically generated document to the terminal. The input is the automatically generated document, and the output is the data sent to the terminal. In this step, data communication occurs again.

[0504] Step 8:

[0505] The user checks the document received on the terminal and makes corrections if necessary. The input is the automatically generated document, and the output is the corrected document. This is the step where user interaction takes place.

[0506] Step 9:

[0507] After the user has confirmed and corrected the final document, it is saved on the terminal and sent to the relevant department. The input is the corrected document, and the output is the saved and sent data. In this step, the document is saved and the data is finally sent.

[0508] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0510] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0511] [Second embodiment]

[0512] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0513] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0515] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0519] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0520] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0521] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0522] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0523] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0524] The present invention is a system for reducing the administrative burden at child consultation centers, nurseries, and other institutions that provide interpersonal services, and providing an environment in which they can concentrate on their work. A specific embodiment of this system is described below.

[0525] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[0526] The converted text data is then sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[0527] The generated document is sent from the server to the terminal and presented to the user. The user can check the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished checking and correcting, the final document is saved and sent to the relevant department as necessary.

[0528] Specific examples

[0529] Cases at child consultation centers

[0530] 1. Start voice input

[0531] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[0532] 2. Collection of audio data

[0533] The device collects audio that says, "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[0534] 3. Sending text data

[0535] The content converted into text data is sent to the server.

[0536] 4. Text Data Analysis

[0537] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[0538] 5. Automatic generation of required documents

[0539] The server embeds the information into the "interview report" template and generates the document.

[0540] 6. Check and correct documents

[0541] The user checks the generated interview report and makes corrections as necessary.

[0542] 7. Saving and sending documents

[0543] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[0544] Cases at nursery schools

[0545] 1. Start voice input

[0546] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0547] 2. Collection of audio data

[0548] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[0549] 3. Sending text data

[0550] The content converted into text data is sent to the server.

[0551] 4. Text Data Analysis

[0552] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[0553] 5. Automatic generation of required documents

[0554] The server embeds the information into a "daily report" template and generates the document.

[0555] 6. Check and correct documents

[0556] The user checks the generated daily report and makes corrections as necessary.

[0557] 7. Saving and sending documents

[0558] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[0559] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[0560] The processing flow will be explained below.

[0561] Step 1:

[0562] The user starts a dedicated application on the device (PC or smartphone) and selects voice input mode. When voice input mode is activated, the device prepares for voice recognition. A message is displayed to the user indicating that the device is ready.

[0563] Step 2:

[0564] The user initiates voice input, for example, saying, "Today's meeting discussed my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[0565] Step 3:

[0566] The collected voice data is converted into text data using voice recognition software on the device. The converted data might look something like, "During today's interview, we discussed my child's school life and home environment."

[0567] Step 4:

[0568] Text data is sent from the device to the server, which then begins analyzing it using natural language processing (NLP) algorithms.

[0569] Step 5:

[0570] The server analyzes the text data and extracts the necessary information, for example identifying key keywords and phrases in the text, such as "school life" and "home environment," and classifying each as an appropriate document item.

[0571] Step 6:

[0572] Based on the extracted information, the server fills in pre-set templates and automatically generates the necessary documents. For example, it fills in the "school life" and "home environment" sections in an "interview report" template.

[0573] Step 7:

[0574] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for checking and modifying the contents.

[0575] Step 8:

[0576] The user checks the document presented on the terminal and makes corrections as necessary, such as entering additional information in the "home environment" section.

[0577] Step 9:

[0578] When the user has finished checking and correcting the document, he or she issues an instruction to save the final document on the terminal, which then sends this instruction to the server.

[0579] Step 10:

[0580] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[0581] This is the specific flow of the program processing for this system. This processing significantly reduces the administrative burden of document preparation, allowing more resources to be allocated to practical work and responding to urgent cases.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In organizations that handle interpersonal services, employees are required to prepare numerous documents and perform data entry tasks, placing a heavy administrative burden on them. This makes it difficult for employees to concentrate on their primary duties, resulting in reduced work efficiency. The present invention aims to solve these problems and provide an environment in which employees can concentrate on their work.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for initiating voice input, means for using a microphone built into the terminal to accept voice input, means for converting collected voice data into text data using voice recognition software, means for transmitting the text data to the server, means for executing a natural language processing algorithm to analyze the text data and extract necessary information, means for embedding information in a pre-set template based on the analyzed information to automatically generate a document, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after completion of the confirmation and correction. This allows users to efficiently create documents using voice input, reducing their administrative burden and allowing them to concentrate on their primary work.

[0587] "User" refers to the employee or staff member who uses the system to input voice data.

[0588] A "terminal" is a device that a user uses to input voice, and refers to hardware such as a PC or smartphone.

[0589] "Microphone" refers to an audio input device built into a terminal for collecting audio data.

[0590] "Speech recognition software" refers to software that converts collected voice data into text data, and is a program that executes a speech recognition algorithm.

[0591] "Text data" refers to character data converted by voice recognition software.

[0592] "Server" refers to a computer system that receives text data sent from a terminal, analyzes it, and generates documents.

[0593] "Natural language processing algorithm" refers to the algorithm used by the server to analyze text data and extract the required information.

[0594] "Document template" refers to a template for entering required information in a specific format.

[0595] "Document" refers to a document that is automatically generated based on analyzed information.

[0596] "Storage" refers to the act of recording the final document in a database or file system.

[0597] "Sending" refers to the operation of electronically sending a stored document to a designated recipient, such as a relevant department.

[0598] "Confirmation" refers to the operation of the user viewing the automatically generated document and checking its contents.

[0599] "Modification" refers to an operation in which a user changes the content of an automatically generated document.

[0600] This system aims to reduce the administrative burden in organizations that handle interpersonal work and provide an environment in which employees can concentrate on their work. This system is composed of three entities: users, terminals, and a server.

[0601] First, the user accesses the system using a device such as a PC or smartphone and activates the voice input mode. The user speaks into the device, for example, saying "Today's interview details." During this step, the device's built-in microphone collects the user's voice in real time.

[0602] The collected voice data is converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition software is installed on the device and runs an algorithm that converts the voice data into text data. The converted text data is then sent from the device to a server.

[0603] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The NLP algorithm extracts important information from the text data (e.g., "school life" and "home environment").

[0604] Next, the server automatically embeds the extracted information into a pre-defined document template. For example, the necessary information is added to a template such as an "interview report" or "daily report," and an automatically generated document is created. The generated document is then sent from the server to the terminal and presented to the user.

[0605] The user can review the document presented on the terminal and make corrections as necessary. An editing function is provided on the terminal, allowing the user to correct text and enter additional information. Once the document has been reviewed and corrected, it can be saved on the server as the final document by clicking the "Save" button. If necessary, the saved document can be sent to the relevant department.

[0606] Specific examples

[0607] Cases at child consultation centers

[0608] 1. Start voice input:

[0609] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[0610] 2. Audio data collection:

[0611] The terminal (PC) collects the following audio: "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[0612] 3. Sending text data:

[0613] The content converted into text data is sent to the server.

[0614] 4. Text data analysis:

[0615] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[0616] 5. Automatic generation of required documents:

[0617] The server embeds the information into the "interview report" template and generates the document.

[0618] 6. Document verification and correction:

[0619] The user checks the generated interview report and makes corrections as necessary.

[0620] 7. Save and send documents:

[0621] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[0622] Cases at nursery schools

[0623] 1. Start voice input:

[0624] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0625] 2. Audio data collection:

[0626] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[0627] 3. Sending text data:

[0628] The content converted into text data is sent to the server.

[0629] 4. Text data analysis:

[0630] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[0631] 5. Automatic generation of required documents:

[0632] The server embeds the information into a "daily report" template and generates the document.

[0633] 6. Document verification and correction:

[0634] The user checks the generated daily report and makes corrections as necessary.

[0635] 7. Save and send documents:

[0636] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[0637] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[0638] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0639] Step 1:

[0640] The user accesses the system using a device such as a PC or smartphone and activates voice input mode. The user clicks the "voice input mode" button and begins speaking. The input voice data is collected in real time by the device's built-in microphone. Specifically, the user speaks "today's interview content" or "today's activities."

[0641] Input: User's voice

[0642] Output: Collected audio data

[0643] Step 2:

[0644] The device uses its built-in microphone to collect voice data, which is then sent to speech recognition software (e.g., Google Cloud Speech-to-Text) and converted into text. The speech recognition software then analyzes the voice data and generates corresponding text data.

[0645] Input: Audio data

[0646] Data processing: Converting voice data into character string data using voice recognition software

[0647] Output: Text data

[0648] Step 3:

[0649] The terminal transmits the converted text data to the server. Specifically, the terminal transmits the text data to the server using an HTTP request.

[0650] Input: Text data

[0651] Data processing: Sending text data to the server

[0652] Output: Text data sent to the server

[0653] Step 4:

[0654] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The server understands the context of the text data and extracts important information (e.g., "school life," "home environment," "activity participation," and "relationships with friends").

[0655] Input: Text data sent to the server

[0656] Data processing: Analyzing text data and extracting important information using natural language processing algorithms

[0657] Output: Extracted important information

[0658] Step 5:

[0659] The server automatically embeds the extracted information into pre-defined document templates. For example, necessary information is added to templates such as "interview report" and "daily report." This automatically generates documents.

[0660] Input: Extracted sensitive information

[0661] Data processing: Automatically embedding information into pre-defined document templates

[0662] Output: Auto-generated document

[0663] Step 6:

[0664] The server sends the generated document to the terminal. Specifically, it returns the document data in an HTTP response and launches an interface for displaying it on the terminal.

[0665] Input: Auto-generated document

[0666] Data processing: Sending document data to the terminal

[0667] Output: Document displayed on the device

[0668] Step 7:

[0669] The user checks the document received on the device and makes any necessary corrections. They adjust the content using the editing function provided on the device. For example, they might correct it to "Yuta participated very actively in the activities at the nursery school."

[0670] Input: Documents displayed on the device

[0671] Data processing: User confirmation and correction of documents

[0672] Output: The corrected document

[0673] Step 8:

[0674] After the user has completed the final confirmation, they click the "Save" button to save the document. The saved document will then be automatically sent to the relevant departments as needed. Specifically, the document will be saved to the server and sent by email to the specified recipient.

[0675] Input: Corrected document

[0676] Data processing: Document storage and transmission to relevant departments

[0677] Output: Final document saved and sent

[0678] (Application example 1)

[0679] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Autonomous vehicles require efficient reception of passenger requests and feedback, and the ability to compile appropriate reports and documents. However, conventional systems have limitations in the accuracy and speed of voice input, which is particularly problematic in moving autonomous vehicles. Furthermore, when passengers communicate their destinations and en-route requirements in real time, the time and effort required to accurately summarize and present that information presents a problem.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0682] In this invention, the server includes means for receiving voice input using a voice input device installed inside the vehicle, means for analyzing information including destinations and requests based on the voice input, and means for automatically generating a report for passengers. This allows passengers' voice input to be collected in real time, the content of the analysis to be analyzed quickly and accurately, and, after confirmation by the passengers themselves, smooth support within the autonomous vehicle.

[0683] The "voice input device" is a device that is installed inside a vehicle and collects passenger voices in real time.

[0684] The "means for converting into text data" refers to software or algorithms for analyzing collected voice data and converting it into data as a string of characters.

[0685] "Analysis means" refers to algorithms or programs for extracting necessary information based on text data.

[0686] The "automatic report generation means" is a function that automatically creates a report in a preset format based on the analyzed information.

[0687] "User presentation means" refers to a display device or interface that allows passengers to check automatically generated documents and reports.

[0688] "Verification and correction means" means an interface or tool that allows passengers to review and, if necessary, correct the documents and reports provided to them.

[0689] "Document storage and transmission means" is a function for storing final reports and documents and transmitting them to relevant departments as necessary.

[0690] A "passenger" is a person riding in an autonomous vehicle and is the subject of voice input.

[0691] "Destination" means the passenger's designated final destination.

[0692] "Requirements" are specific details such as the services that passengers would like to receive from the self-driving vehicle or requests for detours.

[0693] A "server" is a computer system that receives data sent from a voice input device and analyzes and processes it.

[0694] This invention relates to a system for collecting and analyzing passenger voice input in real time in an autonomous vehicle, and automatically generating an appropriate report based on the necessary information. Specific embodiments of this system will be described below.

[0695] This system consists of three main components: a voice input device, a server, and a user. The voice input device is built into the interior of the autonomous vehicle and collects passengers' voices in real time. The collected voice data is then converted into text data using speech recognition software (such as the Google Cloud Speech-to-Text API) on the device.

[0696] The converted text data is sent to a server via a communication device in the autonomous vehicle. The server analyzes the received text data and extracts the necessary information using a natural language processing (NLP) algorithm (such as Google NLP API). This analysis identifies the passenger's requirements, such as the destination and any detours they may wish to make along the way.

[0697] The server automatically generates a report based on the identified information in a pre-defined format (such as an HTML or PDF template). This report is displayed on a display when the passenger exits the autonomous vehicle or at a specified time. The passenger can review the report and make any necessary corrections. The final report is saved after the passenger has been identified and sent to the relevant departments as needed.

[0698] For example, if a passenger speaks into an in-car microphone, "I'm heading to my destination, Shinjuku Station, now. I'd like to stop at a coffee shop on the way," this voice input is collected in real time. The voice data is quickly converted into text data and sent to the server. The server extracts keywords such as "Shinjuku Station" and "coffee shop," and based on this, identifies the destination and desired detours. The generated report is displayed on the in-car display for passengers to review before arriving at Shinjuku Station. After passengers make any necessary corrections, the final report is saved in PDF format.

[0699] Examples of prompts are:

[0700] "Good morning. Today you are riding in a self-driving vehicle. Your destination is Shinjuku Station, and we'd like you to stop at a coffee shop along the way. Please also write this article as a report to provide feedback on your riding experience."

[0701] This system allows passengers to communicate their requests and wishes using only voice input, which the autonomous vehicle's system can then instantly analyze, process, and generate reports on, providing a smooth and comfortable riding experience.

[0702] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0703] Step 1:

[0704] The user speaks their destination and requirements into a voice input device inside the autonomous vehicle, which collects this voice data in real time.

[0705] Input: User's voice

[0706] Output: Audio data

[0707] Step 2:

[0708] The device converts the collected voice data into text data using voice recognition software (Google Cloud Speech-to-Text API).

[0709] Input: Audio data

[0710] Output: Text data

[0711] Step 3:

[0712] The converted text data is sent from the terminal to the server, and the terminal transfers the data using an HTTP request.

[0713] Input: Text data

[0714] Output: Text data sent to the server

[0715] Step 4:

[0716] The server analyzes the received text data and extracts the necessary information using natural language processing (NLP) algorithms (Google NLP API).

[0717] Input: Received text data

[0718] Output: Extracted information (destination, requirements, etc.)

[0719] Step 5:

[0720] The server automatically generates a report based on the extracted information in a predefined format, such as an HTML or PDF template file.

[0721] Input: Extracted information

[0722] Output: Auto-generated report (a file containing the information you have embedded in the report template)

[0723] Step 6:

[0724] The server sends the generated report to a display in the autonomous vehicle, where the user can view the displayed report.

[0725] Input: Generated report

[0726] Output: Report displayed on the display

[0727] Step 7:

[0728] The user checks the report and makes corrections if necessary through the display. The terminal receives input from the user and sends the corrections to the server.

[0729] Input: User correction instructions

[0730] Output: Corrected data

[0731] Step 8:

[0732] The server receives the user's correction instructions, saves the final report, and sends it to the relevant departments as needed, for example, in cloud storage or an internal database.

[0733] Input: Corrected data and final confirmed report

[0734] Output: Final saved report and submitted report

[0735] The above are the processing steps of this system.

[0736] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0737] This invention is a system that reduces the administrative burden at institutions that handle interpersonal work, such as child consultation centers and nurseries, and provides an environment where staff can concentrate on their work. Furthermore, by combining it with an emotion engine that recognizes and analyzes user emotions, it enables flexible document creation that responds to the user's emotions. A specific embodiment of this system is shown below.

[0738] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[0739] The converted text data is sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[0740] Furthermore, the app is equipped with an emotion engine that analyzes the user's emotions from voice input. The emotion engine extracts emotion data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results. For example, if the user is feeling stressed, that emotion data is also retained as an analysis result and reflected in the document.

[0741] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[0742] Specific examples

[0743] Cases at child consultation centers

[0744] 1. Start voice input

[0745] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[0746] 2. Collection of audio data

[0747] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[0748] 3. Sending text data and emotion data

[0749] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[0750] 4. Analysis of text data and reflection of emotional data

[0751] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[0752] 5. Automatic generation of required documents

[0753] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[0754] 6. Check and correct documents

[0755] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[0756] 7. Saving and sending documents

[0757] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[0758] Cases at nursery schools

[0759] 1. Start voice input

[0760] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0761] 2. Collection of audio data

[0762] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[0763] 3. Sending text data and emotion data

[0764] Text data and emotion data are sent from the smartphone to a server.

[0765] 4. Analysis of text data and reflection of emotional data

[0766] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[0767] 5. Automatic generation of required documents

[0768] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[0769] 6. Check and correct documents

[0770] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[0771] 7. Saving and sending documents

[0772] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[0773] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[0774] The processing flow will be explained below.

[0775] Step 1:

[0776] The user starts a dedicated application on their device (PC or smartphone) and selects voice input mode. The device prepares for voice recognition and emotion analysis, and displays a message to the user that it is ready.

[0777] Step 2:

[0778] The user begins voice input, for example, saying, "Today's meeting was about my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[0779] Step 3:

[0780] The collected voice data is converted into text data using voice recognition software within the device. For example, the voice is converted into text data such as, "Today's interview content was a discussion about the child's school life and home environment."

[0781] Step 4:

[0782] The device uses the collected voice data to analyze the user's emotions using a built-in emotion engine, which takes into account voice characteristics such as tone, speed, and strength.

[0783] Step 5:

[0784] The converted text data and analyzed emotion data are sent from the device to the server, which receives the data and begins analyzing it.

[0785] Step 6:

[0786] The server analyzes the text data and extracts the necessary information. For example, it identifies key keywords and phrases in the text, such as "school life" and "home environment," and classifies each as an appropriate document item.

[0787] Step 7:

[0788] Based on the extracted information, the server embeds the data into pre-set templates and automatically generates the necessary documents. For example, it embeds the sections on "school life" and "home environment" into an "interview report" template.

[0789] Step 8:

[0790] The server also incorporates the emotional data analyzed by the emotion engine into the generated document. For example, if the user is feeling stressed, the emotional data is noted in the report.

[0791] Step 9:

[0792] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for reviewing and modifying the contents.

[0793] Step 10:

[0794] The user checks the document presented on the terminal and makes corrections as necessary. For example, they can enter additional information in the "home environment" section. The content of the emotional data can also be checked.

[0795] Step 11:

[0796] After the user has finished checking and correcting the document, he / she instructs the terminal to save the final document, and the terminal sends this instruction to the server.

[0797] Step 12:

[0798] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[0799] This specific processing step streamlines the process of creating various documents and generates rich reports that reflect the user's emotional state, allowing employees and staff to spend more time on their core tasks.

[0800] Example 2

[0801] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0802] In organizations that handle interpersonal services, employees and staff are required to create numerous documents, which takes time and effort, making it difficult for them to concentrate on their work. Another issue is that the user's emotions are not reflected in the documents, making it difficult to create documents that take into account the user's emotional state.

[0803] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0804] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, means for analyzing the text data and extracting necessary information, means for automatically generating a document in a preset format based on the analyzed information, means for analyzing a user's emotions from the voice input, means for reflecting the emotional information in the document based on the analyzed emotional data, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after confirmation and correction are complete. This makes it possible to streamline document creation, allowing employees and staff to concentrate on their work, and further realizing flexible document creation that takes the user's emotions into consideration.

[0805] "Means for accepting voice input" refers to an interface or device that allows a user to input voice.

[0806] "Means for converting voice input into text data" refers to the process or software that uses voice recognition technology to convert voice data into text information (text data).

[0807] "Means for analyzing text data and extracting necessary information" refers to processes and algorithms that use natural language processing techniques to identify and extract relevant information from text data.

[0808] "Means for automatically generating documents according to pre-defined formats" refers to a process or system that uses pre-defined templates to automatically create documents based on collected and analyzed information.

[0809] "Means for analyzing user emotions from voice input" refers to technologies and algorithms that analyze a user's voice tone and paralinguistic information to identify emotions.

[0810] "Means for reflecting emotional information in a document based on analyzed emotional data" refers to a technique or method for adjusting the content and format of a generated document in consideration of the analyzed emotional data.

[0811] "Means for presenting the automatically generated document to the user and for reviewing and correcting it" refers to an interface or tool that displays the generated document to the user and allows the user to review its contents and edit them as necessary.

[0812] "Means for saving and sending the final document after confirmation and correction is completed" refers to a system or process that has the function of saving the document after the user has confirmed and corrected it and sending it to the specified destination.

[0813] This invention is a system that reduces the administrative burden at institutions that handle interpersonal tasks, such as child consultation centers and nurseries, and provides an environment where employees can concentrate on their work. By combining this system with an emotion engine that recognizes and analyzes the user's emotions, this system realizes flexible document creation that responds to the user's emotions.

[0814] This system consists of three entities: the user, the terminal, and the server. The roles of each entity and their specific processing methods are described below.

[0815] First, a user accesses the system using a terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API) within the terminal.

[0816] The converted text data is sent from the device to a server, which then analyzes the received text data and runs natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) to extract the required information.

[0817] The information extracted by the server is automatically embedded in a pre-set document template, which automatically generates the target document (application form, report, daily report, etc.). Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) is built in to analyze the user's emotions from voice input. The emotion engine extracts emotional data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results.

[0818] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[0819] Specific examples

[0820] Cases at child consultation centers

[0821] 1. Start voice input

[0822] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[0823] 2. Collection of audio data

[0824] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[0825] 3. Sending text data and emotion data

[0826] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[0827] 4. Analysis of text data and reflection of emotional data

[0828] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[0829] 5. Automatic generation of required documents

[0830] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[0831] 6. Check and correct documents

[0832] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[0833] 7. Saving and sending documents

[0834] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[0835] Cases at nursery schools

[0836] 1. Start voice input

[0837] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[0838] 2. Collection of audio data

[0839] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[0840] 3. Sending text data and emotion data

[0841] Text data and emotion data are sent from the smartphone to a server.

[0842] 4. Analysis of text data and reflection of emotional data

[0843] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[0844] 5. Automatic generation of required documents

[0845] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[0846] 6. Check and correct documents

[0847] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[0848] 7. Saving and sending documents

[0849] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[0850] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Step 1:

[0853] Start voice input

[0854] The user starts speech input mode using the terminal. The user types the command "Start speech input."

[0855] Specific actions

[0856] The user starts the dedicated application and selects the voice input mode.

[0857] The application activates the built-in microphone and prepares for voice input.

[0858] input

[0859] Spoken commands from the user

[0860] output

[0861] Voice input mode is activated

[0862] Step 2:

[0863] Audio data collection

[0864] The device uses a built-in microphone to collect the user's voice input in real time, and the voice data is temporarily stored in a buffer.

[0865] Specific actions

[0866] A microphone captures the user's voice.

[0867] The audio data is stored in a buffer as digital data at a constant sampling rate.

[0868] input

[0869] User voice data

[0870] output

[0871] Temporarily stored audio data in a buffer

[0872] Step 3:

[0873] Audio data conversion

[0874] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the voice data in the buffer into text data.

[0875] Specific actions

[0876] The speech recognition software retrieves the speech data from the buffer.

[0877] The voice data is converted into text data through phonemic analysis and text conversion algorithms.

[0878] input

[0879] Buffered audio data

[0880] output

[0881] Text data

[0882] Step 4:

[0883] Emotional Data Analysis

[0884] The device uses an emotion engine to extract emotion data from the user's voice tone, speed, strength, etc.

[0885] Specific actions

[0886] The emotion engine analyzes the voice data and extracts voice features.

[0887] Emotional states such as stress and joy are identified from the extracted features.

[0888] input

[0889] Audio data

[0890] output

[0891] Emotional Data

[0892] Step 5:

[0893] Sending data

[0894] The device sends text data and emotion data to the server, generates an HTTP request, and sends it to the endpoint.

[0895] Specific actions

[0896] Format text data and emotion data into JSON format.

[0897] Constructs an HTTP request and sends it to the specified endpoint on the server.

[0898] input

[0899] Text data, emotion data

[0900] output

[0901] Data sent to the server

[0902] Step 6:

[0903] Text data analysis

[0904] The server analyzes the received text data using natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) and extracts the necessary information.

[0905] Specific actions

[0906] Natural language processing algorithms break down the text data into individual sentences.

[0907] Extract specific keywords or phrases.

[0908] input

[0909] Text data

[0910] output

[0911] Extracted information

[0912] Step 7:

[0913] Reflecting emotional data

[0914] The server adds emotional information to the extracted information based on the analyzed emotional data.

[0915] Specific actions

[0916] Integrate emotion data into analysis results.

[0917] Generate composite information.

[0918] input

[0919] Text data analysis results, emotion data

[0920] output

[0921] Complex information including emotional information

[0922] Step 8:

[0923] Automatic generation of required documents

[0924] The server embeds the composite information into a pre-defined document template and automatically generates the document.

[0925] Specific actions

[0926] Use a template engine to embed the extracted information into a template.

[0927] Automatically generate documents according to the format.

[0928] input

[0929] Complex information including emotional information

[0930] output

[0931] Generated documents

[0932] Step 9:

[0933] Document review and correction

[0934] The generated document is sent from the server to the terminal and presented to the user, who can then check the document and make any necessary corrections.

[0935] Specific actions

[0936] The server returns the document as an HTTP response.

[0937] The device displays the document and the user uses editing tools to modify the content.

[0938] input

[0939] Generated documents

[0940] output

[0941] Documents modified by users

[0942] Step 10:

[0943] Saving and sending documents

[0944] After the user completes the final confirmation, they save the document and send it to the relevant department if necessary.

[0945] Specific actions

[0946] The terminal sends the modified document to the server.

[0947] The server stores the document in a database and sends it to the specified destination.

[0948] input

[0949] Documents modified by users

[0950] output

[0951] Final documents saved and sent

[0952] (Application example 2)

[0953] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0954] In current interpersonal services, when staff manually create documents, they often experience problems such as reduced work efficiency and difficulty concentrating on their work. Furthermore, conventional systems have difficulty understanding users' emotions and responding appropriately, which can lead to a decline in the quality of service. To solve these issues, a system that combines voice input and emotion analysis is needed.

[0955] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0956] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, and means for automatically generating documents based on the analyzed information, thereby enabling personnel performing interpersonal tasks to quickly create documents using voice, and further enabling emotional data to be analyzed and reflected in the documents.

[0957] "Voice input" refers to voice data emitted by a user through a microphone.

[0958] "Text data" refers to data that has been converted from voice input into a character string format.

[0959] "Analysis" refers to the process of extracting the intended meaning and necessary information from text data.

[0960] "Format" refers to a pre-defined template used to generate a document.

[0961] "Automatic generation" refers to the process by which the system automatically creates documents based on analyzed information.

[0962] "Presenting" refers to showing the generated document to the user.

[0963] "Modification" refers to the user changing the content of the presented document as needed.

[0964] "Preservation" refers to the recording of the final document in digital or physical form.

[0965] "Transmission" refers to the electronic transfer of the final document to the relevant department or party.

[0966] "Emotion data" refers to data that represents the user's emotional state at the time of voice input.

[0967] "Emotion analysis" refers to the process of analyzing a user's emotions based on the tone, speed, and strength of voice input.

[0968] The system of the present invention is composed of three main components: a user, a terminal, and a server, for quickly creating documents using voice input. The details of each component and their operation are explained below.

[0969] First, a user accesses the system using a device (e.g., a smartphone or tablet). The user activates the device's voice input mode and inputs voice data through the microphone. This voice input is converted into text data in real time using speech recognition software in the device (e.g., Python's speech_recognition library).

[0970] The converted text data is sent from the device to a server. The server receives the text data and analyzes it using natural language processing (NLP) algorithms. The analysis process involves extracting necessary information from the text data. The server also has a built-in emotion engine that analyzes emotional data in voice input, extracting emotional data from the user's voice tone, speed, and strength.

[0971] The analysis results are automatically embedded into a document based on a pre-defined document format. This document can be used, for example, for service guides or proposals. The generated document is then resent from the server to the terminal, where the user can check and modify it. After the user completes the final check, the document is saved and sent to the relevant department.

[0972] A concrete example would be an application where a concierge at a luxury hotel or department store uses a smartphone to input conversations and questions with customers by voice, and then performs sentiment analysis. For example, if the concierge says, "The customer says he wants to relax. I'd like to recommend some spa services," the application converts the speech into text and sends it to a server. The server analyzes the data and automatically generates a proposal for a spa service that matches the customer's relaxation needs.

[0973] Here are some examples of relevant prompts:

[0974] "Your client says they want to relax. Please create a spa service proposal based on the following information:\n\n{Input data about your client's needs}\n\n---\n\nProposal:\n"

[0975] This system allows staff who handle interpersonal tasks to quickly create documents using voice commands and can analyze emotional data and reflect it in the documents, thereby improving the quality of service and realizing efficient work execution.

[0976] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0977] Step 1:

[0978] The user activates the voice input mode of the terminal and inputs voice data. The input voice data is collected through a microphone. At this point, the input is the user's voice, and the output is digital voice data.

[0979] Step 2:

[0980] The voice data collected by the device is converted into text data in real time using voice recognition software (for example, Python's speech_recognition library). The input is voice data and the output is text data. In this step, the voice data is converted into a string.

[0981] Step 3:

[0982] The terminal sends the converted text data to the server. The input is text data, and the output is data sent to the server. Data communication takes place in this step.

[0983] Step 4:

[0984] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is text data, and the output is analyzed information. In this step, the necessary information is extracted from the text data.

[0985] Step 5:

[0986] The server extracts the user's emotional data using an emotion engine that analyzes the emotional data from the voice input. The input is voice data, and the output is emotional data. In this step, data analysis is performed on the voice tone, speed, and strength.

[0987] Step 6:

[0988] The server automatically generates a document based on the analyzed information and emotion data according to a pre-set document format. The input is the analyzed information and emotion data, and the output is an automatically generated document. This step involves the information embedding process.

[0989] Step 7:

[0990] The server sends the automatically generated document to the terminal. The input is the automatically generated document, and the output is the data sent to the terminal. In this step, data communication occurs again.

[0991] Step 8:

[0992] The user checks the document received on the terminal and makes corrections if necessary. The input is the automatically generated document, and the output is the corrected document. This is the step where user interaction takes place.

[0993] Step 9:

[0994] After the user has confirmed and corrected the final document, it is saved on the terminal and sent to the relevant department. The input is the corrected document, and the output is the saved and sent data. In this step, the document is saved and the data is finally sent.

[0995] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0996] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0997] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0998] [Third embodiment]

[0999] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1000] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1001] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1002] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1003] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1004] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1005] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1006] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1007] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1008] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1009] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1010] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1011] The present invention is a system for reducing the administrative burden at child consultation centers, nurseries, and other institutions that provide interpersonal services, and providing an environment in which they can concentrate on their work. A specific embodiment of this system is described below.

[1012] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[1013] The converted text data is then sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[1014] The generated document is sent from the server to the terminal and presented to the user. The user can check the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished checking and correcting, the final document is saved and sent to the relevant department as necessary.

[1015] Specific examples

[1016] Cases at child consultation centers

[1017] 1. Start voice input

[1018] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[1019] 2. Collection of audio data

[1020] The device collects audio that says, "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[1021] 3. Sending text data

[1022] The content converted into text data is sent to the server.

[1023] 4. Text Data Analysis

[1024] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[1025] 5. Automatic generation of required documents

[1026] The server embeds the information into the "interview report" template and generates the document.

[1027] 6. Check and correct documents

[1028] The user checks the generated interview report and makes corrections as necessary.

[1029] 7. Saving and sending documents

[1030] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[1031] Cases at nursery schools

[1032] 1. Start voice input

[1033] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1034] 2. Collection of audio data

[1035] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[1036] 3. Sending text data

[1037] The content converted into text data is sent to the server.

[1038] 4. Text Data Analysis

[1039] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[1040] 5. Automatic generation of required documents

[1041] The server embeds the information into a "daily report" template and generates the document.

[1042] 6. Check and correct documents

[1043] The user checks the generated daily report and makes corrections as necessary.

[1044] 7. Saving and sending documents

[1045] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[1046] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[1047] The processing flow will be explained below.

[1048] Step 1:

[1049] The user starts a dedicated application on the device (PC or smartphone) and selects voice input mode. When voice input mode is activated, the device prepares for voice recognition. A message is displayed to the user indicating that the device is ready.

[1050] Step 2:

[1051] The user initiates voice input, for example, saying, "Today's meeting discussed my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[1052] Step 3:

[1053] The collected voice data is converted into text data using voice recognition software on the device. The converted data might look something like, "During today's interview, we discussed my child's school life and home environment."

[1054] Step 4:

[1055] Text data is sent from the device to the server, which then begins analyzing it using natural language processing (NLP) algorithms.

[1056] Step 5:

[1057] The server analyzes the text data and extracts the necessary information, for example identifying key keywords and phrases in the text, such as "school life" and "home environment," and classifying each as an appropriate document item.

[1058] Step 6:

[1059] Based on the extracted information, the server fills in pre-set templates and automatically generates the necessary documents. For example, it fills in the "school life" and "home environment" sections in an "interview report" template.

[1060] Step 7:

[1061] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for checking and modifying the contents.

[1062] Step 8:

[1063] The user checks the document presented on the terminal and makes corrections as necessary, such as entering additional information in the "home environment" section.

[1064] Step 9:

[1065] When the user has finished checking and correcting the document, he or she issues an instruction to save the final document on the terminal, which then sends this instruction to the server.

[1066] Step 10:

[1067] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[1068] This is the specific flow of the program processing for this system. This processing significantly reduces the administrative burden of document preparation, allowing more resources to be allocated to practical work and responding to urgent cases.

[1069] Example 1

[1070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1071] In organizations that handle interpersonal services, employees are required to prepare numerous documents and perform data entry tasks, placing a heavy administrative burden on them. This makes it difficult for employees to concentrate on their primary duties, resulting in reduced work efficiency. The present invention aims to solve these problems and provide an environment in which employees can concentrate on their work.

[1072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1073] In this invention, the server includes means for initiating voice input, means for using a microphone built into the terminal to accept voice input, means for converting collected voice data into text data using voice recognition software, means for transmitting the text data to the server, means for executing a natural language processing algorithm to analyze the text data and extract necessary information, means for embedding information in a pre-set template based on the analyzed information to automatically generate a document, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after completion of the confirmation and correction. This allows users to efficiently create documents using voice input, reducing their administrative burden and allowing them to concentrate on their primary work.

[1074] "User" refers to the employee or staff member who uses the system to input voice data.

[1075] A "terminal" is a device that a user uses to input voice, and refers to hardware such as a PC or smartphone.

[1076] "Microphone" refers to an audio input device built into a terminal for collecting audio data.

[1077] "Speech recognition software" refers to software that converts collected voice data into text data, and is a program that executes a speech recognition algorithm.

[1078] "Text data" refers to character data converted by voice recognition software.

[1079] "Server" refers to a computer system that receives text data sent from a terminal, analyzes it, and generates documents.

[1080] "Natural language processing algorithm" refers to the algorithm used by the server to analyze text data and extract the required information.

[1081] "Document template" refers to a template for entering required information in a specific format.

[1082] "Document" refers to a document that is automatically generated based on analyzed information.

[1083] "Storage" refers to the act of recording the final document in a database or file system.

[1084] "Sending" refers to the operation of electronically sending a stored document to a designated recipient, such as a relevant department.

[1085] "Confirmation" refers to the operation of the user viewing the automatically generated document and checking its contents.

[1086] "Modification" refers to an operation in which a user changes the content of an automatically generated document.

[1087] This system aims to reduce the administrative burden in organizations that handle interpersonal work and provide an environment in which employees can concentrate on their work. This system is composed of three entities: users, terminals, and a server.

[1088] First, the user accesses the system using a device such as a PC or smartphone and activates the voice input mode. The user speaks into the device, for example, saying "Today's interview details." During this step, the device's built-in microphone collects the user's voice in real time.

[1089] The collected voice data is converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition software is installed on the device and runs an algorithm that converts the voice data into text data. The converted text data is then sent from the device to a server.

[1090] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The NLP algorithm extracts important information from the text data (e.g., "school life" and "home environment").

[1091] Next, the server automatically embeds the extracted information into a pre-defined document template. For example, the necessary information is added to a template such as an "interview report" or "daily report," and an automatically generated document is created. The generated document is then sent from the server to the terminal and presented to the user.

[1092] The user can review the document presented on the terminal and make corrections as necessary. An editing function is provided on the terminal, allowing the user to correct text and enter additional information. Once the document has been reviewed and corrected, it can be saved on the server as the final document by clicking the "Save" button. If necessary, the saved document can be sent to the relevant department.

[1093] Specific examples

[1094] Cases at child consultation centers

[1095] 1. Start voice input:

[1096] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[1097] 2. Audio data collection:

[1098] The terminal (PC) collects the following audio: "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[1099] 3. Sending text data:

[1100] The content converted into text data is sent to the server.

[1101] 4. Text data analysis:

[1102] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[1103] 5. Automatic generation of required documents:

[1104] The server embeds the information into the "interview report" template and generates the document.

[1105] 6. Document verification and correction:

[1106] The user checks the generated interview report and makes corrections as necessary.

[1107] 7. Save and send documents:

[1108] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[1109] Cases at nursery schools

[1110] 1. Start voice input:

[1111] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1112] 2. Audio data collection:

[1113] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[1114] 3. Sending text data:

[1115] The content converted into text data is sent to the server.

[1116] 4. Text data analysis:

[1117] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[1118] 5. Automatic generation of required documents:

[1119] The server embeds the information into a "daily report" template and generates the document.

[1120] 6. Document verification and correction:

[1121] The user checks the generated daily report and makes corrections as necessary.

[1122] 7. Save and send documents:

[1123] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[1124] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[1125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1126] Step 1:

[1127] The user accesses the system using a device such as a PC or smartphone and activates voice input mode. The user clicks the "voice input mode" button and begins speaking. The input voice data is collected in real time by the device's built-in microphone. Specifically, the user speaks "today's interview content" or "today's activities."

[1128] Input: User's voice

[1129] Output: Collected audio data

[1130] Step 2:

[1131] The device uses its built-in microphone to collect voice data, which is then sent to speech recognition software (e.g., Google Cloud Speech-to-Text) and converted into text. The speech recognition software then analyzes the voice data and generates corresponding text data.

[1132] Input: Audio data

[1133] Data processing: Converting voice data into character string data using voice recognition software

[1134] Output: Text data

[1135] Step 3:

[1136] The terminal transmits the converted text data to the server. Specifically, the terminal transmits the text data to the server using an HTTP request.

[1137] Input: Text data

[1138] Data processing: Sending text data to the server

[1139] Output: Text data sent to the server

[1140] Step 4:

[1141] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The server understands the context of the text data and extracts important information (e.g., "school life," "home environment," "activity participation," and "relationships with friends").

[1142] Input: Text data sent to the server

[1143] Data processing: Analyzing text data and extracting important information using natural language processing algorithms

[1144] Output: Extracted important information

[1145] Step 5:

[1146] The server automatically embeds the extracted information into pre-defined document templates. For example, necessary information is added to templates such as "interview report" and "daily report." This automatically generates documents.

[1147] Input: Extracted sensitive information

[1148] Data processing: Automatically embedding information into pre-defined document templates

[1149] Output: Auto-generated document

[1150] Step 6:

[1151] The server sends the generated document to the terminal. Specifically, it returns the document data in an HTTP response and launches an interface for displaying it on the terminal.

[1152] Input: Auto-generated document

[1153] Data processing: Sending document data to the terminal

[1154] Output: Document displayed on the device

[1155] Step 7:

[1156] The user checks the document received on the device and makes any necessary corrections. They adjust the content using the editing function provided on the device. For example, they might correct it to "Yuta participated very actively in the activities at the nursery school."

[1157] Input: Documents displayed on the device

[1158] Data processing: User confirmation and correction of documents

[1159] Output: The corrected document

[1160] Step 8:

[1161] After the user has completed the final confirmation, they click the "Save" button to save the document. The saved document will then be automatically sent to the relevant departments as needed. Specifically, the document will be saved to the server and sent by email to the specified recipient.

[1162] Input: Corrected document

[1163] Data processing: Document storage and transmission to relevant departments

[1164] Output: Final document saved and sent

[1165] (Application example 1)

[1166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1167] Autonomous vehicles require efficient reception of passenger requests and feedback, and the ability to compile appropriate reports and documents. However, conventional systems have limitations in the accuracy and speed of voice input, which is particularly problematic in moving autonomous vehicles. Furthermore, when passengers communicate their destinations and en-route requirements in real time, the time and effort required to accurately summarize and present that information presents a problem.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1169] In this invention, the server includes means for receiving voice input using a voice input device installed inside the vehicle, means for analyzing information including destinations and requests based on the voice input, and means for automatically generating a report for passengers. This allows passengers' voice input to be collected in real time, the content of the analysis to be analyzed quickly and accurately, and, after confirmation by the passengers themselves, smooth support within the autonomous vehicle.

[1170] The "voice input device" is a device that is installed inside a vehicle and collects passenger voices in real time.

[1171] The "means for converting into text data" refers to software or algorithms for analyzing collected voice data and converting it into data as a string of characters.

[1172] "Analysis means" refers to algorithms or programs for extracting necessary information based on text data.

[1173] The "automatic report generation means" is a function that automatically creates a report in a preset format based on the analyzed information.

[1174] "User presentation means" refers to a display device or interface that allows passengers to check automatically generated documents and reports.

[1175] "Verification and correction means" means an interface or tool that allows passengers to review and, if necessary, correct the documents and reports provided to them.

[1176] "Document storage and transmission means" is a function for storing final reports and documents and transmitting them to relevant departments as necessary.

[1177] A "passenger" is a person riding in an autonomous vehicle and is the subject of voice input.

[1178] "Destination" means the passenger's designated final destination.

[1179] "Requirements" are specific details such as the services that passengers would like to receive from the self-driving vehicle or requests for detours.

[1180] A "server" is a computer system that receives data sent from a voice input device and analyzes and processes it.

[1181] This invention relates to a system for collecting and analyzing passenger voice input in real time in an autonomous vehicle, and automatically generating an appropriate report based on the necessary information. Specific embodiments of this system will be described below.

[1182] This system consists of three main components: a voice input device, a server, and a user. The voice input device is built into the interior of the autonomous vehicle and collects passengers' voices in real time. The collected voice data is then converted into text data using speech recognition software (such as the Google Cloud Speech-to-Text API) on the device.

[1183] The converted text data is sent to a server via a communication device in the autonomous vehicle. The server analyzes the received text data and extracts the necessary information using a natural language processing (NLP) algorithm (such as Google NLP API). This analysis identifies the passenger's requirements, such as the destination and any detours they may wish to make along the way.

[1184] The server automatically generates a report based on the identified information in a pre-defined format (such as an HTML or PDF template). This report is displayed on a display when the passenger exits the autonomous vehicle or at a specified time. The passenger can review the report and make any necessary corrections. The final report is saved after the passenger has been identified and sent to the relevant departments as needed.

[1185] For example, if a passenger speaks into an in-car microphone, "I'm heading to my destination, Shinjuku Station, now. I'd like to stop at a coffee shop on the way," this voice input is collected in real time. The voice data is quickly converted into text data and sent to the server. The server extracts keywords such as "Shinjuku Station" and "coffee shop," and based on this, identifies the destination and desired detours. The generated report is displayed on the in-car display for passengers to review before arriving at Shinjuku Station. After passengers make any necessary corrections, the final report is saved in PDF format.

[1186] Examples of prompts are:

[1187] "Good morning. Today you are riding in a self-driving vehicle. Your destination is Shinjuku Station, and we'd like you to stop at a coffee shop along the way. Please also write this article as a report to provide feedback on your riding experience."

[1188] This system allows passengers to communicate their requests and wishes using only voice input, which the autonomous vehicle's system can then instantly analyze, process, and generate reports on, providing a smooth and comfortable riding experience.

[1189] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1190] Step 1:

[1191] The user speaks their destination and requirements into a voice input device inside the autonomous vehicle, which collects this voice data in real time.

[1192] Input: User's voice

[1193] Output: Audio data

[1194] Step 2:

[1195] The device converts the collected voice data into text data using voice recognition software (Google Cloud Speech-to-Text API).

[1196] Input: Audio data

[1197] Output: Text data

[1198] Step 3:

[1199] The converted text data is sent from the terminal to the server, and the terminal transfers the data using an HTTP request.

[1200] Input: Text data

[1201] Output: Text data sent to the server

[1202] Step 4:

[1203] The server analyzes the received text data and extracts the necessary information using natural language processing (NLP) algorithms (Google NLP API).

[1204] Input: Received text data

[1205] Output: Extracted information (destination, requirements, etc.)

[1206] Step 5:

[1207] The server automatically generates a report based on the extracted information in a predefined format, such as an HTML or PDF template file.

[1208] Input: Extracted information

[1209] Output: Auto-generated report (a file containing the information you have embedded in the report template)

[1210] Step 6:

[1211] The server sends the generated report to a display in the autonomous vehicle, where the user can view the displayed report.

[1212] Input: Generated report

[1213] Output: Report displayed on the display

[1214] Step 7:

[1215] The user checks the report and makes corrections if necessary through the display. The terminal receives input from the user and sends the corrections to the server.

[1216] Input: User correction instructions

[1217] Output: Corrected data

[1218] Step 8:

[1219] The server receives the user's correction instructions, saves the final report, and sends it to the relevant departments as needed, for example, in cloud storage or an internal database.

[1220] Input: Corrected data and final confirmed report

[1221] Output: Final saved report and submitted report

[1222] The above are the processing steps of this system.

[1223] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1224] This invention is a system that reduces the administrative burden at institutions that handle interpersonal work, such as child consultation centers and nurseries, and provides an environment where staff can concentrate on their work. Furthermore, by combining it with an emotion engine that recognizes and analyzes user emotions, it enables flexible document creation that responds to the user's emotions. A specific embodiment of this system is shown below.

[1225] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[1226] The converted text data is sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[1227] Furthermore, the app is equipped with an emotion engine that analyzes the user's emotions from voice input. The emotion engine extracts emotion data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results. For example, if the user is feeling stressed, that emotion data is also retained as an analysis result and reflected in the document.

[1228] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[1229] Specific examples

[1230] Cases at child consultation centers

[1231] 1. Start voice input

[1232] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[1233] 2. Collection of audio data

[1234] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[1235] 3. Sending text data and emotion data

[1236] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[1237] 4. Analysis of text data and reflection of emotional data

[1238] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[1239] 5. Automatic generation of required documents

[1240] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[1241] 6. Check and correct documents

[1242] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[1243] 7. Saving and sending documents

[1244] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[1245] Cases at nursery schools

[1246] 1. Start voice input

[1247] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1248] 2. Collection of audio data

[1249] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[1250] 3. Sending text data and emotion data

[1251] Text data and emotion data are sent from the smartphone to a server.

[1252] 4. Analysis of text data and reflection of emotional data

[1253] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[1254] 5. Automatic generation of required documents

[1255] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[1256] 6. Check and correct documents

[1257] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[1258] 7. Saving and sending documents

[1259] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[1260] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[1261] The processing flow will be explained below.

[1262] Step 1:

[1263] The user starts a dedicated application on their device (PC or smartphone) and selects voice input mode. The device prepares for voice recognition and emotion analysis, and displays a message to the user that it is ready.

[1264] Step 2:

[1265] The user begins voice input, for example, saying, "Today's meeting was about my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[1266] Step 3:

[1267] The collected voice data is converted into text data using voice recognition software within the device. For example, the voice is converted into text data such as, "Today's interview content was a discussion about the child's school life and home environment."

[1268] Step 4:

[1269] The device uses the collected voice data to analyze the user's emotions using a built-in emotion engine, which takes into account voice characteristics such as tone, speed, and strength.

[1270] Step 5:

[1271] The converted text data and analyzed emotion data are sent from the device to the server, which receives the data and begins analyzing it.

[1272] Step 6:

[1273] The server analyzes the text data and extracts the necessary information. For example, it identifies key keywords and phrases in the text, such as "school life" and "home environment," and classifies each as an appropriate document item.

[1274] Step 7:

[1275] Based on the extracted information, the server embeds the data into pre-set templates and automatically generates the necessary documents. For example, it embeds the sections on "school life" and "home environment" into an "interview report" template.

[1276] Step 8:

[1277] The server also incorporates the emotional data analyzed by the emotion engine into the generated document. For example, if the user is feeling stressed, the emotional data is noted in the report.

[1278] Step 9:

[1279] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for reviewing and modifying the contents.

[1280] Step 10:

[1281] The user checks the document presented on the terminal and makes corrections as necessary. For example, they can enter additional information in the "home environment" section. The content of the emotional data can also be checked.

[1282] Step 11:

[1283] After the user has finished checking and correcting the document, he / she instructs the terminal to save the final document, and the terminal sends this instruction to the server.

[1284] Step 12:

[1285] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[1286] This specific processing step streamlines the process of creating various documents and generates rich reports that reflect the user's emotional state, allowing employees and staff to spend more time on their core tasks.

[1287] Example 2

[1288] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1289] In organizations that handle interpersonal services, employees and staff are required to create numerous documents, which takes time and effort, making it difficult for them to concentrate on their work. Another issue is that the user's emotions are not reflected in the documents, making it difficult to create documents that take into account the user's emotional state.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1291] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, means for analyzing the text data and extracting necessary information, means for automatically generating a document in a preset format based on the analyzed information, means for analyzing a user's emotions from the voice input, means for reflecting the emotional information in the document based on the analyzed emotional data, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after confirmation and correction are complete. This makes it possible to streamline document creation, allowing employees and staff to concentrate on their work, and further realizing flexible document creation that takes the user's emotions into consideration.

[1292] "Means for accepting voice input" refers to an interface or device that allows a user to input voice.

[1293] "Means for converting voice input into text data" refers to the process or software that uses voice recognition technology to convert voice data into text information (text data).

[1294] "Means for analyzing text data and extracting necessary information" refers to processes and algorithms that use natural language processing techniques to identify and extract relevant information from text data.

[1295] "Means for automatically generating documents according to pre-defined formats" refers to a process or system that uses pre-defined templates to automatically create documents based on collected and analyzed information.

[1296] "Means for analyzing user emotions from voice input" refers to technologies and algorithms that analyze a user's voice tone and paralinguistic information to identify emotions.

[1297] "Means for reflecting emotional information in a document based on analyzed emotional data" refers to a technique or method for adjusting the content and format of a generated document in consideration of the analyzed emotional data.

[1298] "Means for presenting the automatically generated document to the user and for reviewing and correcting it" refers to an interface or tool that displays the generated document to the user and allows the user to review its contents and edit them as necessary.

[1299] "Means for saving and sending the final document after confirmation and correction is completed" refers to a system or process that has the function of saving the document after the user has confirmed and corrected it and sending it to the specified destination.

[1300] This invention is a system that reduces the administrative burden at institutions that handle interpersonal tasks, such as child consultation centers and nurseries, and provides an environment where employees can concentrate on their work. By combining this system with an emotion engine that recognizes and analyzes the user's emotions, this system realizes flexible document creation that responds to the user's emotions.

[1301] This system consists of three entities: the user, the terminal, and the server. The roles of each entity and their specific processing methods are described below.

[1302] First, a user accesses the system using a terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API) within the terminal.

[1303] The converted text data is sent from the device to a server, which then analyzes the received text data and runs natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) to extract the required information.

[1304] The information extracted by the server is automatically embedded in a pre-set document template, which automatically generates the target document (application form, report, daily report, etc.). Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) is built in to analyze the user's emotions from voice input. The emotion engine extracts emotional data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results.

[1305] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[1306] Specific examples

[1307] Cases at child consultation centers

[1308] 1. Start voice input

[1309] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[1310] 2. Collection of audio data

[1311] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[1312] 3. Sending text data and emotion data

[1313] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[1314] 4. Analysis of text data and reflection of emotional data

[1315] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[1316] 5. Automatic generation of required documents

[1317] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[1318] 6. Check and correct documents

[1319] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[1320] 7. Saving and sending documents

[1321] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[1322] Cases at nursery schools

[1323] 1. Start voice input

[1324] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1325] 2. Collection of audio data

[1326] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[1327] 3. Sending text data and emotion data

[1328] Text data and emotion data are sent from the smartphone to a server.

[1329] 4. Analysis of text data and reflection of emotional data

[1330] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[1331] 5. Automatic generation of required documents

[1332] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[1333] 6. Check and correct documents

[1334] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[1335] 7. Saving and sending documents

[1336] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[1337] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[1338] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1339] Step 1:

[1340] Start voice input

[1341] The user starts speech input mode using the terminal. The user types the command "Start speech input."

[1342] Specific actions

[1343] The user starts the dedicated application and selects the voice input mode.

[1344] The application activates the built-in microphone and prepares for voice input.

[1345] input

[1346] Spoken commands from the user

[1347] output

[1348] Voice input mode is activated

[1349] Step 2:

[1350] Audio data collection

[1351] The device uses a built-in microphone to collect the user's voice input in real time, and the voice data is temporarily stored in a buffer.

[1352] Specific actions

[1353] A microphone captures the user's voice.

[1354] The audio data is stored in a buffer as digital data at a constant sampling rate.

[1355] input

[1356] User voice data

[1357] output

[1358] Temporarily stored audio data in a buffer

[1359] Step 3:

[1360] Audio data conversion

[1361] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the voice data in the buffer into text data.

[1362] Specific actions

[1363] The speech recognition software retrieves the speech data from the buffer.

[1364] The voice data is converted into text data through phonemic analysis and text conversion algorithms.

[1365] input

[1366] Buffered audio data

[1367] output

[1368] Text data

[1369] Step 4:

[1370] Emotional Data Analysis

[1371] The device uses an emotion engine to extract emotion data from the user's voice tone, speed, strength, etc.

[1372] Specific actions

[1373] The emotion engine analyzes the voice data and extracts voice features.

[1374] Emotional states such as stress and joy are identified from the extracted features.

[1375] input

[1376] Audio data

[1377] output

[1378] Emotional Data

[1379] Step 5:

[1380] Sending data

[1381] The device sends text data and emotion data to the server, generates an HTTP request, and sends it to the endpoint.

[1382] Specific actions

[1383] Format text data and emotion data into JSON format.

[1384] Constructs an HTTP request and sends it to the specified endpoint on the server.

[1385] input

[1386] Text data, emotion data

[1387] output

[1388] Data sent to the server

[1389] Step 6:

[1390] Text data analysis

[1391] The server analyzes the received text data using natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) and extracts the necessary information.

[1392] Specific actions

[1393] Natural language processing algorithms break down the text data into individual sentences.

[1394] Extract specific keywords or phrases.

[1395] input

[1396] Text data

[1397] output

[1398] Extracted information

[1399] Step 7:

[1400] Reflecting emotional data

[1401] The server adds emotional information to the extracted information based on the analyzed emotional data.

[1402] Specific actions

[1403] Integrate emotion data into analysis results.

[1404] Generate composite information.

[1405] input

[1406] Text data analysis results, emotion data

[1407] output

[1408] Complex information including emotional information

[1409] Step 8:

[1410] Automatic generation of required documents

[1411] The server embeds the composite information into a pre-defined document template and automatically generates the document.

[1412] Specific actions

[1413] Use a template engine to embed the extracted information into a template.

[1414] Automatically generate documents according to the format.

[1415] input

[1416] Complex information including emotional information

[1417] output

[1418] Generated documents

[1419] Step 9:

[1420] Document review and correction

[1421] The generated document is sent from the server to the terminal and presented to the user, who can then check the document and make any necessary corrections.

[1422] Specific actions

[1423] The server returns the document as an HTTP response.

[1424] The device displays the document and the user uses editing tools to modify the content.

[1425] input

[1426] Generated documents

[1427] output

[1428] Documents modified by users

[1429] Step 10:

[1430] Saving and sending documents

[1431] After the user completes the final confirmation, they save the document and send it to the relevant department if necessary.

[1432] Specific actions

[1433] The terminal sends the modified document to the server.

[1434] The server stores the document in a database and sends it to the specified destination.

[1435] input

[1436] Documents modified by users

[1437] output

[1438] Final documents saved and sent

[1439] (Application example 2)

[1440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] In current interpersonal services, when staff manually create documents, they often experience problems such as reduced work efficiency and difficulty concentrating on their work. Furthermore, conventional systems have difficulty understanding users' emotions and responding appropriately, which can lead to a decline in the quality of service. To solve these issues, a system that combines voice input and emotion analysis is needed.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1443] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, and means for automatically generating documents based on the analyzed information, thereby enabling personnel performing interpersonal tasks to quickly create documents using voice, and further enabling emotional data to be analyzed and reflected in the documents.

[1444] "Voice input" refers to voice data emitted by a user through a microphone.

[1445] "Text data" refers to data that has been converted from voice input into a character string format.

[1446] "Analysis" refers to the process of extracting the intended meaning and necessary information from text data.

[1447] "Format" refers to a pre-defined template used to generate a document.

[1448] "Automatic generation" refers to the process by which the system automatically creates documents based on analyzed information.

[1449] "Presenting" refers to showing the generated document to the user.

[1450] "Modification" refers to the user changing the content of the presented document as needed.

[1451] "Preservation" refers to the recording of the final document in digital or physical form.

[1452] "Transmission" refers to the electronic transfer of the final document to the relevant department or party.

[1453] "Emotion data" refers to data that represents the user's emotional state at the time of voice input.

[1454] "Emotion analysis" refers to the process of analyzing a user's emotions based on the tone, speed, and strength of voice input.

[1455] The system of the present invention is composed of three main components: a user, a terminal, and a server, for quickly creating documents using voice input. The details of each component and their operation are explained below.

[1456] First, a user accesses the system using a device (e.g., a smartphone or tablet). The user activates the device's voice input mode and inputs voice data through the microphone. This voice input is converted into text data in real time using speech recognition software in the device (e.g., Python's speech_recognition library).

[1457] The converted text data is sent from the device to a server. The server receives the text data and analyzes it using natural language processing (NLP) algorithms. The analysis process involves extracting necessary information from the text data. The server also has a built-in emotion engine that analyzes emotional data in voice input, extracting emotional data from the user's voice tone, speed, and strength.

[1458] The analysis results are automatically embedded into a document based on a pre-defined document format. This document can be used, for example, for service guides or proposals. The generated document is then resent from the server to the terminal, where the user can check and modify it. After the user completes the final check, the document is saved and sent to the relevant department.

[1459] A concrete example would be an application where a concierge at a luxury hotel or department store uses a smartphone to input conversations and questions with customers by voice, and then performs sentiment analysis. For example, if the concierge says, "The customer says he wants to relax. I'd like to recommend some spa services," the application converts the speech into text and sends it to a server. The server analyzes the data and automatically generates a proposal for a spa service that matches the customer's relaxation needs.

[1460] Here are some examples of relevant prompts:

[1461] "Your client says they want to relax. Please create a spa service proposal based on the following information:\n\n{Input data about your client's needs}\n\n---\n\nProposal:\n"

[1462] This system allows staff who handle interpersonal tasks to quickly create documents using voice commands and can analyze emotional data and reflect it in the documents, thereby improving the quality of service and realizing efficient work execution.

[1463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1464] Step 1:

[1465] The user activates the voice input mode of the terminal and inputs voice data. The input voice data is collected through a microphone. At this point, the input is the user's voice, and the output is digital voice data.

[1466] Step 2:

[1467] The voice data collected by the device is converted into text data in real time using voice recognition software (for example, Python's speech_recognition library). The input is voice data and the output is text data. In this step, the voice data is converted into a string.

[1468] Step 3:

[1469] The terminal sends the converted text data to the server. The input is text data, and the output is data sent to the server. Data communication takes place in this step.

[1470] Step 4:

[1471] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is text data, and the output is analyzed information. In this step, the necessary information is extracted from the text data.

[1472] Step 5:

[1473] The server extracts the user's emotional data using an emotion engine that analyzes the emotional data from the voice input. The input is voice data, and the output is emotional data. In this step, data analysis is performed on the voice tone, speed, and strength.

[1474] Step 6:

[1475] The server automatically generates a document based on the analyzed information and emotion data according to a pre-set document format. The input is the analyzed information and emotion data, and the output is an automatically generated document. This step involves the information embedding process.

[1476] Step 7:

[1477] The server sends the automatically generated document to the terminal. The input is the automatically generated document, and the output is the data sent to the terminal. In this step, data communication occurs again.

[1478] Step 8:

[1479] The user checks the document received on the terminal and makes corrections if necessary. The input is the automatically generated document, and the output is the corrected document. This is the step where user interaction takes place.

[1480] Step 9:

[1481] After the user has confirmed and corrected the final document, it is saved on the terminal and sent to the relevant department. The input is the corrected document, and the output is the saved and sent data. In this step, the document is saved and the data is finally sent.

[1482] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1483] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1484] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1485] [Fourth embodiment]

[1486] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1487] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1488] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1489] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1490] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1491] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1492] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1493] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1494] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1495] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1496] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1497] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1498] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1499] The present invention is a system for reducing the administrative burden at child consultation centers, nurseries, and other institutions that provide interpersonal services, and providing an environment in which they can concentrate on their work. A specific embodiment of this system is described below.

[1500] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[1501] The converted text data is then sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[1502] The generated document is sent from the server to the terminal and presented to the user. The user can check the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished checking and correcting, the final document is saved and sent to the relevant department as necessary.

[1503] Specific examples

[1504] Cases at child consultation centers

[1505] 1. Start voice input

[1506] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[1507] 2. Collection of audio data

[1508] The device collects audio that says, "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[1509] 3. Sending text data

[1510] The content converted into text data is sent to the server.

[1511] 4. Text Data Analysis

[1512] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[1513] 5. Automatic generation of required documents

[1514] The server embeds the information into the "interview report" template and generates the document.

[1515] 6. Check and correct documents

[1516] The user checks the generated interview report and makes corrections as necessary.

[1517] 7. Saving and sending documents

[1518] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[1519] Cases at nursery schools

[1520] 1. Start voice input

[1521] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1522] 2. Collection of audio data

[1523] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[1524] 3. Sending text data

[1525] The content converted into text data is sent to the server.

[1526] 4. Text Data Analysis

[1527] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[1528] 5. Automatic generation of required documents

[1529] The server embeds the information into a "daily report" template and generates the document.

[1530] 6. Check and correct documents

[1531] The user checks the generated daily report and makes corrections as necessary.

[1532] 7. Saving and sending documents

[1533] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[1534] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[1535] The processing flow will be explained below.

[1536] Step 1:

[1537] The user starts a dedicated application on the device (PC or smartphone) and selects voice input mode. When voice input mode is activated, the device prepares for voice recognition. A message is displayed to the user indicating that the device is ready.

[1538] Step 2:

[1539] The user initiates voice input, for example, saying, "Today's meeting discussed my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[1540] Step 3:

[1541] The collected voice data is converted into text data using voice recognition software on the device. The converted data might look something like, "During today's interview, we discussed my child's school life and home environment."

[1542] Step 4:

[1543] Text data is sent from the device to the server, which then begins analyzing it using natural language processing (NLP) algorithms.

[1544] Step 5:

[1545] The server analyzes the text data and extracts the necessary information, for example identifying key keywords and phrases in the text, such as "school life" and "home environment," and classifying each as an appropriate document item.

[1546] Step 6:

[1547] Based on the extracted information, the server fills in pre-set templates and automatically generates the necessary documents. For example, it fills in the "school life" and "home environment" sections in an "interview report" template.

[1548] Step 7:

[1549] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for checking and modifying the contents.

[1550] Step 8:

[1551] The user checks the document presented on the terminal and makes corrections as necessary, such as entering additional information in the "home environment" section.

[1552] Step 9:

[1553] When the user has finished checking and correcting the document, he or she issues an instruction to save the final document on the terminal, which then sends this instruction to the server.

[1554] Step 10:

[1555] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[1556] This is the specific flow of the program processing for this system. This processing significantly reduces the administrative burden of document preparation, allowing more resources to be allocated to practical work and responding to urgent cases.

[1557] Example 1

[1558] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1559] In organizations that handle interpersonal services, employees are required to prepare numerous documents and perform data entry tasks, placing a heavy administrative burden on them. This makes it difficult for employees to concentrate on their primary duties, resulting in reduced work efficiency. The present invention aims to solve these problems and provide an environment in which employees can concentrate on their work.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1561] In this invention, the server includes means for initiating voice input, means for using a microphone built into the terminal to accept voice input, means for converting collected voice data into text data using voice recognition software, means for transmitting the text data to the server, means for executing a natural language processing algorithm to analyze the text data and extract necessary information, means for embedding information in a pre-set template based on the analyzed information to automatically generate a document, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after completion of the confirmation and correction. This allows users to efficiently create documents using voice input, reducing their administrative burden and allowing them to concentrate on their primary work.

[1562] "User" refers to the employee or staff member who uses the system to input voice data.

[1563] A "terminal" is a device that a user uses to input voice, and refers to hardware such as a PC or smartphone.

[1564] "Microphone" refers to an audio input device built into a terminal for collecting audio data.

[1565] "Speech recognition software" refers to software that converts collected voice data into text data, and is a program that executes a speech recognition algorithm.

[1566] "Text data" refers to character data converted by voice recognition software.

[1567] "Server" refers to a computer system that receives text data sent from a terminal, analyzes it, and generates documents.

[1568] "Natural language processing algorithm" refers to the algorithm used by the server to analyze text data and extract the required information.

[1569] "Document template" refers to a template for entering required information in a specific format.

[1570] "Document" refers to a document that is automatically generated based on analyzed information.

[1571] "Storage" refers to the act of recording the final document in a database or file system.

[1572] "Sending" refers to the operation of electronically sending a stored document to a designated recipient, such as a relevant department.

[1573] "Confirmation" refers to the operation of the user viewing the automatically generated document and checking its contents.

[1574] "Modification" refers to an operation in which a user changes the content of an automatically generated document.

[1575] This system aims to reduce the administrative burden in organizations that handle interpersonal work and provide an environment in which employees can concentrate on their work. This system is composed of three entities: users, terminals, and a server.

[1576] First, the user accesses the system using a device such as a PC or smartphone and activates the voice input mode. The user speaks into the device, for example, saying "Today's interview details." During this step, the device's built-in microphone collects the user's voice in real time.

[1577] The collected voice data is converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition software is installed on the device and runs an algorithm that converts the voice data into text data. The converted text data is then sent from the device to a server.

[1578] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The NLP algorithm extracts important information from the text data (e.g., "school life" and "home environment").

[1579] Next, the server automatically embeds the extracted information into a pre-defined document template. For example, the necessary information is added to a template such as an "interview report" or "daily report," and an automatically generated document is created. The generated document is then sent from the server to the terminal and presented to the user.

[1580] The user can review the document presented on the terminal and make corrections as necessary. An editing function is provided on the terminal, allowing the user to correct text and enter additional information. Once the document has been reviewed and corrected, it can be saved on the server as the final document by clicking the "Save" button. If necessary, the saved document can be sent to the relevant department.

[1581] Specific examples

[1582] Cases at child consultation centers

[1583] 1. Start voice input:

[1584] The user (a staff member at a child consultation center) activates the voice input mode on a PC and speaks the "contents of today's consultation."

[1585] 2. Audio data collection:

[1586] The terminal (PC) collects the following audio: "During today's interview, we discussed my child's current school life. We also checked on his home environment."

[1587] 3. Sending text data:

[1588] The content converted into text data is sent to the server.

[1589] 4. Text data analysis:

[1590] The server analyzes information on "school life" and "home environment" and extracts the necessary items.

[1591] 5. Automatic generation of required documents:

[1592] The server embeds the information into the "interview report" template and generates the document.

[1593] 6. Document verification and correction:

[1594] The user checks the generated interview report and makes corrections as necessary.

[1595] 7. Save and send documents:

[1596] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the relevant department.

[1597] Cases at nursery schools

[1598] 1. Start voice input:

[1599] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1600] 2. Audio data collection:

[1601] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends."

[1602] 3. Sending text data:

[1603] The content converted into text data is sent to the server.

[1604] 4. Text data analysis:

[1605] The server analyzes information such as "activity participation status" and "relationships with friends" and extracts the necessary items.

[1606] 5. Automatic generation of required documents:

[1607] The server embeds the information into a "daily report" template and generates the document.

[1608] 6. Document verification and correction:

[1609] The user checks the generated daily report and makes corrections as necessary.

[1610] 7. Save and send documents:

[1611] Once the user has completed the final confirmation, they click the "Save" button to save the document and send it to the parent.

[1612] This system will streamline the creation of various documents, allowing employees and staff to devote more time to their core duties.

[1613] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1614] Step 1:

[1615] The user accesses the system using a device such as a PC or smartphone and activates voice input mode. The user clicks the "voice input mode" button and begins speaking. The input voice data is collected in real time by the device's built-in microphone. Specifically, the user speaks "today's interview content" or "today's activities."

[1616] Input: User's voice

[1617] Output: Collected audio data

[1618] Step 2:

[1619] The device uses its built-in microphone to collect voice data, which is then sent to speech recognition software (e.g., Google Cloud Speech-to-Text) and converted into text. The speech recognition software then analyzes the voice data and generates corresponding text data.

[1620] Input: Audio data

[1621] Data processing: Converting voice data into character string data using voice recognition software

[1622] Output: Text data

[1623] Step 3:

[1624] The terminal transmits the converted text data to the server. Specifically, the terminal transmits the text data to the server using an HTTP request.

[1625] Input: Text data

[1626] Data processing: Sending text data to the server

[1627] Output: Text data sent to the server

[1628] Step 4:

[1629] The server runs a natural language processing (NLP) algorithm (e.g., OpenAI GPT-4) to analyze the received text data. The server understands the context of the text data and extracts important information (e.g., "school life," "home environment," "activity participation," and "relationships with friends").

[1630] Input: Text data sent to the server

[1631] Data processing: Analyzing text data and extracting important information using natural language processing algorithms

[1632] Output: Extracted important information

[1633] Step 5:

[1634] The server automatically embeds the extracted information into pre-defined document templates. For example, necessary information is added to templates such as "interview report" and "daily report." This automatically generates documents.

[1635] Input: Extracted sensitive information

[1636] Data processing: Automatically embedding information into pre-defined document templates

[1637] Output: Auto-generated document

[1638] Step 6:

[1639] The server sends the generated document to the terminal. Specifically, it returns the document data in an HTTP response and launches an interface for displaying it on the terminal.

[1640] Input: Auto-generated document

[1641] Data processing: Sending document data to the terminal

[1642] Output: Document displayed on the device

[1643] Step 7:

[1644] The user checks the document received on the device and makes any necessary corrections. They adjust the content using the editing function provided on the device. For example, they might correct it to "Yuta participated very actively in the activities at the nursery school."

[1645] Input: Documents displayed on the device

[1646] Data processing: User confirmation and correction of documents

[1647] Output: The corrected document

[1648] Step 8:

[1649] After the user has completed the final confirmation, they click the "Save" button to save the document. The saved document will then be automatically sent to the relevant departments as needed. Specifically, the document will be saved to the server and sent by email to the specified recipient.

[1650] Input: Corrected document

[1651] Data processing: Document storage and transmission to relevant departments

[1652] Output: Final document saved and sent

[1653] (Application example 1)

[1654] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] Autonomous vehicles require efficient reception of passenger requests and feedback, and the ability to compile appropriate reports and documents. However, conventional systems have limitations in the accuracy and speed of voice input, which is particularly problematic in moving autonomous vehicles. Furthermore, when passengers communicate their destinations and en-route requirements in real time, the time and effort required to accurately summarize and present that information presents a problem.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1657] In this invention, the server includes means for receiving voice input using a voice input device installed inside the vehicle, means for analyzing information including destinations and requests based on the voice input, and means for automatically generating a report for passengers. This allows passengers' voice input to be collected in real time, the content of the analysis to be analyzed quickly and accurately, and, after confirmation by the passengers themselves, smooth support within the autonomous vehicle.

[1658] The "voice input device" is a device that is installed inside a vehicle and collects passenger voices in real time.

[1659] The "means for converting into text data" refers to software or algorithms for analyzing collected voice data and converting it into data as a string of characters.

[1660] "Analysis means" refers to algorithms or programs for extracting necessary information based on text data.

[1661] The "automatic report generation means" is a function that automatically creates a report in a preset format based on the analyzed information.

[1662] "User presentation means" refers to a display device or interface that allows passengers to check automatically generated documents and reports.

[1663] "Verification and correction means" means an interface or tool that allows passengers to review and, if necessary, correct the documents and reports provided to them.

[1664] "Document storage and transmission means" is a function for storing final reports and documents and transmitting them to relevant departments as necessary.

[1665] A "passenger" is a person riding in an autonomous vehicle and is the subject of voice input.

[1666] "Destination" means the passenger's designated final destination.

[1667] "Requirements" are specific details such as the services that passengers would like to receive from the self-driving vehicle or requests for detours.

[1668] A "server" is a computer system that receives data sent from a voice input device and analyzes and processes it.

[1669] This invention relates to a system for collecting and analyzing passenger voice input in real time in an autonomous vehicle, and automatically generating an appropriate report based on the necessary information. Specific embodiments of this system will be described below.

[1670] This system consists of three main components: a voice input device, a server, and a user. The voice input device is built into the interior of the autonomous vehicle and collects passengers' voices in real time. The collected voice data is then converted into text data using speech recognition software (such as the Google Cloud Speech-to-Text API) on the device.

[1671] The converted text data is sent to a server via a communication device in the autonomous vehicle. The server analyzes the received text data and extracts the necessary information using a natural language processing (NLP) algorithm (such as Google NLP API). This analysis identifies the passenger's requirements, such as the destination and any detours they may wish to make along the way.

[1672] The server automatically generates a report based on the identified information in a pre-defined format (such as an HTML or PDF template). This report is displayed on a display when the passenger exits the autonomous vehicle or at a specified time. The passenger can review the report and make any necessary corrections. The final report is saved after the passenger has been identified and sent to the relevant departments as needed.

[1673] For example, if a passenger speaks into an in-car microphone, "I'm heading to my destination, Shinjuku Station, now. I'd like to stop at a coffee shop on the way," this voice input is collected in real time. The voice data is quickly converted into text data and sent to the server. The server extracts keywords such as "Shinjuku Station" and "coffee shop," and based on this, identifies the destination and desired detours. The generated report is displayed on the in-car display for passengers to review before arriving at Shinjuku Station. After passengers make any necessary corrections, the final report is saved in PDF format.

[1674] Examples of prompts are:

[1675] "Good morning. Today you are riding in a self-driving vehicle. Your destination is Shinjuku Station, and we'd like you to stop at a coffee shop along the way. Please also write this article as a report to provide feedback on your riding experience."

[1676] This system allows passengers to communicate their requests and wishes using only voice input, which the autonomous vehicle's system can then instantly analyze, process, and generate reports on, providing a smooth and comfortable riding experience.

[1677] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1678] Step 1:

[1679] The user speaks their destination and requirements into a voice input device inside the autonomous vehicle, which collects this voice data in real time.

[1680] Input: User's voice

[1681] Output: Audio data

[1682] Step 2:

[1683] The device converts the collected voice data into text data using voice recognition software (Google Cloud Speech-to-Text API).

[1684] Input: Audio data

[1685] Output: Text data

[1686] Step 3:

[1687] The converted text data is sent from the terminal to the server, and the terminal transfers the data using an HTTP request.

[1688] Input: Text data

[1689] Output: Text data sent to the server

[1690] Step 4:

[1691] The server analyzes the received text data and extracts the necessary information using natural language processing (NLP) algorithms (Google NLP API).

[1692] Input: Received text data

[1693] Output: Extracted information (destination, requirements, etc.)

[1694] Step 5:

[1695] The server automatically generates a report based on the extracted information in a predefined format, such as an HTML or PDF template file.

[1696] Input: Extracted information

[1697] Output: Auto-generated report (a file containing the information you have embedded in the report template)

[1698] Step 6:

[1699] The server sends the generated report to a display in the autonomous vehicle, where the user can view the displayed report.

[1700] Input: Generated report

[1701] Output: Report displayed on the display

[1702] Step 7:

[1703] The user checks the report and makes corrections if necessary through the display. The terminal receives input from the user and sends the corrections to the server.

[1704] Input: User correction instructions

[1705] Output: Corrected data

[1706] Step 8:

[1707] The server receives the user's correction instructions, saves the final report, and sends it to the relevant departments as needed, for example, in cloud storage or an internal database.

[1708] Input: Corrected data and final confirmed report

[1709] Output: Final saved report and submitted report

[1710] The above are the processing steps of this system.

[1711] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1712] This invention is a system that reduces the administrative burden at institutions that handle interpersonal work, such as child consultation centers and nurseries, and provides an environment where staff can concentrate on their work. Furthermore, by combining it with an emotion engine that recognizes and analyzes user emotions, it enables flexible document creation that responds to the user's emotions. A specific embodiment of this system is shown below.

[1713] This system consists of three main components: the user, the terminal, and the server. First, the user accesses the system using the terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software within the terminal.

[1714] The converted text data is sent from the device to a server. The server analyzes the received text data and runs a natural language processing (NLP) algorithm to extract the necessary information. The information extracted by the server is automatically embedded into a pre-defined document template. This automatically generates the target document (application form, report, daily report, etc.).

[1715] Furthermore, the app is equipped with an emotion engine that analyzes the user's emotions from voice input. The emotion engine extracts emotion data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results. For example, if the user is feeling stressed, that emotion data is also retained as an analysis result and reflected in the document.

[1716] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[1717] Specific examples

[1718] Cases at child consultation centers

[1719] 1. Start voice input

[1720] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[1721] 2. Collection of audio data

[1722] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[1723] 3. Sending text data and emotion data

[1724] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[1725] 4. Analysis of text data and reflection of emotional data

[1726] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[1727] 5. Automatic generation of required documents

[1728] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[1729] 6. Check and correct documents

[1730] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[1731] 7. Saving and sending documents

[1732] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[1733] Cases at nursery schools

[1734] 1. Start voice input

[1735] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1736] 2. Collection of audio data

[1737] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[1738] 3. Sending text data and emotion data

[1739] Text data and emotion data are sent from the smartphone to a server.

[1740] 4. Analysis of text data and reflection of emotional data

[1741] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[1742] 5. Automatic generation of required documents

[1743] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[1744] 6. Check and correct documents

[1745] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[1746] 7. Saving and sending documents

[1747] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[1748] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[1749] The processing flow will be explained below.

[1750] Step 1:

[1751] The user starts a dedicated application on their device (PC or smartphone) and selects voice input mode. The device prepares for voice recognition and emotion analysis, and displays a message to the user that it is ready.

[1752] Step 2:

[1753] The user begins voice input, for example, saying, "Today's meeting was about my child's school life and home environment." The device uses its built-in microphone to collect this voice input in real time.

[1754] Step 3:

[1755] The collected voice data is converted into text data using voice recognition software within the device. For example, the voice is converted into text data such as, "Today's interview content was a discussion about the child's school life and home environment."

[1756] Step 4:

[1757] The device uses the collected voice data to analyze the user's emotions using a built-in emotion engine, which takes into account voice characteristics such as tone, speed, and strength.

[1758] Step 5:

[1759] The converted text data and analyzed emotion data are sent from the device to the server, which receives the data and begins analyzing it.

[1760] Step 6:

[1761] The server analyzes the text data and extracts the necessary information. For example, it identifies key keywords and phrases in the text, such as "school life" and "home environment," and classifies each as an appropriate document item.

[1762] Step 7:

[1763] Based on the extracted information, the server embeds the data into pre-set templates and automatically generates the necessary documents. For example, it embeds the sections on "school life" and "home environment" into an "interview report" template.

[1764] Step 8:

[1765] The server also incorporates the emotional data analyzed by the emotion engine into the generated document. For example, if the user is feeling stressed, the emotional data is noted in the report.

[1766] Step 9:

[1767] The generated document is sent from the server to the terminal, which presents the document to the user and provides an interface for reviewing and modifying the contents.

[1768] Step 10:

[1769] The user checks the document presented on the terminal and makes corrections as necessary. For example, they can enter additional information in the "home environment" section. The content of the emotional data can also be checked.

[1770] Step 11:

[1771] After the user has finished checking and correcting the document, he / she instructs the terminal to save the final document, and the terminal sends this instruction to the server.

[1772] Step 12:

[1773] The server stores the final documents and sends them electronically to the relevant departments as needed, for example automatically emailing the completed interview report to the relevant department.

[1774] This specific processing step streamlines the process of creating various documents and generates rich reports that reflect the user's emotional state, allowing employees and staff to spend more time on their core tasks.

[1775] Example 2

[1776] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1777] In organizations that handle interpersonal services, employees and staff are required to create numerous documents, which takes time and effort, making it difficult for them to concentrate on their work. Another issue is that the user's emotions are not reflected in the documents, making it difficult to create documents that take into account the user's emotional state.

[1778] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1779] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, means for analyzing the text data and extracting necessary information, means for automatically generating a document in a preset format based on the analyzed information, means for analyzing a user's emotions from the voice input, means for reflecting the emotional information in the document based on the analyzed emotional data, means for presenting the automatically generated document to the user for confirmation and correction, and means for saving and transmitting the final document after confirmation and correction are complete. This makes it possible to streamline document creation, allowing employees and staff to concentrate on their work, and further realizing flexible document creation that takes the user's emotions into consideration.

[1780] "Means for accepting voice input" refers to an interface or device that allows a user to input voice.

[1781] "Means for converting voice input into text data" refers to the process or software that uses voice recognition technology to convert voice data into text information (text data).

[1782] "Means for analyzing text data and extracting necessary information" refers to processes and algorithms that use natural language processing techniques to identify and extract relevant information from text data.

[1783] "Means for automatically generating documents according to pre-defined formats" refers to a process or system that uses pre-defined templates to automatically create documents based on collected and analyzed information.

[1784] "Means for analyzing user emotions from voice input" refers to technologies and algorithms that analyze a user's voice tone and paralinguistic information to identify emotions.

[1785] "Means for reflecting emotional information in a document based on analyzed emotional data" refers to a technique or method for adjusting the content and format of a generated document in consideration of the analyzed emotional data.

[1786] "Means for presenting the automatically generated document to the user and for reviewing and correcting it" refers to an interface or tool that displays the generated document to the user and allows the user to review its contents and edit them as necessary.

[1787] "Means for saving and sending the final document after confirmation and correction is completed" refers to a system or process that has the function of saving the document after the user has confirmed and corrected it and sending it to the specified destination.

[1788] This invention is a system that reduces the administrative burden at institutions that handle interpersonal tasks, such as child consultation centers and nurseries, and provides an environment where employees can concentrate on their work. By combining this system with an emotion engine that recognizes and analyzes the user's emotions, this system realizes flexible document creation that responds to the user's emotions.

[1789] This system consists of three entities: the user, the terminal, and the server. The roles of each entity and their specific processing methods are described below.

[1790] First, a user accesses the system using a terminal and activates the voice input mode. The terminal uses a built-in microphone to collect the user's voice input in real time. The collected voice data is converted into text data using voice recognition software (e.g., Google Speech-to-Text API) within the terminal.

[1791] The converted text data is sent from the device to a server, which then analyzes the received text data and runs natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) to extract the required information.

[1792] The information extracted by the server is automatically embedded in a pre-set document template, which automatically generates the target document (application form, report, daily report, etc.). Furthermore, an emotion engine (e.g., IBM Watson Tone Analyzer) is built in to analyze the user's emotions from voice input. The emotion engine extracts emotional data from the content of the user's speech and the tone, speed, and strength of the voice, and adds it to the analysis results.

[1793] The generated document is sent from the server to the terminal and presented to the user. The user can review the presented document on the terminal and is provided with an editing function to make corrections as necessary. Once the user has finished reviewing and correcting, the final document is saved and sent to the relevant department as needed. Emotion data analyzed by the emotion engine is also fed back to the user, and measures can be taken as needed.

[1794] Specific examples

[1795] Cases at child consultation centers

[1796] 1. Start voice input

[1797] The user (a staff member at a child consultation center) activates the voice input mode using a PC and speaks "today's consultation content." The device collects this voice input in real time using the built-in microphone.

[1798] 2. Collection of audio data

[1799] The collected voice data is converted into text data by the device's voice recognition software, and then an emotion engine analyzes the user's emotions from the voice.

[1800] 3. Sending text data and emotion data

[1801] The text data and emotion data are sent from the device to the server, which receives them and begins analyzing them.

[1802] 4. Analysis of text data and reflection of emotional data

[1803] The server analyzes the text data and extracts information about "school life" and "home environment." Furthermore, the analysis results of the emotion engine are reflected in the document. For example, if a user feels stressed during an interview, that emotion is noted in the report.

[1804] 5. Automatic generation of required documents

[1805] The server embeds the information into a template for an "interview report" and generates a document. Emotional data is also reflected in the appropriate parts of the report.

[1806] 6. Check and correct documents

[1807] The user can review the generated interview report and make corrections as necessary. The content of the emotional data is also fed back to the user, allowing them to check their emotional state.

[1808] 7. Saving and sending documents

[1809] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the report including the emotion data to the relevant department.

[1810] Cases at nursery schools

[1811] 1. Start voice input

[1812] The user (nursery school staff) uses a smartphone to activate the "voice input mode" and speak about the day's activities.

[1813] 2. Collection of audio data

[1814] The smartphone collects audio such as, "Today, Yuuta actively participated in activities at the nursery and had fun playing with his friends." The emotion engine analyzes the user's emotions from the audio.

[1815] 3. Sending text data and emotion data

[1816] Text data and emotion data are sent from the smartphone to a server.

[1817] 4. Analysis of text data and reflection of emotional data

[1818] The server analyzes the text data and extracts information such as "activity participation status" and "relationships with friends." If the user is enjoying talking, their emotions are reflected in the daily report.

[1819] 5. Automatic generation of required documents

[1820] The server embeds the information into a "daily report" template and generates a document. Emotional data is also reflected in the appropriate parts of the daily report.

[1821] 6. Check and correct documents

[1822] The user can check the generated daily report and make corrections as necessary. The content of the emotion data is also fed back to the user, allowing the user to check their emotional state.

[1823] 7. Saving and sending documents

[1824] Once the user has completed the final confirmation, they click the "Save" button to save the document and send the daily report including the emotion data to the parent.

[1825] This system streamlines the creation of various documents, allowing employees and staff to devote more time to their core tasks. In addition, the emotion engine reflects the user's emotional state in the documents, resulting in more comprehensive report creation.

[1826] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1827] Step 1:

[1828] Start voice input

[1829] The user starts speech input mode using the terminal. The user types the command "Start speech input."

[1830] Specific actions

[1831] The user starts the dedicated application and selects the voice input mode.

[1832] The application activates the built-in microphone and prepares for voice input.

[1833] input

[1834] Spoken commands from the user

[1835] output

[1836] Voice input mode is activated

[1837] Step 2:

[1838] Audio data collection

[1839] The device uses a built-in microphone to collect the user's voice input in real time, and the voice data is temporarily stored in a buffer.

[1840] Specific actions

[1841] A microphone captures the user's voice.

[1842] The audio data is stored in a buffer as digital data at a constant sampling rate.

[1843] input

[1844] User voice data

[1845] output

[1846] Temporarily stored audio data in a buffer

[1847] Step 3:

[1848] Audio data conversion

[1849] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the voice data in the buffer into text data.

[1850] Specific actions

[1851] The speech recognition software retrieves the speech data from the buffer.

[1852] The voice data is converted into text data through phonemic analysis and text conversion algorithms.

[1853] input

[1854] Buffered audio data

[1855] output

[1856] Text data

[1857] Step 4:

[1858] Emotional Data Analysis

[1859] The device uses an emotion engine to extract emotion data from the user's voice tone, speed, strength, etc.

[1860] Specific actions

[1861] The emotion engine analyzes the voice data and extracts voice features.

[1862] Emotional states such as stress and joy are identified from the extracted features.

[1863] input

[1864] Audio data

[1865] output

[1866] Emotional Data

[1867] Step 5:

[1868] Sending data

[1869] The device sends text data and emotion data to the server, generates an HTTP request, and sends it to the endpoint.

[1870] Specific actions

[1871] Format text data and emotion data into JSON format.

[1872] Constructs an HTTP request and sends it to the specified endpoint on the server.

[1873] input

[1874] Text data, emotion data

[1875] output

[1876] Data sent to the server

[1877] Step 6:

[1878] Text data analysis

[1879] The server analyzes the received text data using natural language processing algorithms (e.g., spaCy or Hugging Face Transformers) and extracts the necessary information.

[1880] Specific actions

[1881] Natural language processing algorithms break down the text data into individual sentences.

[1882] Extract specific keywords or phrases.

[1883] input

[1884] Text data

[1885] output

[1886] Extracted information

[1887] Step 7:

[1888] Reflecting emotional data

[1889] The server adds emotional information to the extracted information based on the analyzed emotional data.

[1890] Specific actions

[1891] Integrate emotion data into analysis results.

[1892] Generate composite information.

[1893] input

[1894] Text data analysis results, emotion data

[1895] output

[1896] Complex information including emotional information

[1897] Step 8:

[1898] Automatic generation of required documents

[1899] The server embeds the composite information into a pre-defined document template and automatically generates the document.

[1900] Specific actions

[1901] Use a template engine to embed the extracted information into a template.

[1902] Automatically generate documents according to the format.

[1903] input

[1904] Complex information including emotional information

[1905] output

[1906] Generated documents

[1907] Step 9:

[1908] Document review and correction

[1909] The generated document is sent from the server to the terminal and presented to the user, who can then check the document and make any necessary corrections.

[1910] Specific actions

[1911] The server returns the document as an HTTP response.

[1912] The device displays the document and the user uses editing tools to modify the content.

[1913] input

[1914] Generated documents

[1915] output

[1916] Documents modified by users

[1917] Step 10:

[1918] Saving and sending documents

[1919] After the user completes the final confirmation, they save the document and send it to the relevant department if necessary.

[1920] Specific actions

[1921] The terminal sends the modified document to the server.

[1922] The server stores the document in a database and sends it to the specified destination.

[1923] input

[1924] Documents modified by users

[1925] output

[1926] Final documents saved and sent

[1927] (Application example 2)

[1928] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1929] In current interpersonal services, when staff manually create documents, they often experience problems such as reduced work efficiency and difficulty concentrating on their work. Furthermore, conventional systems have difficulty understanding users' emotions and responding appropriately, which can lead to a decline in the quality of service. To solve these issues, a system that combines voice input and emotion analysis is needed.

[1930] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1931] In this invention, the server includes means for accepting voice input, means for converting the voice input into text data, and means for automatically generating documents based on the analyzed information, thereby enabling personnel performing interpersonal tasks to quickly create documents using voice, and further enabling emotional data to be analyzed and reflected in the documents.

[1932] "Voice input" refers to voice data emitted by a user through a microphone.

[1933] "Text data" refers to data that has been converted from voice input into a character string format.

[1934] "Analysis" refers to the process of extracting the intended meaning and necessary information from text data.

[1935] "Format" refers to a pre-defined template used to generate a document.

[1936] "Automatic generation" refers to the process by which the system automatically creates documents based on analyzed information.

[1937] "Presenting" refers to showing the generated document to the user.

[1938] "Modification" refers to the user changing the content of the presented document as needed.

[1939] "Preservation" refers to the recording of the final document in digital or physical form.

[1940] "Transmission" refers to the electronic transfer of the final document to the relevant department or party.

[1941] "Emotion data" refers to data that represents the user's emotional state at the time of voice input.

[1942] "Emotion analysis" refers to the process of analyzing a user's emotions based on the tone, speed, and strength of voice input.

[1943] The system of the present invention is composed of three main components: a user, a terminal, and a server, for quickly creating documents using voice input. The details of each component and their operation are explained below.

[1944] First, a user accesses the system using a device (e.g., a smartphone or tablet). The user activates the device's voice input mode and inputs voice data through the microphone. This voice input is converted into text data in real time using speech recognition software in the device (e.g., Python's speech_recognition library).

[1945] The converted text data is sent from the device to a server. The server receives the text data and analyzes it using natural language processing (NLP) algorithms. The analysis process involves extracting necessary information from the text data. The server also has a built-in emotion engine that analyzes emotional data in voice input, extracting emotional data from the user's voice tone, speed, and strength.

[1946] The analysis results are automatically embedded into a document based on a pre-defined document format. This document can be used, for example, for service guides or proposals. The generated document is then resent from the server to the terminal, where the user can check and modify it. After the user completes the final check, the document is saved and sent to the relevant department.

[1947] A concrete example would be an application where a concierge at a luxury hotel or department store uses a smartphone to input conversations and questions with customers by voice, and then performs sentiment analysis. For example, if the concierge says, "The customer says he wants to relax. I'd like to recommend some spa services," the application converts the speech into text and sends it to a server. The server analyzes the data and automatically generates a proposal for a spa service that matches the customer's relaxation needs.

[1948] Here are some examples of relevant prompts:

[1949] "Your client says they want to relax. Please create a spa service proposal based on the following information:\n\n{Input data about your client's needs}\n\n---\n\nProposal:\n"

[1950] This system allows staff who handle interpersonal tasks to quickly create documents using voice commands and can analyze emotional data and reflect it in the documents, thereby improving the quality of service and realizing efficient work execution.

[1951] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1952] Step 1:

[1953] The user activates the voice input mode of the terminal and inputs voice data. The input voice data is collected through a microphone. At this point, the input is the user's voice, and the output is digital voice data.

[1954] Step 2:

[1955] The voice data collected by the device is converted into text data in real time using voice recognition software (for example, Python's speech_recognition library). The input is voice data and the output is text data. In this step, the voice data is converted into a string.

[1956] Step 3:

[1957] The terminal sends the converted text data to the server. The input is text data, and the output is data sent to the server. Data communication takes place in this step.

[1958] Step 4:

[1959] The server analyzes the received text data using a natural language processing (NLP) algorithm. The input is text data, and the output is analyzed information. In this step, the necessary information is extracted from the text data.

[1960] Step 5:

[1961] The server extracts the user's emotional data using an emotion engine that analyzes the emotional data from the voice input. The input is voice data, and the output is emotional data. In this step, data analysis is performed on the voice tone, speed, and strength.

[1962] Step 6:

[1963] The server automatically generates a document based on the analyzed information and emotion data according to a pre-set document format. The input is the analyzed information and emotion data, and the output is an automatically generated document. This step involves the information embedding process.

[1964] Step 7:

[1965] The server sends the automatically generated document to the terminal. The input is the automatically generated document, and the output is the data sent to the terminal. In this step, data communication occurs again.

[1966] Step 8:

[1967] The user checks the document received on the terminal and makes corrections if necessary. The input is the automatically generated document, and the output is the corrected document. This is the step where user interaction takes place.

[1968] Step 9:

[1969] After the user has confirmed and corrected the final document, it is saved on the terminal and sent to the relevant department. The input is the corrected document, and the output is the saved and sent data. In this step, the document is saved and the data is finally sent.

[1970] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1971] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1972] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1973] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1974] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1975] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1976] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1977] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1978] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1979] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1980] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1981] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1982] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1983] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1984] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1985] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1986] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1987] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1988] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1989] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1990] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1991] The following is further disclosed regarding the above embodiment.

[1992] (Claim 1)

[1993] means for accepting voice input;

[1994] means for converting the voice input into text data;

[1995] means for analyzing the text data and extracting necessary information;

[1996] means for automatically generating a document in accordance with a preset format based on the analyzed information;

[1997] means for presenting the automatically generated document to a user for confirmation and correction;

[1998] means for storing and transmitting the final document after said review and correction is completed;

[1999] A system including:

[2000] (Claim 2)

[2001] 2. The system of claim 1, further comprising: means for collecting said speech input in real time; and means for converting said real-time collected speech data into text data.

[2002] (Claim 3)

[2003] 10. The system of claim 1, further comprising means for using natural language processing algorithms to extract the required information.

[2004] (Claim 4)

[2005] 10. The system of claim 1, further comprising means for said document template to be in a pre-defined format.

[2006] (Claim 5)

[2007] 10. The system of claim 1, further comprising means for providing the user with the ability to review and modify the presented document.

[2008] "Example 1"

[2009] (Claim 1)

[2010] means for a user to initiate speech input;

[2011] means for using a microphone built into the terminal to accept said voice input;

[2012] A means for converting the collected voice data into text data using voice recognition software;

[2013] means for transmitting the text data to a server;

[2014] means for executing a natural language processing algorithm to analyze the text data and extract necessary information;

[2015] a means for automatically generating a document by embedding information into a preset template based on the analyzed information;

[2016] means for presenting the automatically generated document to a user for confirmation and correction;

[2017] means for storing and transmitting the final document after said review and correction is completed;

[2018] A system including:

[2019] (Claim 2)

[2020] 10. The system of claim 1, further comprising: means for collecting said voice input in real time; and means for converting said real time collected voice data into text data using voice recognition software.

[2021] (Claim 3)

[2022] 10. The system of claim 1, further comprising: means for using a natural language processing algorithm to extract the required information; and means for automatically embedding the extracted information into a pre-defined document template.

[2023] "Application Example 1"

[2024] (Claim 1)

[2025] means for accepting voice input;

[2026] means for converting the voice input into text data;

[2027] means for analyzing the text data and extracting necessary information;

[2028] means for automatically generating a document in accordance with a preset format based on the analyzed information;

[2029] means for presenting the automatically generated document to a user for confirmation and correction;

[2030] means for storing and transmitting the final document after said review and correction is completed;

[2031] means for receiving passenger voice input using a voice input device installed in the interior of the vehicle;

[2032] means for analyzing information including destination and requirements based on the voice input and automatically generating a report for the passenger;

[2033] means for passengers to review and provide said automatically generated report upon arrival;

[2034] A system including:

[2035] (Claim 2)

[2036] 2. The system of claim 1, further comprising: means for collecting said speech input in real time; and means for converting said real-time collected speech data into text data.

[2037] (Claim 3)

[2038] 10. The system of claim 1, further comprising means for using natural language processing algorithms to extract the required information.

[2039] "Example 2: Combining Emotion Engines"

[2040] (Claim 1)

[2041] means for accepting voice input;

[2042] means for converting the voice input into text data;

[2043] means for analyzing the text data and extracting necessary information;

[2044] means for automatically generating a document in accordance with a preset format based on the analyzed information;

[2045] means for analyzing a user's emotion from the voice input;

[2046] a means for reflecting emotional information in a document based on the analyzed emotional data;

[2047] means for presenting the automatically generated document to a user for confirmation and correction;

[2048] means for storing and transmitting the final document after said review and correction is completed;

[2049] A system including:

[2050] (Claim 2)

[2051] 2. The system of claim 1, further comprising: means for collecting said speech input in real time; and means for converting said real-time collected speech data into text data.

[2052] (Claim 3)

[2053] 10. The system of claim 1, further comprising means for using natural language processing algorithms to extract the required information.

[2054] "Application example 2 when combining emotion engines"

[2055] (Claim 1)

[2056] means for accepting voice input;

[2057] means for converting the voice input into text data;

[2058] means for analyzing the text data and extracting necessary information;

[2059] means for automatically generating a document in accordance with a preset format based on the analyzed information;

[2060] means for presenting the automatically generated document to a user for confirmation and correction;

[2061] means for storing and transmitting the final document after said review and correction is completed;

[2062] means for analyzing emotion data from the voice input and reflecting the emotion data in a generated document;

[2063] A system including:

[2064] (Claim 2)

[2065] 2. The system of claim 1, further comprising: means for collecting said speech input in real time; and means for converting said real-time collected speech data into text data.

[2066] (Claim 3)

[2067] 10. The system of claim 1, further comprising means for using natural language processing algorithms to extract the required information. [Explanation of symbols]

[2068] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting voice input; means for converting the voice input into text data; means for analyzing the text data and extracting necessary information; means for automatically generating a document in accordance with a preset format based on the analyzed information; means for presenting the automatically generated document to a user for confirmation and correction; means for storing and transmitting the final document after said review and correction is completed; A system including:

2. 2. The system of claim 1, further comprising: means for collecting said speech input in real time; and means for converting said real-time collected speech data into text data.

3. 10. The system of claim 1, further comprising means for using a natural language processing algorithm to extract said required information.

4. 2. The system of claim 1, further comprising means for said document template being in a preset format.

5. 10. The system of claim 1, further comprising means for providing said user with the ability to review and modify said submitted document.

Images