System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a voice input and converting a user's voice into character data; natural language processing means for analyzing the character data and understanding a request content; generation model means for generating a document based on the generated character data; means for referring to a past document created by a company and customizing the document; means for receiving a voice instruction and correcting the document; and means for confirming and outputting a final version.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today's desk workers suffer from eye strain due to long hours of PC work and online meetings. This can often lead to serious health problems such as VDT syndrome, cataracts, and glaucoma, resulting in reduced productivity. Furthermore, for companies, employee withdrawal from work and reduced productivity have a negative impact on business performance. There is a need for a solution to these issues, allowing desk workers to create documents efficiently while reducing eye strain. [Means for solving the problem]

[0005] The present invention is a system that includes means for accepting voice input, means for converting the user's voice into text data, natural language processing means for analyzing the text data and understanding the requested content, generative model means for generating documents based on the generated text data, means for customizing documents by referencing documents created in the past by the company, means for revising documents in response to voice instructions, and means for checking and outputting the final version. This allows users to create high-quality documents using only voice instructions without having to look at a PC screen, thereby reducing eye strain.

[0006] "Voice input" is a means by which a user communicates instructions to a system using their voice.

[0007] "Character data" is data converted from voice input into text format.

[0008] "Natural language processing means" is a technology for analyzing character data and understanding its context and meaning.

[0009] "Generative model means" is a technology that automatically generates documents based on the results of natural language processing.

[0010] "Previously created documents" are documents and materials created by a company in the past that are used as templates or style guides.

[0011] "Means for customization" refers to techniques that refer to previously created documents and adapt newly generated documents to a company's specific format and style.

[0012] "Voice instructions" are instructions for correction or confirmation given by the user to the system using voice.

[0013] The "final version" is the final document that reflects the user's instructions and customizations.

[0014] "Means for output" refers to the technology used to display text or read aloud the final document to the user. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention provides a system for generating documents through voice input, thereby reducing eye strain for desk workers. The overall flow of this system and specific embodiments are described below.

[0037] System Overview

[0038] This system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and an output means. A user gives instructions by voice, and the instructions are analyzed through natural language processing, and finally a document is generated and output.

[0039] Program processing overview

[0040] Voice input and conversion

[0041] 1. The user speaks into the terminal to request the creation of materials.

[0042] For example: "I'd like you to prepare a proposal for next week."

[0043] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[0044] Audio data: "Please prepare a proposal for next week."

[0045] Converted text data: "Please prepare a proposal for next week."

[0046] Parsing and Document Generation

[0047] 1. The terminal sends the converted text data to the server.

[0048] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[0049] Analysis result: "Create a proposal for next week"

[0050] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[0051] Customization and Documentation Reference

[0052] 1. The server references previously created documents from the company database and retrieves appropriate templates and style guides.

[0053] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0054] Example: "Format it into the format of a past marketing proposal."

[0055] Correction instructions and final version generation

[0056] 1. The user gives instructions to correct the document by voice into the terminal.

[0057] For example: "Delete the third paragraph and add a new point."

[0058] 2. The device converts this voice instruction into text data and sends it to the server.

[0059] Audio: "Delete the third paragraph and add a new point."

[0060] Converted text: "Delete the third paragraph and add a new point."

[0061] 3. The server receives the text data and modifies it using the generative modeling means.

[0062] Check and output

[0063] 1. The user issues a final confirmation instruction to the terminal.

[0064] Example: "Please finalize and submit the report."

[0065] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[0066] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[0067] Example: "No problem, send it."

[0068] 4. The server sends the final document to the receiving location specified by the user.

[0069] Specific examples

[0070] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[0071] This system allows desk workers to create high-quality documents using only their voice, without looking at a screen, reducing eye strain and supporting a healthier working style.

[0072] The processing flow will be explained below.

[0073] Step 1:

[0074] The user gives instructions for creating materials by voice into the terminal.

[0075] For example: "I'd like you to prepare a proposal for next week."

[0076] Step 2:

[0077] The device captures the user's voice with a microphone.

[0078] Audio data: "Please prepare a proposal for next week."

[0079] Step 3:

[0080] The device uses a speech recognition engine to convert the captured voice into text data.

[0081] Converted text data: "Please prepare a proposal for next week."

[0082] Step 4:

[0083] The terminal transmits the text data to the server.

[0084] Step 5:

[0085] The text data received by the server is analyzed using a natural language processing engine.

[0086] Analysis result: Understand the instruction "Write a proposal."

[0087] Step 6:

[0088] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[0089] A first draft proposal is generated.

[0090] Step 7:

[0091] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[0092] Step 8:

[0093] The server applies templates and style guides to the generated initial version to create customized documents.

[0094] The customized proposal draft is completed.

[0095] Step 9:

[0096] The user gives instructions to correct the document by voice into the terminal.

[0097] For example: "Delete the third paragraph and add a new point."

[0098] Step 10:

[0099] The terminal converts the user's voice into text data and sends it to the server.

[0100] Converted text: "Delete the third paragraph and add a new point."

[0101] Step 11:

[0102] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0103] Step 12:

[0104] The server uses the generative model to modify the document.

[0105] A revised proposal is generated.

[0106] Step 13:

[0107] The user issues a final confirmation instruction by voice to the terminal.

[0108] Example: "Please finalize and submit the report."

[0109] Step 14:

[0110] The server generates the final document and sends it to the terminal.

[0111] The final proposal is sent to the device.

[0112] Step 15:

[0113] Your device will read the final document aloud or display it as text.

[0114] Step 16:

[0115] After the user makes a final confirmation, he or she issues a voice command saying, "There are no problems, so send it."

[0116] Step 17:

[0117] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[0118] For example: a user's email address or a company's shared folder.

[0119] This allows users to create high-quality documents using only their voice, without using their eyes, enabling a healthier way of working.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Today's desk workers often suffer from eye strain and stress due to long hours of screen time. Furthermore, document creation takes a lot of time and effort, making it difficult to carry out work efficiently. For this reason, there is a demand for technology that utilizes voice input to efficiently generate documents and reduce eye strain.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for referencing previously created documents and customizing the document, means for correcting the document in response to voice instructions, means for confirming and outputting the final version, means for reading the final version aloud or displaying it in text to prompt the user for confirmation, and means for managing the operation of the entire system based on a program flow divided into processing steps. This allows users to efficiently create, correct, confirm, and output documents using only voice input, reducing eye strain and improving work efficiency.

[0125] "Voice input" is an input method in which the system receives voice data spoken by the user.

[0126] "Character data" refers to text data converted from voice input using voice recognition technology.

[0127] "Natural language processing" is a technology that analyzes text data and understands its content and meaning.

[0128] A "generative model" is a technology that generates new documents or content based on input data.

[0129] "Customization" refers to the process of tailoring a generated document to a particular template or historical document style.

[0130] "Voice instructions" refer to instructions for corrections or operations given by the user using voice.

[0131] The "final version" refers to a document that is completed at the final stage of document creation or revision.

[0132] "Output" refers to the process of displaying, printing, or transmitting the generated document.

[0133] "Read aloud" refers to the process by which the system converts text data into audio and plays it aloud for the user.

[0134] "Program flow" refers to the flow of a program that shows the processing procedures and steps of the entire system.

[0135] "Managing the operation of the entire system" refers to the management activities of controlling each function of the system in an integrated manner and operating it efficiently.

[0136] The present invention provides a system for generating documents through voice input to reduce eye strain for desk workers. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, and an output unit.

[0137] Voice input and conversion

[0138] The user requests the creation of a document by voice, and the device captures this voice using a microphone. The captured voice data is converted into text data by a voice recognition engine (for example, Google Cloud Speech-to-Text API). As a result, the voice data "Please create a proposal for next week" is converted into text data "Please create a proposal for next week."

[0139] Parsing and Document Generation

[0140] The device sends the converted text data to the server. The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the text data and understand the user's request. For example, the analysis result may be "Create a proposal for next week." Based on this analysis result, the server launches a generative model (e.g., OpenAI GPT-3) to generate the first version of the document.

[0141] Customization and Documentation Reference

[0142] The server references past documents from a corporate database (e.g., a MySQL database) and retrieves the appropriate templates and style guides. It then uses these templates and style guides to customize the first version of the document, for example, to match the format of a past marketing proposal.

[0143] Correction instructions and final version generation

[0144] The user gives instructions to edit the document by voice, and the device captures the voice data, converts it into text data, and sends it to the server. For example, the instruction might be, "Delete the third paragraph and add a new main point." The server analyzes this and invokes the generative model to edit the document.

[0145] Check and output

[0146] The user gives a voice command for final confirmation, and the server generates a final version of the document and sends it to the terminal. The terminal reads this final version aloud or displays it as text. The user checks it, and if there are no problems, gives a voice command for final transmission. For example, "There are no problems, so send it." The server sends the final version of the document to the specified receiving location.

[0147] Specific examples

[0148] The specific operation flow is shown below.

[0149] The user says, "I want you to create a marketing strategy report for next week." The device converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says, "It's fine, so send it," and the server sends the final version to the specified receiving location.

[0150] This system allows users to efficiently create high-quality documents through voice input and work while reducing eye strain.

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1: Getting voice input

[0153] The user requests the creation of a document by voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the user speaks into the device's microphone, saying, "I want you to create a proposal for next week."

[0154] Step 2: Capture audio

[0155] The terminal uses a microphone to capture the user's voice. The input is the user's voice data, and the output is the captured voice file. Specifically, the terminal collects voice data in real time.

[0156] Step 3: Convert audio data to text

[0157] The device uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice data into text data. The input is the captured voice data, and the output is text data. Specifically, the device converts voice data such as "Please write a proposal for next week" into text data such as "Please write a proposal for next week."

[0158] Step 4: Sending text data

[0159] The terminal sends the converted text data to the server. The input is text data, and the output is the text data sent to the server. Specifically, the text data is encrypted and transmitted to the server via the Internet.

[0160] Step 5: Parsing the text

[0161] The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the received text data. The input is the received text data, and the output is the analysis result. Specifically, it analyzes the text "Please create a proposal for next week" and extracts the intent "to create a proposal for next week."

[0162] Step 6: Generate the first version of the document

[0163] Based on the analysis results, the server launches a generative AI model (e.g., OpenAI GPT-3) to generate the first version of the document. The input is the analysis results, and the output is the first version of the document. Specifically, the prompt "Create a proposal for next week" is input into the generative AI model, which then generates the first version of the document.

[0164] Step 7: Referencing past documents

[0165] The server references past documents from the company's database (e.g., MySQL database). The input is a request to reference a past document, and the output is the past document data. Specifically, it uses an SQL query to retrieve templates of past marketing proposals.

[0166] Step 8: Customize the initial document

[0167] The server customizes the first version of the document using the acquired templates and style guides. The input is the first version of the document and past document data, and the output is a customized document. Specifically, the first version of the document is formatted to match the format of past marketing proposals.

[0168] Step 9: Speak correction instructions

[0169] The user gives instructions to edit the document by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Delete the third paragraph and add a new main point."

[0170] Step 10: Convert correction instructions to text

[0171] The terminal captures the voice of the correction instruction and converts it into text data using a voice recognition engine. The input is the captured voice data, and the output is text data. Specifically, the voice data is converted into text and sent to the server.

[0172] Step 11: Modifying the document

[0173] The server analyzes the received text data and uses a generative AI model to correct the document. The input is the text data of the correction instructions, and the output is the corrected document. Specifically, the document is regenerated using the AI ​​model based on the correction instructions.

[0174] Step 12: Speak your final confirmation instructions

[0175] The user gives instructions for final confirmation by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Please finalize the report and send it."

[0176] Step 13: Generate and send the final version

[0177] The server generates the final document and sends it to the terminal. The input is the revised document, and the output is the final document. The terminal reads this final version aloud or displays it as text. Specifically, the server generates a final document file and transfers it to the terminal for display or aloud reading.

[0178] Step 14: Send document instructions

[0179] If there are no problems, the user gives the final command for transmission by voice. The input is the user's voice data, and the output is the captured voice data. The specific operation is to say, "There are no problems, so send it."

[0180] Step 15: Final submission of document

[0181] The server sends the final document to the receiving location specified by the user. The input is the final sending instruction and the final document, and the output is the document sent to the receiving location. The specific operation is to upload the final version to a mail server or cloud storage.

[0182] (Application example 1)

[0183] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0184] Employees in brick-and-mortar stores are required to provide fast, high-quality customer service and customer service on a daily basis. However, achieving this requires detailed manuals and guidelines, and creating them takes time and effort. Furthermore, employees may experience eye strain from looking at a screen for long periods of time, which can have a negative impact on their productivity and health. To solve these issues, a system is needed that uses voice input to automatically generate customer service manuals quickly and efficiently.

[0185] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0186] In this invention, the server includes means for accepting voice input and converting the user's voice into text data, natural language processing means for analyzing the text data and understanding the request content, generative model means for generating documents based on the generated text data, means for referencing past documents and customizing the documents, and means for automatically generating daily work manuals and service guides. This enables store employees to quickly and accurately generate customer service manuals using only voice input, improving productivity and reducing eye strain.

[0187] "Voice input" is the process of receiving a user's voice as digital data and converting it into text data.

[0188] "Natural language processing means" is a technology that analyzes input text data, understands and processes human language, and is used to recognize the content of a user's request.

[0189] "Generative model means" refers to algorithms or models that automatically generate new documents based on analyzed text data.

[0190] "Customization" is the process of referencing previous documents and adjusting the generated document based on them to fit a specific format or style.

[0191] "Voice instruction" refers to the act of a user giving voice instructions to the system to perform a specific operation or correction.

[0192] The "means for modifying a document" refers to a function for editing an existing document and making changes or additions based on the user's voice instructions.

[0193] "Final" refers to the final document after all modifications and customizations have been completed.

[0194] "Means for outputting" refers to a function for presenting the final document to the user or transmitting it to a specified location.

[0195] "Means for automatically generating daily work manuals and service guides" refers to a process for automatically creating and providing instructions and explanations required for daily work and customer service.

[0196] This invention is a system that automatically generates customer service manuals and service guides for brick-and-mortar stores using voice input. The system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and a final version confirmation and output means.

[0197] System Configuration

[0198] Hardware

[0199] Microphone: A device for capturing audio input.

[0200] Computer or smartphone: a device that processes and stores data.

[0201] software

[0202] speech_recognition library: A library for converting speech data into text.

[0203] Transformers library and general-purpose generative models: A library for natural language processing and text generation. Specifically, it uses the Rinna / Japanese GPT-2 model.

[0204] Data processing and calculation

[0205] Voice input and conversion

[0206] The server captures the voice data that the user speaks into the microphone. This voice data is converted into text data using the speech_recognition library. For example, if the user instructs, "Create a manual to explain to customers about today's special menu," the voice is converted into text.

[0207] Parsing and Document Generation

[0208] The server receives the converted text data and analyzes it using natural language processing. Based on the analysis results, it activates a generative model to generate the first edition of the customer service manual. As a generative model, it uses the transformers library to activate the Japanese GPT-2 model.

[0209] Customization

[0210] The server references past customer service manuals and service guides from the company's database and formats the generated documents into appropriate templates and style guides, thereby ensuring consistency in the documents.

[0211] Corrections and final check

[0212] The user can make corrections to the generated document by issuing additional voice commands. For example, they can say, "Update the second paragraph with the latest trend information." This voice command is also converted into text by the speech_recognition library, and the document is corrected by the server. Finally, the user confirms and sends the document by saying, "There are no problems, so please send it."

[0213] output

[0214] The server generates the final document and presents it to the user, who then confirms it and sends it to the specified location.

[0215] Examples and prompts

[0216] For example, if an employee at a physical store says, "Please create a manual to explain to customers about today's special menu items," the speech is converted into text, and natural language processing is performed based on that text data to generate the manual. The employee then gives additional instructions, such as "Update the second paragraph with new trend information," and finally instructs, "There are no problems, so please submit," and the completed manual is generated.

[0217] (Example of a prompt)

[0218] "Create a manual to explain today's special menu to customers."

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The user verbally requests the creation of a document into the microphone. For example, "Create a manual explaining today's special menu to customers." The input is the user's voice data. The output is the captured voice data.

[0222] Step 2:

[0223] The device converts this voice data into text data using a speech recognition engine (speech_recognition library). The input is the captured voice data, specifically voice waveform data. The device obtains text data by inputting the voice waveform data into the voice recognition engine. The output is the converted text data. For example, the voice data "Please create a manual to explain to customers about today's special menu" is converted into text data "Please create a manual to explain to customers about today's special menu."

[0224] Step 3:

[0225] The terminal sends this text data to the server. The input is the converted text data, and the output is the text data sent to the server.

[0226] Step 4:

[0227] The server analyzes the received text data using natural language processing means to understand the user's request. The input is text data, specifically the aforementioned text data "Please create a manual to explain to customers about today's special menu." The server uses natural language processing technology to analyze the request. The output is the analyzed request, for example, "Request to create a manual explaining the special menu."

[0228] Step 5:

[0229] The server invokes the generative model based on the analysis results to generate the first version of the document. The input is the parsed request content, and the output is the generated first version of the document. Specifically, the document is generated using the transformers library and a generative AI model (Japanese GPT-2).

[0230] Step 6:

[0231] The server references previously created documents from the company's database and customizes the generated document. The input is the first version of the generated document and the referenced previous document data. The server uses this information to adjust the format and style of the document. The output is a customized document.

[0232] Step 7:

[0233] The user speaks into the terminal to instruct the device to edit a document. For example, they say, "Update the second paragraph with the new trend information." The input is the voice data of the user's edit instructions. The output is the captured voice data.

[0234] Step 8:

[0235] The terminal converts this voice correction instruction into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "Update the second paragraph with the new trend information."

[0236] Step 9:

[0237] The server receives the text data and modifies the document using the generative model means. The input is the text data of the user's modification instructions, and the output is the modified document. Specifically, the server uses the generative model means to update the document based on the instructions.

[0238] Step 10:

[0239] The user gives a final confirmation instruction. For example, they say, "There's no problem, so send it." The input is the user's voice data of the final confirmation. The output is the captured voice data.

[0240] Step 11:

[0241] The terminal converts this final confirmation voice into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "There are no problems, so please send it."

[0242] Step 12:

[0243] The server generates the final document and sends it to the terminal. The input is the final confirmation instruction text data and the customized / modified document. The server generates the final document based on this. The output is the final document. The terminal reads this final version aloud or displays it in text.

[0244] Step 13:

[0245] The user performs a final check and, if there are no problems, gives a final voice command. The server sends the final document to the specified receiving location. The input is the user's voice command, and the output is the final document sent to the destination.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] The present invention is a system for generating documents through voice input and adjusting the tone and content of the documents taking into account the user's emotional state. The overall flow of this system and specific embodiments are described below.

[0248] System Overview

[0249] The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit. A user gives an instruction by voice, the instruction is analyzed through natural language processing, and finally a document is generated and output based on the user's emotional state.

[0250] Program processing overview

[0251] Voice input and conversion

[0252] 1. The user speaks into the terminal to request the creation of materials.

[0253] For example: "I'd like you to prepare a proposal for next week."

[0254] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[0255] Audio data: "Please prepare a proposal for next week."

[0256] Converted text data: "Please prepare a proposal for next week."

[0257] Parsing and Document Generation

[0258] 1. The terminal sends the converted text data to the server.

[0259] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[0260] Analysis result: "Creating a proposal"

[0261] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[0262] A first draft proposal is generated.

[0263] Customization and Emotion Recognition

[0264] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[0265] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0266] Example: "Format it into the format of a past marketing proposal."

[0267] 3. The server uses the emotion engine means to recognize emotions from the user's voice. For example, emotions such as "tense" or "relaxed" are recognized from the user's voice.

[0268] 4. The server adjusts the tone and content of the generated document based on the user's emotions.

[0269] Example: If a user is "nervous," adjust the tone of the document to be more relaxing.

[0270] Correction instructions and final version generation

[0271] 1. The user gives instructions to correct the document by voice into the terminal.

[0272] For example: "Delete the third paragraph and add a new point."

[0273] 2. The device converts this voice instruction into text data and sends it to the server.

[0274] Converted text: "Delete the third paragraph and add a new point."

[0275] 3. The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0276] 4. The server modifies the document using the generative model means.

[0277] A revised proposal is generated.

[0278] Check and output

[0279] 1. The user issues a final confirmation instruction to the terminal.

[0280] Example: "Please finalize and submit the report."

[0281] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[0282] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[0283] Example: "No problem, send it."

[0284] 4. The server sends the final document to the receiving location specified by the user.

[0285] For example: a user's email address or a company's shared folder.

[0286] Specific examples

[0287] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats it into an appropriate template. At the same time, an emotion engine is used to analyze the emotions from the user's voice, and if it determines, for example, that the user is nervous, the tone of the document is softened. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[0288] This system not only allows desk workers to create high-quality documents using only their voice without using their eyes, but also provides documents with an appropriate tone and content depending on the user's emotional state, creating a less stressful work environment.

[0289] The processing flow will be explained below.

[0290] Step 1:

[0291] The user gives instructions for creating materials by voice into the terminal.

[0292] For example: "I'd like you to prepare a proposal for next week."

[0293] Step 2:

[0294] The device captures the user's voice with a microphone.

[0295] Audio data: "Please prepare a proposal for next week."

[0296] Step 3:

[0297] The device uses a speech recognition engine to convert the captured voice into text data.

[0298] Converted text data: "Please prepare a proposal for next week."

[0299] Step 4:

[0300] The terminal transmits the text data to the server.

[0301] Step 5:

[0302] The text data received by the server is analyzed using a natural language processing engine.

[0303] Analysis result: Understand the instruction "Write a proposal."

[0304] Step 6:

[0305] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[0306] A first draft proposal is generated.

[0307] Step 7:

[0308] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[0309] Step 8:

[0310] The server applies templates and style guides to the generated initial version to create customized documents.

[0311] The customized proposal draft is completed.

[0312] Step 9:

[0313] The server uses an emotion engine to recognize emotions from the user's voice.

[0314] For example, recognizing emotional states such as "tense," "relaxed," and "excited."

[0315] Step 10:

[0316] The server adjusts the tone and content of the generated document based on the emotion recognition results.

[0317] Example: If the user is in a "tense" state, adjust the tone of the document to be softer.

[0318] Step 11:

[0319] The user gives instructions to correct the document by voice into the terminal.

[0320] For example: "Delete the third paragraph and add a new point."

[0321] Step 12:

[0322] The terminal converts the user's voice into text data and sends it to the server.

[0323] Converted text: "Delete the third paragraph and add a new point."

[0324] Step 13:

[0325] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0326] Step 14:

[0327] The server uses the generative model to modify the document.

[0328] A revised proposal is generated.

[0329] Step 15:

[0330] The user issues a final confirmation instruction by voice to the terminal.

[0331] Example: "Please finalize and submit the report."

[0332] Step 16:

[0333] The server generates the final document and sends it to the terminal.

[0334] The final proposal is sent to the device.

[0335] Step 17:

[0336] Your device will read the final document aloud or display it as text.

[0337] Step 18:

[0338] After the user makes a final check, he or she issues a voice command saying, "There are no problems, so send it."

[0339] Step 19:

[0340] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[0341] For example: a user's email address or a company's shared folder.

[0342] This allows users to create high-quality documents using only their voice, without using their eyes, and provides the optimal tone and content according to their emotional state, creating a less stressful work environment.

[0343] Example 2

[0344] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0345] In conventional document creation systems, when users give instructions by voice, they not only convert the voice input into text data, but also lack the functionality to generate and edit documents taking into account the user's emotions and the tone of the document. As a result, even when users give instructions that convey their emotions, these are not easily reflected in the document, resulting in a poor user experience. In addition, the functionality to apply templates by referencing past documents is insufficient, which makes it difficult to efficiently generate documents with a consistent feel.

[0346] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for accepting voice input and converting the user's voice into character data; natural language processing means for analyzing the character data and understanding the requested content; generative model means for generating a document based on the generated character data; means for customizing the document by referring to previously created documents; means for recognizing emotions from the user's voice and adjusting the tone and content of the document; means for correcting the document in response to voice instructions; and means for checking and outputting the final version. This enables high-quality document generation and correction that reflects the user's voice and emotions, thereby improving the user experience and realizing efficient document creation.

[0347] "Voice input" is a technology that captures what a user says as voice data.

[0348] "Character data" refers to data obtained by converting voice data into characters.

[0349] "Natural language processing means" is a technology that analyzes character data and understands the content of a user's request.

[0350] A "generative model means" is an algorithm that generates a document based on input data.

[0351] "Previously created documents" are documents created in the past that are used as templates or reference materials.

[0352] "Means of customization" refers to techniques for modifying and adjusting generated documents in accordance with specific style guides and templates.

[0353] "Means for recognizing emotions" refers to technology that analyzes and recognizes the emotional state of a user from their voice.

[0354] "Means for adjusting the tone and content of a document" refers to techniques for changing the expression and content of a document based on the user's emotions and needs.

[0355] "Voice instructions" are instructions or commands about a document given by the user through voice.

[0356] "Means for modifying documents" refers to technology that adds, deletes, or modifies existing documents based on voice instructions.

[0357] The "means for checking and outputting the final version" is a technique for outputting a document that has been finally checked by a user.

[0358] MODE FOR CARRYING OUT THE INVENTION

[0359] The present invention provides a system for generating documents through voice input and adjusting the tone and content of the documents in consideration of a user's emotional state. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit.

[0360] Hardware and Software Configuration

[0361] This system is configured using the following hardware and software.

[0362] 1. Device:

[0363] Audio capture device (e.g. microphone)

[0364] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[0365] 2. Server:

[0366] Natural language processing tools (e.g., spaCy, BERT)

[0367] Generative modeling tools (e.g., OpenAI GPT-4)

[0368] Database search methods (e.g., SQL databases)

[0369] Emotion engine means (e.g. IBM Watson Tone Analyzer)

[0370] Program processing flow

[0371] When a user speaks, the speech is captured by the device and converted into text using a speech recognition engine. The text data is then sent to a server, where the user's request is analyzed using natural language processing means. Based on the analysis results, a generative model means is activated to generate a first version of the document. The server then searches for past documents in the company's database and customizes the document by applying appropriate templates and style guides. Next, an emotion engine means is used to recognize emotions from the user's voice and adjust the tone and content of the document based on the emotions.

[0372] If the user makes voice corrections to the document, the voice is converted into text again and sent to the server for analysis and generation. Finally, the user gives a final check, and if there are no problems, the server outputs the final version of the document.

[0373] Explanation with concrete examples

[0374] For example, if a user says to a terminal, "I want you to write a marketing strategy report for next week," the terminal captures this speech and uses a speech recognition engine to convert it into text data saying, "I want you to write a marketing strategy report for next week." This text data is sent to a server and analyzed by natural language processing means. Based on the analysis results, a generative model means creates a first edition of a marketing strategy report. The server then references past marketing reports and applies an appropriate template. At the same time, the emotion engine means recognizes emotions from the user's voice, and if it determines, for example, that the user is nervous, it adjusts the tone of the document to be softer.

[0375] When the user instructs, "Update the second paragraph with the new trend information," the device again captures the voice, converts it into text data, and sends it to the server. The server makes the corrections and generates a document containing the new trend information. Finally, when the user instructs, "There are no problems, so send it," the server sends the final document to the specified recipient (e.g., email address or shared folder).

[0376] Prompt Sentence Examples

[0377] User: "I want a marketing strategy report for next week."

[0378] Device: "Converting speech to text"

[0379] Server: "Generating first version"

[0380] Server: "Applying template by referencing past reports"

[0381] Server: "Analyzing user sentiment"

[0382] Server: "Adjusting the tone of the document"

[0383] User: "Update the second paragraph with new trending information."

[0384] Device: "Converting speech to text"

[0385] Server: "Generating updated document"

[0386] User: "No problem, send it."

[0387] Server: "Sending final version to specified location"

[0388] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0389] Step 1: Capture and convert audio input

[0390] 1. The user gives instructions through a voice input device. Example: "Please prepare a proposal for next week."

[0391] 2. The device captures the user's voice using an audio capture device (e.g., a microphone).

[0392] Input: User's voice data

[0393] 3. The device converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0394] Data processing: Converting voice data into text data

[0395] Output: Text data "Please create a proposal for next week."

[0396] Step 2: Sending text data

[0397] 1. The terminal sends the converted character data to the server.

[0398] Input: Text data "Please create a proposal for next week."

[0399] Output: Character data sent to the server

[0400] Step 3: Analyzing the text data

[0401] 1. The server analyzes the received text data using natural language processing tools (e.g., spaCy, BERT) to understand the user's request.

[0402] Input: Text data "Please create a proposal for next week."

[0403] Data Computing: Analyzing Request Contents Using Natural Language Processing

[0404] Output: Analysis result "Creating a proposal"

[0405] Step 4: Generate the initial document

[0406] 1. The server launches a generative model (e.g., OpenAI GPT-4) based on the analysis results to generate the first version of the document.

[0407] Input: Analysis result "Create a proposal"

[0408] Data Computation: First Edition Document Generation with Document Generation Algorithms

[0409] Output: First edition proposal

[0410] Step 5: Apply and customize the template

[0411] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[0412] Input: Company database

[0413] Data processing: Searching and retrieving past documents

[0414] Output: Templates and style guides

[0415] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0416] Input: First edition proposals, templates and style guides

[0417] Data calculations: Customization based on template application and style guides

[0418] Output: Customized document

[0419] Step 6: Emotional awareness and tone adjustment

[0420] 1. The server recognizes emotions from the user's voice using an emotion engine means (e.g., IBM Watson Tone Analyzer).

[0421] Input: User's voice data

[0422] Data Computing: Sentiment Analysis

[0423] Output: Emotional state (e.g., "I'm nervous")

[0424] 2. The server adjusts the tone and content of the generated document based on the user's emotions.

[0425] Input: Emotional state "Tense", customized document

[0426] Data operations: tone and content adjustment

[0427] Output: Adjusted document

[0428] Step 7: Processing correction instructions

[0429] 1. The user speaks into the device to instruct the device to revise the document. For example, "Delete the third paragraph and add a new main point."

[0430] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[0431] Input: Spoken correction instructions

[0432] Data processing: Converting voice data into text data

[0433] Output: Text data "Delete the third paragraph and add a new point."

[0434] 3. The terminal sends the converted character data to the server.

[0435] Input: Text data "Delete the third paragraph and add a new point."

[0436] Output: Character data sent to the server

[0437] 4. The server again performs natural language processing to analyze the correction instructions.

[0438] Input: Text data "Delete the third paragraph and add a new point."

[0439] Data calculation: Analysis of correction instructions

[0440] Output: Analysis result "Delete the third paragraph and add a new main point"

[0441] 5. The server modifies the document using the generative model.

[0442] Input: Analysis result "Delete third paragraph and add new main point", adjusted document

[0443] Data Computation: Regeneration by Document Correction Algorithms

[0444] Output: The modified document

[0445] Step 8: Final check and output

[0446] 1. The user issues a final confirmation instruction to the terminal. Example: "Please finalize and submit the report."

[0447] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[0448] Input: Final confirmation voice

[0449] Data processing: Converting voice data into text data

[0450] Output: Text data "Please finalize the report and send it."

[0451] 3. The terminal sends the converted character data to the server.

[0452] Input: Text data "Please finalize the report and send it."

[0453] Output: Character data sent to the server

[0454] 4. The server generates the final document and sends it to the terminal.

[0455] Input: Adjusted and amended document

[0456] Output: Final document

[0457] 5. The device will read this final version aloud or display it as text.

[0458] Input: Final document

[0459] Output: Presentation to the user (audio or text display)

[0460] 6. The user performs a final check and, if there are no problems, gives a final voice command. For example, "There are no problems, so please send."

[0461] 7. The server sends the final document to the receiving location specified by the user.

[0462] Input: Final confirmation and sending instructions

[0463] Output: Send the final document to the specified receiving location

[0464] (Application example 2)

[0465] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0466] Modern virtual stores are required to efficiently provide diverse product information and quickly and appropriately address customer questions and concerns. However, conventional systems have difficulty responding flexibly to customer emotions, and have lacked effective means to increase customer satisfaction. In particular, they have been unable to adjust the tone and content of documents generated based on voice input to match the user's emotional state, making it difficult to provide personalized service to customers.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0468] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for customizing the document by referring to documents created in the past by the company, means for correcting the document in response to voice instructions, emotion engine means for analyzing the user's emotions from the voice, means for adjusting the tone of the document according to the user's emotional state, and means for checking and outputting the final version. This makes it possible to automate customer service and the generation of product descriptions in a virtual store and provide personalized services according to the customer's emotions.

[0469] "Voice input means" refers to technology for receiving and processing a user's voice.

[0470] "Text data conversion means" refers to a technology that converts the voice captured by the voice input means into text format.

[0471] "Natural language processing means" refers to technology for analyzing text data and understanding the content of user requests.

[0472] "Generative modeling means" refers to technology that automatically generates documents based on analyzed text data.

[0473] "Document customization means" refers to technology that references a company's previously created documents and adjusts the generated documents in accordance with templates and style guides.

[0474] "Voice instruction means" refers to a technology that receives instructions given by voice from a user and modifies a document based on the contents of the instructions.

[0475] The "emotion engine means" refers to a technology that analyzes emotions from the user's voice and identifies the user's emotional state.

[0476] "Tone adjustment" refers to a technique for adjusting the tone and content of a generated document depending on the user's emotional state.

[0477] "Output means" refers to a technique for providing the final generated document to the user in audio or text format.

[0478] The present invention provides a system for generating documents in response to user requests through voice input and adjusting the tone and content of the documents based on the user's emotional state. The system includes a voice input unit, a character data conversion unit, a natural language processing unit, a generative model unit, a document customization unit, a voice instruction unit, an emotion engine unit, a tone adjustment unit, and an output unit.

[0479] System configuration

[0480] The system consists of the following major hardware and software components:

[0481] Hardware: microphone, computer, display device

[0482] software:

[0483] speech_recognition library for implementing speech recognition technology

[0484] Transformers library for natural language processing of text data

[0485] j-hartmann / emotion-english-distilroberta-base model for sentiment analysis

[0486] The rinna / japanese-gpt2-medium model for text generation

[0487] System Operation Overview

[0488] 1. Voice input and conversion

[0489] The user uses a microphone to provide voice input. The device converts the voice data into text using the speech_recognition library. For example, if the user says, "What are the features of this product?", the device converts the voice data into text: "What are the features of this product?"

[0490] 2. Text analysis and first edition generation

[0491] The server receives the converted text data, performs natural language processing using the transformers library, analyzes the user's request, and then generates the first version of the document using the rinna / japanese-gpt2-medium model based on the analysis results.

[0492] 3. Customization and Sentiment Analysis

[0493] The server references the company's past documents and customizes the generated document according to templates and style guides. At the same time, it analyzes the user's voice using the j-hartmann / emotion-english-distilroberta-base model to detect emotions. If the user's voice is recognized as "tense," it adjusts the tone of the document to be more relaxed.

[0494] 4. Correction instructions and final version generation

[0495] The user gives instructions for corrections by voice through a microphone. The device again converts the voice into text, and the server updates the document based on the instructions. For example, if the user instructs the server to "delete the third paragraph and add a new point," the document is updated accordingly.

[0496] 5. Check and final output

[0497] The user then gives a final confirmation, and the server generates the final document and sends it to the device. The device then reads the document aloud or displays it in text. If the user determines there are no problems, the document is sent to their email address, a shared folder, or other location.

[0498] Specific examples

[0499] The user speaks into the device, saying, "Please tell me about the features of this product." The device converts this speech into text data, and the server analyzes the received text data. A product description is generated using the generative AI model rinna / japanese-gpt2-medium, which is then customized based on the company's past documents. At the same time, if the emotion engine detects "tension" in the user's voice, the tone of the text is adjusted to be more relaxed. For example, it could be adjusted to say, "I will explain the features of this product. This product is high quality and can be used safely. Please relax and enjoy it."

[0500] As a concrete example of a prompt, if a user says, "Please tell me the features of this product," the prompt to the generative AI model would be:

[0501] "Please describe the features of this product as follows: This product is high quality, reliable, and very beneficial to the user."

[0502] In this way, the system of the present invention enables personalized document generation based on voice input and in response to the user's emotional state.

[0503] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0504] Step 1:

[0505] The user provides voice input.

[0506] The user speaks a question or request into the microphone, e.g., "What are the features of this product?". Voice data is obtained as input.

[0507] Step 2:

[0508] The terminal converts the voice data into text data.

[0509] The device receives voice data input from the user and converts it into text data using the speech_recognition library. The input is voice data and the output is text data. The converted text data is "Please tell me the features of this product."

[0510] Step 3:

[0511] The server analyzes the text data and understands the request.

[0512] The server receives the converted text data and performs natural language processing using the transformers library, which then analyzes the user's request. The input is text data, and the output is the analyzed request (e.g., "I want to know the product's features").

[0513] Step 4:

[0514] The server generates a document based on the analysis results.

[0515] The server generates the first version of the document using the rinna / japanese-gpt2-medium model based on the parsed request content. The input is the parsed request content, and the output is the generated text. An example of the generated document is: "This product is high quality, reliable, and very useful to users."

[0516] Step 5:

[0517] The server references the company's past documents and customizes the document.

[0518] The server searches the company's database for previously created documents and retrieves the templates and style guides to be applied. The input is the generated text, and the output is the customized document. An example of a customized document is: "This product is high quality, reliable, and very beneficial to users. Please use it with confidence."

[0519] Step 6:

[0520] The server analyzes the user's emotions from the voice.

[0521] The server uses the user's voice data to perform emotion analysis using the j-hartmann / emotion-english-distilroberta-base model. The input is the user's voice data, and the output is the user's emotional state (e.g., "I'm nervous").

[0522] Step 7:

[0523] The server adjusts the tone of the document based on the user's emotional state.

[0524] The server adjusts the tone of the generated document based on the analyzed emotional state. The input is a customized document and the emotional state, and the output is a tone-adjusted document. An example of an adjusted document is: "This product is high quality and reliable, so you can use it with confidence. Please relax and enjoy it."

[0525] Step 8:

[0526] The user gives correction instructions by voice.

[0527] The user issues correction instructions through a microphone, for example, "Delete the third paragraph and add a new main point." The input is voice data, and the output is the voice data of the correction instructions.

[0528] Step 9:

[0529] The device converts the speech back into text and sends it to the server.

[0530] The device converts the voice instructions for correction into text data and sends it to the server. The input is voice data and the output is text data. Converted text data: "Delete the third paragraph and add a new main point."

[0531] Step 10:

[0532] The server updates the document based on the correction instructions.

[0533] The server receives the text data of the correction instructions and updates the document using natural language processing. The input is the text data of the correction instructions and the output is the updated document.

[0534] Step 11:

[0535] The user gives a final confirmation instruction.

[0536] The user uses the microphone to give a final confirmation instruction, e.g., "There are no problems, so send it." The input is voice data, and the output is the voice data of the final confirmation.

[0537] Step 12:

[0538] The server generates and outputs the final document.

[0539] The server generates a final, verified document and sends it to the terminal, which then reads the final document aloud or displays it as text. The input is the updated document, and the output is the final document.

[0540] In this way, a document generation system based on a user's voice input can analyze the user's request, generate an appropriate document according to the user's emotional state, and ultimately provide a personalized document.

[0541] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0542] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0543] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0544] [Second embodiment]

[0545] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0546] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0548] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0552] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0555] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0556] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0557] The present invention provides a system for generating documents through voice input, thereby reducing eye strain for desk workers. The overall flow of this system and specific embodiments are described below.

[0558] System Overview

[0559] This system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and an output means. A user gives instructions by voice, and the instructions are analyzed through natural language processing, and finally a document is generated and output.

[0560] Program processing overview

[0561] Voice input and conversion

[0562] 1. The user speaks into the terminal to request the creation of materials.

[0563] For example: "I'd like you to prepare a proposal for next week."

[0564] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[0565] Audio data: "Please prepare a proposal for next week."

[0566] Converted text data: "Please prepare a proposal for next week."

[0567] Parsing and Document Generation

[0568] 1. The terminal sends the converted text data to the server.

[0569] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[0570] Analysis result: "Create a proposal for next week"

[0571] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[0572] Customization and Documentation Reference

[0573] 1. The server references previously created documents from the company database and retrieves appropriate templates and style guides.

[0574] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0575] Example: "Format it into the format of a past marketing proposal."

[0576] Correction instructions and final version generation

[0577] 1. The user gives instructions to correct the document by voice into the terminal.

[0578] For example: "Delete the third paragraph and add a new point."

[0579] 2. The device converts this voice instruction into text data and sends it to the server.

[0580] Audio: "Delete the third paragraph and add a new point."

[0581] Converted text: "Delete the third paragraph and add a new point."

[0582] 3. The server receives the text data and modifies it using the generative modeling means.

[0583] Check and output

[0584] 1. The user issues a final confirmation instruction to the terminal.

[0585] Example: "Please finalize and submit the report."

[0586] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[0587] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[0588] Example: "No problem, send it."

[0589] 4. The server sends the final document to the receiving location specified by the user.

[0590] Specific examples

[0591] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[0592] This system allows desk workers to create high-quality documents using only their voice, without looking at a screen, reducing eye strain and supporting a healthier working style.

[0593] The processing flow will be explained below.

[0594] Step 1:

[0595] The user gives instructions for creating materials by voice into the terminal.

[0596] For example: "I'd like you to prepare a proposal for next week."

[0597] Step 2:

[0598] The device captures the user's voice with a microphone.

[0599] Audio data: "Please prepare a proposal for next week."

[0600] Step 3:

[0601] The device uses a speech recognition engine to convert the captured voice into text data.

[0602] Converted text data: "Please prepare a proposal for next week."

[0603] Step 4:

[0604] The terminal transmits the text data to the server.

[0605] Step 5:

[0606] The text data received by the server is analyzed using a natural language processing engine.

[0607] Analysis result: Understand the instruction "Write a proposal."

[0608] Step 6:

[0609] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[0610] A first draft proposal is generated.

[0611] Step 7:

[0612] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[0613] Step 8:

[0614] The server applies templates and style guides to the generated initial version to create customized documents.

[0615] The customized proposal draft is completed.

[0616] Step 9:

[0617] The user gives instructions to correct the document by voice into the terminal.

[0618] For example: "Delete the third paragraph and add a new point."

[0619] Step 10:

[0620] The terminal converts the user's voice into text data and sends it to the server.

[0621] Converted text: "Delete the third paragraph and add a new point."

[0622] Step 11:

[0623] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0624] Step 12:

[0625] The server uses the generative model to modify the document.

[0626] A revised proposal is generated.

[0627] Step 13:

[0628] The user issues a final confirmation instruction by voice to the terminal.

[0629] Example: "Please finalize and submit the report."

[0630] Step 14:

[0631] The server generates the final document and sends it to the terminal.

[0632] The final proposal is sent to the device.

[0633] Step 15:

[0634] Your device will read the final document aloud or display it as text.

[0635] Step 16:

[0636] After the user makes a final confirmation, he or she issues a voice command saying, "There are no problems, so send it."

[0637] Step 17:

[0638] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[0639] For example: a user's email address or a company's shared folder.

[0640] This allows users to create high-quality documents using only their voice, without using their eyes, enabling a healthier way of working.

[0641] Example 1

[0642] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] Today's desk workers often suffer from eye strain and stress due to long hours of screen time. Furthermore, document creation takes a lot of time and effort, making it difficult to carry out work efficiently. For this reason, there is a demand for technology that utilizes voice input to efficiently generate documents and reduce eye strain.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0645] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for referencing previously created documents and customizing the document, means for correcting the document in response to voice instructions, means for confirming and outputting the final version, means for reading the final version aloud or displaying it in text to prompt the user for confirmation, and means for managing the operation of the entire system based on a program flow divided into processing steps. This allows users to efficiently create, correct, confirm, and output documents using only voice input, reducing eye strain and improving work efficiency.

[0646] "Voice input" is an input method in which the system receives voice data spoken by the user.

[0647] "Character data" refers to text data converted from voice input using voice recognition technology.

[0648] "Natural language processing" is a technology that analyzes text data and understands its content and meaning.

[0649] A "generative model" is a technology that generates new documents or content based on input data.

[0650] "Customization" refers to the process of tailoring a generated document to a particular template or historical document style.

[0651] "Voice instructions" refer to instructions for corrections or operations given by the user using voice.

[0652] The "final version" refers to a document that is completed at the final stage of document creation or revision.

[0653] "Output" refers to the process of displaying, printing, or transmitting the generated document.

[0654] "Read aloud" refers to the process by which the system converts text data into audio and plays it aloud for the user.

[0655] "Program flow" refers to the flow of a program that shows the processing procedures and steps of the entire system.

[0656] "Managing the operation of the entire system" refers to the management activities of controlling each function of the system in an integrated manner and operating it efficiently.

[0657] The present invention provides a system for generating documents through voice input to reduce eye strain for desk workers. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, and an output unit.

[0658] Voice input and conversion

[0659] The user requests the creation of a document by voice, and the device captures this voice using a microphone. The captured voice data is converted into text data by a voice recognition engine (for example, Google Cloud Speech-to-Text API). As a result, the voice data "Please create a proposal for next week" is converted into text data "Please create a proposal for next week."

[0660] Parsing and Document Generation

[0661] The device sends the converted text data to the server. The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the text data and understand the user's request. For example, the analysis result may be "Create a proposal for next week." Based on this analysis result, the server launches a generative model (e.g., OpenAI GPT-3) to generate the first version of the document.

[0662] Customization and Documentation Reference

[0663] The server references past documents from a corporate database (e.g., a MySQL database) and retrieves the appropriate templates and style guides. It then uses these templates and style guides to customize the first version of the document, for example, to match the format of a past marketing proposal.

[0664] Correction instructions and final version generation

[0665] The user gives instructions to edit the document by voice, and the device captures the voice data, converts it into text data, and sends it to the server. For example, the instruction might be, "Delete the third paragraph and add a new main point." The server analyzes this and invokes the generative model to edit the document.

[0666] Check and output

[0667] The user gives a voice command for final confirmation, and the server generates a final version of the document and sends it to the terminal. The terminal reads this final version aloud or displays it as text. The user checks it, and if there are no problems, gives a voice command for final transmission. For example, "There are no problems, so send it." The server sends the final version of the document to the specified receiving location.

[0668] Specific examples

[0669] The specific operation flow is shown below.

[0670] The user says, "I want you to create a marketing strategy report for next week." The device converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says, "It's fine, so send it," and the server sends the final version to the specified receiving location.

[0671] This system allows users to efficiently create high-quality documents through voice input and work while reducing eye strain.

[0672] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0673] Step 1: Getting voice input

[0674] The user requests the creation of a document by voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the user speaks into the device's microphone, saying, "I want you to create a proposal for next week."

[0675] Step 2: Capture audio

[0676] The terminal uses a microphone to capture the user's voice. The input is the user's voice data, and the output is the captured voice file. Specifically, the terminal collects voice data in real time.

[0677] Step 3: Convert audio data to text

[0678] The device uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice data into text data. The input is the captured voice data, and the output is text data. Specifically, the device converts voice data such as "Please write a proposal for next week" into text data such as "Please write a proposal for next week."

[0679] Step 4: Sending text data

[0680] The terminal sends the converted text data to the server. The input is text data, and the output is the text data sent to the server. Specifically, the text data is encrypted and transmitted to the server via the Internet.

[0681] Step 5: Parsing the text

[0682] The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the received text data. The input is the received text data, and the output is the analysis result. Specifically, it analyzes the text "Please create a proposal for next week" and extracts the intent "to create a proposal for next week."

[0683] Step 6: Generate the first version of the document

[0684] Based on the analysis results, the server launches a generative AI model (e.g., OpenAI GPT-3) to generate the first version of the document. The input is the analysis results, and the output is the first version of the document. Specifically, the prompt "Create a proposal for next week" is input into the generative AI model, which then generates the first version of the document.

[0685] Step 7: Referencing past documents

[0686] The server references past documents from the company's database (e.g., MySQL database). The input is a request to reference a past document, and the output is the past document data. Specifically, it uses an SQL query to retrieve templates of past marketing proposals.

[0687] Step 8: Customize the initial document

[0688] The server customizes the first version of the document using the acquired templates and style guides. The input is the first version of the document and past document data, and the output is a customized document. Specifically, the first version of the document is formatted to match the format of past marketing proposals.

[0689] Step 9: Speak correction instructions

[0690] The user gives instructions to edit the document by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Delete the third paragraph and add a new main point."

[0691] Step 10: Convert correction instructions to text

[0692] The terminal captures the voice of the correction instruction and converts it into text data using a voice recognition engine. The input is the captured voice data, and the output is text data. Specifically, the voice data is converted into text and sent to the server.

[0693] Step 11: Modifying the document

[0694] The server analyzes the received text data and uses a generative AI model to correct the document. The input is the text data of the correction instructions, and the output is the corrected document. Specifically, the document is regenerated using the AI ​​model based on the correction instructions.

[0695] Step 12: Speak your final confirmation instructions

[0696] The user gives instructions for final confirmation by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Please finalize the report and send it."

[0697] Step 13: Generate and send the final version

[0698] The server generates the final document and sends it to the terminal. The input is the revised document, and the output is the final document. The terminal reads this final version aloud or displays it as text. Specifically, the server generates a final document file and transfers it to the terminal for display or aloud reading.

[0699] Step 14: Send document instructions

[0700] If there are no problems, the user gives the final command for transmission by voice. The input is the user's voice data, and the output is the captured voice data. The specific operation is to say, "There are no problems, so send it."

[0701] Step 15: Final submission of document

[0702] The server sends the final document to the receiving location specified by the user. The input is the final sending instruction and the final document, and the output is the document sent to the receiving location. The specific operation is to upload the final version to a mail server or cloud storage.

[0703] (Application example 1)

[0704] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Employees in brick-and-mortar stores are required to provide fast, high-quality customer service and customer service on a daily basis. However, achieving this requires detailed manuals and guidelines, and creating them takes time and effort. Furthermore, employees may experience eye strain from looking at a screen for long periods of time, which can have a negative impact on their productivity and health. To solve these issues, a system is needed that uses voice input to automatically generate customer service manuals quickly and efficiently.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0707] In this invention, the server includes means for accepting voice input and converting the user's voice into text data, natural language processing means for analyzing the text data and understanding the request content, generative model means for generating documents based on the generated text data, means for referencing past documents and customizing the documents, and means for automatically generating daily work manuals and service guides. This enables store employees to quickly and accurately generate customer service manuals using only voice input, improving productivity and reducing eye strain.

[0708] "Voice input" is the process of receiving a user's voice as digital data and converting it into text data.

[0709] "Natural language processing means" is a technology that analyzes input text data, understands and processes human language, and is used to recognize the content of a user's request.

[0710] "Generative model means" refers to algorithms or models that automatically generate new documents based on analyzed text data.

[0711] "Customization" is the process of referencing previous documents and adjusting the generated document based on them to fit a specific format or style.

[0712] "Voice instruction" refers to the act of a user giving voice instructions to the system to perform a specific operation or correction.

[0713] The "means for modifying a document" refers to a function for editing an existing document and making changes or additions based on the user's voice instructions.

[0714] "Final" refers to the final document after all modifications and customizations have been completed.

[0715] "Means for outputting" refers to a function for presenting the final document to the user or transmitting it to a specified location.

[0716] "Means for automatically generating daily work manuals and service guides" refers to a process for automatically creating and providing instructions and explanations required for daily work and customer service.

[0717] This invention is a system that automatically generates customer service manuals and service guides for brick-and-mortar stores using voice input. The system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and a final version confirmation and output means.

[0718] System Configuration

[0719] Hardware

[0720] Microphone: A device for capturing audio input.

[0721] Computer or smartphone: a device that processes and stores data.

[0722] software

[0723] speech_recognition library: A library for converting speech data into text.

[0724] Transformers library and general-purpose generative models: A library for natural language processing and text generation. Specifically, it uses the Rinna / Japanese GPT-2 model.

[0725] Data processing and calculation

[0726] Voice input and conversion

[0727] The server captures the voice data that the user speaks into the microphone. This voice data is converted into text data using the speech_recognition library. For example, if the user instructs, "Create a manual to explain to customers about today's special menu," the voice is converted into text.

[0728] Parsing and Document Generation

[0729] The server receives the converted text data and analyzes it using natural language processing. Based on the analysis results, it activates a generative model to generate the first edition of the customer service manual. As a generative model, it uses the transformers library to activate the Japanese GPT-2 model.

[0730] Customization

[0731] The server references past customer service manuals and service guides from the company's database and formats the generated documents into appropriate templates and style guides, thereby ensuring consistency in the documents.

[0732] Corrections and final check

[0733] The user can make corrections to the generated document by issuing additional voice commands. For example, they can say, "Update the second paragraph with the latest trend information." This voice command is also converted into text by the speech_recognition library, and the document is corrected by the server. Finally, the user confirms and sends the document by saying, "There are no problems, so please send it."

[0734] output

[0735] The server generates the final document and presents it to the user, who then confirms it and sends it to the specified location.

[0736] Examples and prompts

[0737] For example, if an employee at a physical store says, "Please create a manual to explain to customers about today's special menu items," the speech is converted into text, and natural language processing is performed based on that text data to generate the manual. The employee then gives additional instructions, such as "Update the second paragraph with new trend information," and finally instructs, "There are no problems, so please submit," and the completed manual is generated.

[0738] (Example of a prompt)

[0739] "Create a manual to explain today's special menu to customers."

[0740] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0741] Step 1:

[0742] The user verbally requests the creation of a document into the microphone. For example, "Create a manual explaining today's special menu to customers." The input is the user's voice data. The output is the captured voice data.

[0743] Step 2:

[0744] The device converts this voice data into text data using a speech recognition engine (speech_recognition library). The input is the captured voice data, specifically voice waveform data. The device obtains text data by inputting the voice waveform data into the voice recognition engine. The output is the converted text data. For example, the voice data "Please create a manual to explain to customers about today's special menu" is converted into text data "Please create a manual to explain to customers about today's special menu."

[0745] Step 3:

[0746] The terminal sends this text data to the server. The input is the converted text data, and the output is the text data sent to the server.

[0747] Step 4:

[0748] The server analyzes the received text data using natural language processing means to understand the user's request. The input is text data, specifically the aforementioned text data "Please create a manual to explain to customers about today's special menu." The server uses natural language processing technology to analyze the request. The output is the analyzed request, for example, "Request to create a manual explaining the special menu."

[0749] Step 5:

[0750] The server invokes the generative model based on the analysis results to generate the first version of the document. The input is the parsed request content, and the output is the generated first version of the document. Specifically, the document is generated using the transformers library and a generative AI model (Japanese GPT-2).

[0751] Step 6:

[0752] The server references previously created documents from the company's database and customizes the generated document. The input is the first version of the generated document and the referenced previous document data. The server uses this information to adjust the format and style of the document. The output is a customized document.

[0753] Step 7:

[0754] The user speaks into the terminal to instruct the device to edit a document. For example, they say, "Update the second paragraph with the new trend information." The input is the voice data of the user's edit instructions. The output is the captured voice data.

[0755] Step 8:

[0756] The terminal converts this voice correction instruction into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "Update the second paragraph with the new trend information."

[0757] Step 9:

[0758] The server receives the text data and modifies the document using the generative model means. The input is the text data of the user's modification instructions, and the output is the modified document. Specifically, the server uses the generative model means to update the document based on the instructions.

[0759] Step 10:

[0760] The user gives a final confirmation instruction. For example, they say, "There's no problem, so send it." The input is the user's voice data of the final confirmation. The output is the captured voice data.

[0761] Step 11:

[0762] The terminal converts this final confirmation voice into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "There are no problems, so please send it."

[0763] Step 12:

[0764] The server generates the final document and sends it to the terminal. The input is the final confirmation instruction text data and the customized / modified document. The server generates the final document based on this. The output is the final document. The terminal reads this final version aloud or displays it in text.

[0765] Step 13:

[0766] The user performs a final check and, if there are no problems, gives a final voice command. The server sends the final document to the specified receiving location. The input is the user's voice command, and the output is the final document sent to the destination.

[0767] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0768] The present invention is a system for generating documents through voice input and adjusting the tone and content of the documents taking into account the user's emotional state. The overall flow of this system and specific embodiments are described below.

[0769] System Overview

[0770] The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit. A user gives an instruction by voice, the instruction is analyzed through natural language processing, and finally a document is generated and output based on the user's emotional state.

[0771] Program processing overview

[0772] Voice input and conversion

[0773] 1. The user speaks into the terminal to request the creation of materials.

[0774] For example: "I'd like you to prepare a proposal for next week."

[0775] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[0776] Audio data: "Please prepare a proposal for next week."

[0777] Converted text data: "Please prepare a proposal for next week."

[0778] Parsing and Document Generation

[0779] 1. The terminal sends the converted text data to the server.

[0780] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[0781] Analysis result: "Creating a proposal"

[0782] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[0783] A first draft proposal is generated.

[0784] Customization and Emotion Recognition

[0785] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[0786] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0787] Example: "Format it into the format of a past marketing proposal."

[0788] 3. The server uses the emotion engine means to recognize emotions from the user's voice. For example, emotions such as "tense" or "relaxed" are recognized from the user's voice.

[0789] 4. The server adjusts the tone and content of the generated document based on the user's emotions.

[0790] Example: If a user is "nervous," adjust the tone of the document to be more relaxing.

[0791] Correction instructions and final version generation

[0792] 1. The user gives instructions to correct the document by voice into the terminal.

[0793] For example: "Delete the third paragraph and add a new point."

[0794] 2. The device converts this voice instruction into text data and sends it to the server.

[0795] Converted text: "Delete the third paragraph and add a new point."

[0796] 3. The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0797] 4. The server modifies the document using the generative model means.

[0798] A revised proposal is generated.

[0799] Check and output

[0800] 1. The user issues a final confirmation instruction to the terminal.

[0801] Example: "Please finalize and submit the report."

[0802] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[0803] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[0804] Example: "No problem, send it."

[0805] 4. The server sends the final document to the receiving location specified by the user.

[0806] For example: a user's email address or a company's shared folder.

[0807] Specific examples

[0808] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats it into an appropriate template. At the same time, an emotion engine is used to analyze the emotions from the user's voice, and if it determines, for example, that the user is nervous, the tone of the document is softened. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[0809] This system not only allows desk workers to create high-quality documents using only their voice without using their eyes, but also provides documents with an appropriate tone and content depending on the user's emotional state, creating a less stressful work environment.

[0810] The processing flow will be explained below.

[0811] Step 1:

[0812] The user gives instructions for creating materials by voice into the terminal.

[0813] For example: "I'd like you to prepare a proposal for next week."

[0814] Step 2:

[0815] The device captures the user's voice with a microphone.

[0816] Audio data: "Please prepare a proposal for next week."

[0817] Step 3:

[0818] The device uses a speech recognition engine to convert the captured voice into text data.

[0819] Converted text data: "Please prepare a proposal for next week."

[0820] Step 4:

[0821] The terminal transmits the text data to the server.

[0822] Step 5:

[0823] The text data received by the server is analyzed using a natural language processing engine.

[0824] Analysis result: Understand the instruction "Write a proposal."

[0825] Step 6:

[0826] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[0827] A first draft proposal is generated.

[0828] Step 7:

[0829] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[0830] Step 8:

[0831] The server applies templates and style guides to the generated initial version to create customized documents.

[0832] The customized proposal draft is completed.

[0833] Step 9:

[0834] The server uses an emotion engine to recognize emotions from the user's voice.

[0835] For example, recognizing emotional states such as "tense," "relaxed," and "excited."

[0836] Step 10:

[0837] The server adjusts the tone and content of the generated document based on the emotion recognition results.

[0838] For example, if the user is "tense," adjust the tone of the document to be softer.

[0839] Step 11:

[0840] The user gives instructions to correct the document by voice into the terminal.

[0841] For example: "Delete the third paragraph and add a new point."

[0842] Step 12:

[0843] The terminal converts the user's voice into text data and sends it to the server.

[0844] Converted text: "Delete the third paragraph and add a new point."

[0845] Step 13:

[0846] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[0847] Step 14:

[0848] The server uses the generative model to modify the document.

[0849] A revised proposal is generated.

[0850] Step 15:

[0851] The user issues a final confirmation instruction by voice to the terminal.

[0852] Example: "Please finalize and submit the report."

[0853] Step 16:

[0854] The server generates the final document and sends it to the terminal.

[0855] The final proposal is sent to the device.

[0856] Step 17:

[0857] Your device will read the final document aloud or display it as text.

[0858] Step 18:

[0859] After the user makes a final check, he or she issues a voice command saying, "There are no problems, so send it."

[0860] Step 19:

[0861] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[0862] For example: a user's email address or a company's shared folder.

[0863] This allows users to create high-quality documents using only their voice, without using their eyes, and provides the optimal tone and content according to their emotional state, creating a less stressful work environment.

[0864] Example 2

[0865] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0866] In conventional document creation systems, when users give instructions by voice, they not only convert the voice input into text data, but also lack the functionality to generate and edit documents taking into account the user's emotions and the tone of the document. As a result, even when users give instructions that convey their emotions, these are not easily reflected in the document, resulting in a poor user experience. In addition, the functionality to apply templates by referencing past documents is insufficient, which makes it difficult to efficiently generate documents with a consistent feel.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for accepting voice input and converting the user's voice into character data; natural language processing means for analyzing the character data and understanding the requested content; generative model means for generating a document based on the generated character data; means for customizing the document by referring to previously created documents; means for recognizing emotions from the user's voice and adjusting the tone and content of the document; means for correcting the document in response to voice instructions; and means for checking and outputting the final version. This enables high-quality document generation and correction that reflects the user's voice and emotions, thereby improving the user experience and realizing efficient document creation.

[0868] "Voice input" is a technology that captures what a user says as voice data.

[0869] "Character data" refers to data obtained by converting voice data into characters.

[0870] "Natural language processing means" is a technology that analyzes character data and understands the content of a user's request.

[0871] A "generative model means" is an algorithm that generates a document based on input data.

[0872] "Previously created documents" are documents created in the past that are used as templates or reference materials.

[0873] "Means of customization" refers to techniques for modifying and adjusting generated documents in accordance with specific style guides and templates.

[0874] "Means for recognizing emotions" refers to technology that analyzes and recognizes the emotional state of a user from their voice.

[0875] "Means for adjusting the tone and content of a document" refers to techniques for changing the expression and content of a document based on the user's emotions and needs.

[0876] "Voice instructions" are instructions or commands about a document given by the user through voice.

[0877] "Means for modifying documents" refers to technology that adds, deletes, or modifies existing documents based on voice instructions.

[0878] The "means for checking and outputting the final version" is a technique for outputting a document that has been finally checked by a user.

[0879] MODE FOR CARRYING OUT THE INVENTION

[0880] The present invention provides a system for generating documents through voice input and adjusting the tone and content of the documents in consideration of a user's emotional state. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit.

[0881] Hardware and Software Configuration

[0882] This system is configured using the following hardware and software.

[0883] 1. Device:

[0884] Audio capture device (e.g. microphone)

[0885] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[0886] 2. Server:

[0887] Natural language processing tools (e.g., spaCy, BERT)

[0888] Generative modeling tools (e.g., OpenAI GPT-4)

[0889] Database search methods (e.g., SQL databases)

[0890] Emotion engine means (e.g. IBM Watson Tone Analyzer)

[0891] Program processing flow

[0892] When a user speaks, the speech is captured by the device and converted into text using a speech recognition engine. The text data is then sent to a server, where the user's request is analyzed using natural language processing means. Based on the analysis results, a generative model means is activated to generate a first version of the document. The server then searches for past documents in the company's database and customizes the document by applying appropriate templates and style guides. Next, an emotion engine means is used to recognize emotions from the user's voice and adjust the tone and content of the document based on the emotions.

[0893] If the user makes voice corrections to the document, the voice is converted into text again and sent to the server for analysis and generation. Finally, the user gives a final check, and if there are no problems, the server outputs the final version of the document.

[0894] Explanation with concrete examples

[0895] For example, if a user says to a terminal, "I want you to write a marketing strategy report for next week," the terminal captures this speech and uses a speech recognition engine to convert it into text data saying, "I want you to write a marketing strategy report for next week." This text data is sent to a server and analyzed by natural language processing means. Based on the analysis results, a generative model means creates a first edition of a marketing strategy report. The server then references past marketing reports and applies an appropriate template. At the same time, the emotion engine means recognizes emotions from the user's voice, and if it determines, for example, that the user is nervous, it adjusts the tone of the document to be softer.

[0896] When the user instructs, "Update the second paragraph with the new trend information," the device again captures the voice, converts it into text data, and sends it to the server. The server makes the corrections and generates a document containing the new trend information. Finally, when the user instructs, "There are no problems, so send it," the server sends the final document to the specified recipient (e.g., email address or shared folder).

[0897] Prompt Sentence Examples

[0898] User: "I want a marketing strategy report for next week."

[0899] Device: "Converting speech to text"

[0900] Server: "Generating first version"

[0901] Server: "Applying template by referencing past reports"

[0902] Server: "Analyzing user sentiment"

[0903] Server: "Adjusting the tone of the document"

[0904] User: "Update the second paragraph with new trending information."

[0905] Device: "Converting speech to text"

[0906] Server: "Generating updated document"

[0907] User: "No problem, send it."

[0908] Server: "Sending final version to specified location"

[0909] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0910] Step 1: Capture and convert audio input

[0911] 1. The user gives instructions through a voice input device. Example: "Please prepare a proposal for next week."

[0912] 2. The device captures the user's voice using an audio capture device (e.g., a microphone).

[0913] Input: User's voice data

[0914] 3. The device converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0915] Data processing: Converting voice data into text data

[0916] Output: Text data "Please create a proposal for next week."

[0917] Step 2: Sending text data

[0918] 1. The terminal sends the converted character data to the server.

[0919] Input: Text data "Please create a proposal for next week."

[0920] Output: Character data sent to the server

[0921] Step 3: Analyzing the text data

[0922] 1. The server analyzes the received text data using natural language processing tools (e.g., spaCy, BERT) to understand the user's request.

[0923] Input: Text data "Please create a proposal for next week."

[0924] Data Computing: Analyzing Request Contents Using Natural Language Processing

[0925] Output: Analysis result "Creating a proposal"

[0926] Step 4: Generate the initial document

[0927] 1. The server launches a generative model (e.g., OpenAI GPT-4) based on the analysis results to generate the first version of the document.

[0928] Input: Analysis result "Create a proposal"

[0929] Data Computation: First Edition Document Generation with Document Generation Algorithms

[0930] Output: First edition proposal

[0931] Step 5: Apply and customize the template

[0932] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[0933] Input: Company database

[0934] Data processing: Searching and retrieving past documents

[0935] Output: Templates and style guides

[0936] 2. The server customizes the generated first version using the retrieved templates and style guides.

[0937] Input: First edition proposals, templates and style guides

[0938] Data calculations: Customization based on template application and style guides

[0939] Output: Customized document

[0940] Step 6: Emotional awareness and tone adjustment

[0941] 1. The server recognizes emotions from the user's voice using an emotion engine means (e.g., IBM Watson Tone Analyzer).

[0942] Input: User's voice data

[0943] Data Computing: Sentiment Analysis

[0944] Output: Emotional state (e.g., "I'm nervous")

[0945] 2. The server adjusts the tone and content of the generated document based on the user's emotions.

[0946] Input: Emotional state "Tense", customized document

[0947] Data operations: tone and content adjustment

[0948] Output: Adjusted document

[0949] Step 7: Processing correction instructions

[0950] 1. The user speaks into the device to instruct the device to revise the document. For example, "Delete the third paragraph and add a new main point."

[0951] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[0952] Input: Spoken correction instructions

[0953] Data processing: Converting voice data into text data

[0954] Output: Text data "Delete the third paragraph and add a new point."

[0955] 3. The terminal sends the converted character data to the server.

[0956] Input: Text data "Delete the third paragraph and add a new point."

[0957] Output: Character data sent to the server

[0958] 4. The server again performs natural language processing to analyze the correction instructions.

[0959] Input: Text data "Delete the third paragraph and add a new point."

[0960] Data calculation: Analysis of correction instructions

[0961] Output: Analysis result "Delete the third paragraph and add a new main point"

[0962] 5. The server modifies the document using the generative model.

[0963] Input: Analysis result "Delete third paragraph and add new main point", adjusted document

[0964] Data Computation: Regeneration by Document Correction Algorithms

[0965] Output: The modified document

[0966] Step 8: Final check and output

[0967] 1. The user issues a final confirmation instruction to the terminal. Example: "Please finalize and submit the report."

[0968] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[0969] Input: Final confirmation voice

[0970] Data processing: Converting voice data into text data

[0971] Output: Text data "Please finalize the report and send it."

[0972] 3. The terminal sends the converted character data to the server.

[0973] Input: Text data "Please finalize the report and send it."

[0974] Output: Character data sent to the server

[0975] 4. The server generates the final document and sends it to the terminal.

[0976] Input: Adjusted and amended document

[0977] Output: Final document

[0978] 5. The device will read this final version aloud or display it as text.

[0979] Input: Final document

[0980] Output: Presentation to the user (audio or text display)

[0981] 6. The user performs a final check and, if there are no problems, gives a final voice command. For example, "There are no problems, so please send."

[0982] 7. The server sends the final document to the receiving location specified by the user.

[0983] Input: Final confirmation and sending instructions

[0984] Output: Send the final document to the specified receiving location

[0985] (Application example 2)

[0986] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0987] Modern virtual stores are required to efficiently provide diverse product information and quickly and appropriately address customer questions and concerns. However, conventional systems have difficulty responding flexibly to customer emotions, and have lacked effective means to increase customer satisfaction. In particular, they have been unable to adjust the tone and content of documents generated based on voice input to match the user's emotional state, making it difficult to provide personalized service to customers.

[0988] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0989] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for customizing the document by referring to documents created in the past by the company, means for correcting the document in response to voice instructions, emotion engine means for analyzing the user's emotions from the voice, means for adjusting the tone of the document according to the user's emotional state, and means for checking and outputting the final version. This makes it possible to automate customer service and the generation of product descriptions in a virtual store and provide personalized services according to the customer's emotions.

[0990] "Voice input means" refers to technology for receiving and processing a user's voice.

[0991] "Text data conversion means" refers to a technology that converts the voice captured by the voice input means into text format.

[0992] "Natural language processing means" refers to technology for analyzing text data and understanding the content of user requests.

[0993] "Generative modeling means" refers to technology that automatically generates documents based on analyzed text data.

[0994] "Document customization means" refers to technology that references a company's previously created documents and adjusts the generated documents in accordance with templates and style guides.

[0995] "Voice instruction means" refers to a technology that receives instructions given by voice from a user and modifies a document based on the contents of the instructions.

[0996] The "emotion engine means" refers to a technology that analyzes emotions from the user's voice and identifies the user's emotional state.

[0997] "Tone adjustment" refers to a technique for adjusting the tone and content of a generated document depending on the user's emotional state.

[0998] "Output means" refers to a technique for providing the final generated document to the user in audio or text format.

[0999] The present invention provides a system for generating documents in response to user requests through voice input and adjusting the tone and content of the documents based on the user's emotional state. The system includes a voice input unit, a character data conversion unit, a natural language processing unit, a generative model unit, a document customization unit, a voice instruction unit, an emotion engine unit, a tone adjustment unit, and an output unit.

[1000] System configuration

[1001] The system consists of the following major hardware and software components:

[1002] Hardware: microphone, computer, display device

[1003] software:

[1004] speech_recognition library for implementing speech recognition technology

[1005] Transformers library for natural language processing of text data

[1006] j-hartmann / emotion-english-distilroberta-base model for sentiment analysis

[1007] The rinna / japanese-gpt2-medium model for text generation

[1008] System Operation Overview

[1009] 1. Voice input and conversion

[1010] The user uses a microphone to provide voice input. The device converts the voice data into text using the speech_recognition library. For example, if the user says, "What are the features of this product?", the device converts the voice data into text: "What are the features of this product?"

[1011] 2. Text analysis and first edition generation

[1012] The server receives the converted text data, performs natural language processing using the transformers library, analyzes the user's request, and then generates the first version of the document using the rinna / japanese-gpt2-medium model based on the analysis results.

[1013] 3. Customization and Sentiment Analysis

[1014] The server references the company's past documents and customizes the generated document according to templates and style guides. At the same time, it analyzes the user's voice using the j-hartmann / emotion-english-distilroberta-base model to detect emotions. If the user's voice is recognized as "tense," it adjusts the tone of the document to be more relaxed.

[1015] 4. Correction instructions and final version generation

[1016] The user gives instructions for corrections by voice through a microphone. The device again converts the voice into text, and the server updates the document based on the instructions. For example, if the user instructs the server to "delete the third paragraph and add a new point," the document is updated accordingly.

[1017] 5. Check and final output

[1018] The user then gives a final confirmation, and the server generates the final document and sends it to the device. The device then reads the document aloud or displays it in text. If the user determines there are no problems, the document is sent to their email address, a shared folder, or other location.

[1019] Specific examples

[1020] The user speaks into the device, saying, "Please tell me about the features of this product." The device converts this speech into text data, and the server analyzes the received text data. A product description is generated using the generative AI model rinna / japanese-gpt2-medium, which is then customized based on the company's past documents. At the same time, if the emotion engine detects "tension" in the user's voice, the tone of the text is adjusted to be more relaxed. For example, it could be adjusted to say, "I will explain the features of this product. This product is high quality and can be used safely. Please relax and enjoy it."

[1021] As a concrete example of a prompt, if a user says, "Please tell me the features of this product," the prompt to the generative AI model would be:

[1022] "Please describe the features of this product as follows: This product is high quality, reliable, and very beneficial to the user."

[1023] In this way, the system of the present invention enables personalized document generation based on voice input and in response to the user's emotional state.

[1024] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1025] Step 1:

[1026] The user provides voice input.

[1027] The user speaks a question or request into the microphone, e.g., "What are the features of this product?". Voice data is obtained as input.

[1028] Step 2:

[1029] The terminal converts the voice data into text data.

[1030] The device receives voice data input from the user and converts it into text data using the speech_recognition library. The input is voice data and the output is text data. The converted text data is "Please tell me the features of this product."

[1031] Step 3:

[1032] The server analyzes the text data and understands the request.

[1033] The server receives the converted text data and performs natural language processing using the transformers library, which then analyzes the user's request. The input is text data, and the output is the analyzed request (e.g., "I want to know the product's features").

[1034] Step 4:

[1035] The server generates a document based on the analysis results.

[1036] The server generates the first version of the document using the rinna / japanese-gpt2-medium model based on the parsed request content. The input is the parsed request content, and the output is the generated text. An example of the generated document is: "This product is high quality, reliable, and very useful to users."

[1037] Step 5:

[1038] The server references the company's past documents and customizes the document.

[1039] The server searches the company's database for previously created documents and retrieves the templates and style guides to be applied. The input is the generated text, and the output is the customized document. An example of a customized document is: "This product is high quality, reliable, and very beneficial to users. Please use it with confidence."

[1040] Step 6:

[1041] The server analyzes the user's emotions from the voice.

[1042] The server uses the user's voice data to perform emotion analysis using the j-hartmann / emotion-english-distilroberta-base model. The input is the user's voice data, and the output is the user's emotional state (e.g., "I'm nervous").

[1043] Step 7:

[1044] The server adjusts the tone of the document based on the user's emotional state.

[1045] The server adjusts the tone of the generated document based on the analyzed emotional state. The input is a customized document and the emotional state, and the output is a tone-adjusted document. An example of an adjusted document is: "This product is high quality and reliable, so you can use it with confidence. Please relax and enjoy it."

[1046] Step 8:

[1047] The user gives correction instructions by voice.

[1048] The user issues correction instructions through a microphone, for example, "Delete the third paragraph and add a new main point." The input is voice data, and the output is the voice data of the correction instructions.

[1049] Step 9:

[1050] The device converts the speech back into text and sends it to the server.

[1051] The device converts the voice instructions for correction into text data and sends it to the server. The input is voice data and the output is text data. Converted text data: "Delete the third paragraph and add a new main point."

[1052] Step 10:

[1053] The server updates the document based on the correction instructions.

[1054] The server receives the text data of the correction instructions and updates the document using natural language processing. The input is the text data of the correction instructions and the output is the updated document.

[1055] Step 11:

[1056] The user gives a final confirmation instruction.

[1057] The user uses the microphone to give a final confirmation instruction, e.g., "There are no problems, so send it." The input is voice data, and the output is the voice data of the final confirmation.

[1058] Step 12:

[1059] The server generates and outputs the final document.

[1060] The server generates a final, verified document and sends it to the terminal, which then reads the final document aloud or displays it as text. The input is the updated document, and the output is the final document.

[1061] In this way, a document generation system based on a user's voice input can analyze the user's request, generate an appropriate document according to the user's emotional state, and ultimately provide a personalized document.

[1062] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1063] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1064] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1065] [Third embodiment]

[1066] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1067] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1068] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1069] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1070] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1071] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1072] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1073] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1074] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1075] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1076] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1077] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1078] The present invention provides a system for generating documents through voice input, thereby reducing eye strain for desk workers. The overall flow of this system and specific embodiments are described below.

[1079] System Overview

[1080] This system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and an output means. A user gives instructions by voice, and the instructions are analyzed through natural language processing, and finally a document is generated and output.

[1081] Program processing overview

[1082] Voice input and conversion

[1083] 1. The user speaks into the terminal to request the creation of materials.

[1084] For example: "I'd like you to prepare a proposal for next week."

[1085] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[1086] Audio data: "Please prepare a proposal for next week."

[1087] Converted text data: "Please prepare a proposal for next week."

[1088] Parsing and Document Generation

[1089] 1. The terminal sends the converted text data to the server.

[1090] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[1091] Analysis result: "Create a proposal for next week"

[1092] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[1093] Customization and Documentation Reference

[1094] 1. The server references previously created documents from the company database and retrieves appropriate templates and style guides.

[1095] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1096] Example: "Format it into the format of a past marketing proposal."

[1097] Correction instructions and final version generation

[1098] 1. The user gives instructions to correct the document by voice into the terminal.

[1099] For example: "Delete the third paragraph and add a new point."

[1100] 2. The device converts this voice instruction into text data and sends it to the server.

[1101] Audio: "Delete the third paragraph and add a new point."

[1102] Converted text: "Delete the third paragraph and add a new point."

[1103] 3. The server receives the text data and modifies it using the generative modeling means.

[1104] Check and output

[1105] 1. The user issues a final confirmation instruction to the terminal.

[1106] Example: "Please finalize and submit the report."

[1107] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[1108] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[1109] Example: "No problem, send it."

[1110] 4. The server sends the final document to the receiving location specified by the user.

[1111] Specific examples

[1112] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[1113] This system allows desk workers to create high-quality documents using only their voice, without looking at a screen, reducing eye strain and supporting a healthier working style.

[1114] The processing flow will be explained below.

[1115] Step 1:

[1116] The user gives instructions for creating materials by voice into the terminal.

[1117] For example: "I'd like you to prepare a proposal for next week."

[1118] Step 2:

[1119] The device captures the user's voice with a microphone.

[1120] Audio data: "Please prepare a proposal for next week."

[1121] Step 3:

[1122] The device uses a speech recognition engine to convert the captured voice into text data.

[1123] Converted text data: "Please prepare a proposal for next week."

[1124] Step 4:

[1125] The terminal transmits the text data to the server.

[1126] Step 5:

[1127] The text data received by the server is analyzed using a natural language processing engine.

[1128] Analysis result: Understand the instruction "Write a proposal."

[1129] Step 6:

[1130] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[1131] A first draft proposal is generated.

[1132] Step 7:

[1133] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[1134] Step 8:

[1135] The server applies templates and style guides to the generated initial version to create customized documents.

[1136] The customized proposal draft is completed.

[1137] Step 9:

[1138] The user gives instructions to correct the document by voice into the terminal.

[1139] For example: "Delete the third paragraph and add a new point."

[1140] Step 10:

[1141] The terminal converts the user's voice into text data and sends it to the server.

[1142] Converted text: "Delete the third paragraph and add a new point."

[1143] Step 11:

[1144] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1145] Step 12:

[1146] The server uses the generative model to modify the document.

[1147] A revised proposal is generated.

[1148] Step 13:

[1149] The user issues a final confirmation instruction by voice to the terminal.

[1150] Example: "Please finalize and submit the report."

[1151] Step 14:

[1152] The server generates the final document and sends it to the terminal.

[1153] The final proposal is sent to the device.

[1154] Step 15:

[1155] Your device will read the final document aloud or display it as text.

[1156] Step 16:

[1157] After the user makes a final confirmation, he or she issues a voice command saying, "There are no problems, so send it."

[1158] Step 17:

[1159] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[1160] For example: a user's email address or a company's shared folder.

[1161] This allows users to create high-quality documents using only their voice, without using their eyes, enabling a healthier way of working.

[1162] Example 1

[1163] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] Today's desk workers often suffer from eye strain and stress due to long hours of screen time. Furthermore, document creation takes a lot of time and effort, making it difficult to carry out work efficiently. For this reason, there is a demand for technology that utilizes voice input to efficiently generate documents and reduce eye strain.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1166] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for referencing previously created documents and customizing the document, means for correcting the document in response to voice instructions, means for confirming and outputting the final version, means for reading the final version aloud or displaying it in text to prompt the user for confirmation, and means for managing the operation of the entire system based on a program flow divided into processing steps. This allows users to efficiently create, correct, confirm, and output documents using only voice input, reducing eye strain and improving work efficiency.

[1167] "Voice input" is an input method in which the system receives voice data spoken by the user.

[1168] "Character data" refers to text data converted from voice input using voice recognition technology.

[1169] "Natural language processing" is a technology that analyzes text data and understands its content and meaning.

[1170] A "generative model" is a technology that generates new documents or content based on input data.

[1171] "Customization" refers to the process of tailoring a generated document to a particular template or historical document style.

[1172] "Voice instructions" refer to instructions for corrections or operations given by the user using voice.

[1173] The "final version" refers to a document that is completed at the final stage of document creation or revision.

[1174] "Output" refers to the process of displaying, printing, or transmitting the generated document.

[1175] "Read aloud" refers to the process by which the system converts text data into audio and plays it aloud for the user.

[1176] "Program flow" refers to the flow of a program that shows the processing procedures and steps of the entire system.

[1177] "Managing the operation of the entire system" refers to the management activities of controlling each function of the system in an integrated manner and operating it efficiently.

[1178] The present invention provides a system for generating documents through voice input to reduce eye strain for desk workers. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, and an output unit.

[1179] Voice input and conversion

[1180] The user requests the creation of a document by voice, and the device captures this voice using a microphone. The captured voice data is converted into text data by a voice recognition engine (for example, Google Cloud Speech-to-Text API). As a result, the voice data "Please create a proposal for next week" is converted into text data "Please create a proposal for next week."

[1181] Parsing and Document Generation

[1182] The device sends the converted text data to the server. The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the text data and understand the user's request. For example, the analysis result may be "Create a proposal for next week." Based on this analysis result, the server launches a generative model (e.g., OpenAI GPT-3) to generate the first version of the document.

[1183] Customization and Documentation Reference

[1184] The server references past documents from a corporate database (e.g., a MySQL database) and retrieves the appropriate templates and style guides. It then uses these templates and style guides to customize the first version of the document, for example, to match the format of a past marketing proposal.

[1185] Correction instructions and final version generation

[1186] The user gives instructions to edit the document by voice, and the device captures the voice data, converts it into text data, and sends it to the server. For example, the instruction might be, "Delete the third paragraph and add a new main point." The server analyzes this and invokes the generative model to edit the document.

[1187] Check and output

[1188] The user gives a voice command for final confirmation, and the server generates a final version of the document and sends it to the terminal. The terminal reads this final version aloud or displays it as text. The user checks it, and if there are no problems, gives a voice command for final transmission. For example, "There are no problems, so send it." The server sends the final version of the document to the specified receiving location.

[1189] Specific examples

[1190] The specific operation flow is shown below.

[1191] The user says, "I want you to create a marketing strategy report for next week." The device converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says, "It's fine, so send it," and the server sends the final version to the specified receiving location.

[1192] This system allows users to efficiently create high-quality documents through voice input and work while reducing eye strain.

[1193] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1194] Step 1: Getting voice input

[1195] The user requests the creation of a document by voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the user speaks into the device's microphone, saying, "I want you to create a proposal for next week."

[1196] Step 2: Capture audio

[1197] The terminal uses a microphone to capture the user's voice. The input is the user's voice data, and the output is the captured voice file. Specifically, the terminal collects voice data in real time.

[1198] Step 3: Convert audio data to text

[1199] The device uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice data into text data. The input is the captured voice data, and the output is text data. Specifically, the device converts voice data such as "Please write a proposal for next week" into text data such as "Please write a proposal for next week."

[1200] Step 4: Sending text data

[1201] The terminal sends the converted text data to the server. The input is text data, and the output is the text data sent to the server. Specifically, the text data is encrypted and transmitted to the server via the Internet.

[1202] Step 5: Parsing the text

[1203] The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the received text data. The input is the received text data, and the output is the analysis result. Specifically, it analyzes the text "Please create a proposal for next week" and extracts the intent "to create a proposal for next week."

[1204] Step 6: Generate the first version of the document

[1205] Based on the analysis results, the server launches a generative AI model (e.g., OpenAI GPT-3) to generate the first version of the document. The input is the analysis results, and the output is the first version of the document. Specifically, the prompt "Create a proposal for next week" is input into the generative AI model, which then generates the first version of the document.

[1206] Step 7: Referencing past documents

[1207] The server references past documents from the company's database (e.g., MySQL database). The input is a request to reference a past document, and the output is the past document data. Specifically, it uses an SQL query to retrieve templates of past marketing proposals.

[1208] Step 8: Customize the initial document

[1209] The server customizes the first version of the document using the acquired templates and style guides. The input is the first version of the document and past document data, and the output is a customized document. Specifically, the first version of the document is formatted to match the format of past marketing proposals.

[1210] Step 9: Speak correction instructions

[1211] The user gives instructions to edit the document by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Delete the third paragraph and add a new main point."

[1212] Step 10: Convert correction instructions to text

[1213] The terminal captures the voice of the correction instruction and converts it into text data using a voice recognition engine. The input is the captured voice data, and the output is text data. Specifically, the voice data is converted into text and sent to the server.

[1214] Step 11: Modifying the document

[1215] The server analyzes the received text data and uses a generative AI model to correct the document. The input is the text data of the correction instructions, and the output is the corrected document. Specifically, the document is regenerated using the AI ​​model based on the correction instructions.

[1216] Step 12: Speak your final confirmation instructions

[1217] The user gives instructions for final confirmation by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Please finalize the report and send it."

[1218] Step 13: Generate and send the final version

[1219] The server generates the final document and sends it to the terminal. The input is the revised document, and the output is the final document. The terminal reads this final version aloud or displays it as text. Specifically, the server generates a final document file and transfers it to the terminal for display or aloud reading.

[1220] Step 14: Send document instructions

[1221] If there are no problems, the user gives the final command for transmission by voice. The input is the user's voice data, and the output is the captured voice data. The specific operation is to say, "There are no problems, so send it."

[1222] Step 15: Final submission of document

[1223] The server sends the final document to the receiving location specified by the user. The input is the final sending instruction and the final document, and the output is the document sent to the receiving location. The specific operation is to upload the final version to a mail server or cloud storage.

[1224] (Application example 1)

[1225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] Employees in brick-and-mortar stores are required to provide fast, high-quality customer service and customer service on a daily basis. However, achieving this requires detailed manuals and guidelines, and creating them takes time and effort. Furthermore, employees may experience eye strain from looking at a screen for long periods of time, which can have a negative impact on their productivity and health. To solve these issues, a system is needed that uses voice input to automatically generate customer service manuals quickly and efficiently.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1228] In this invention, the server includes means for accepting voice input and converting the user's voice into text data, natural language processing means for analyzing the text data and understanding the request content, generative model means for generating documents based on the generated text data, means for referencing past documents and customizing the documents, and means for automatically generating daily work manuals and service guides. This enables store employees to quickly and accurately generate customer service manuals using only voice input, improving productivity and reducing eye strain.

[1229] "Voice input" is the process of receiving a user's voice as digital data and converting it into text data.

[1230] "Natural language processing means" is a technology that analyzes input text data, understands and processes human language, and is used to recognize the content of a user's request.

[1231] "Generative model means" refers to algorithms or models that automatically generate new documents based on analyzed text data.

[1232] "Customization" is the process of referencing previous documents and adjusting the generated document based on them to fit a specific format or style.

[1233] "Voice instruction" refers to the act of a user giving voice instructions to the system to perform a specific operation or correction.

[1234] The "means for modifying a document" refers to a function for editing an existing document and making changes or additions based on the user's voice instructions.

[1235] "Final" refers to the final document after all modifications and customizations have been completed.

[1236] "Means for outputting" refers to a function for presenting the final document to the user or transmitting it to a specified location.

[1237] "Means for automatically generating daily work manuals and service guides" refers to a process for automatically creating and providing instructions and explanations required for daily work and customer service.

[1238] This invention is a system that automatically generates customer service manuals and service guides for brick-and-mortar stores using voice input. The system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and a final version confirmation and output means.

[1239] System Configuration

[1240] Hardware

[1241] Microphone: A device for capturing audio input.

[1242] Computer or smartphone: a device that processes and stores data.

[1243] software

[1244] speech_recognition library: A library for converting speech data into text.

[1245] Transformers library and general-purpose generative models: A library for natural language processing and text generation. Specifically, it uses the Rinna / Japanese GPT-2 model.

[1246] Data processing and calculation

[1247] Voice input and conversion

[1248] The server captures the voice data that the user speaks into the microphone. This voice data is converted into text data using the speech_recognition library. For example, if the user instructs, "Create a manual to explain to customers about today's special menu," the voice is converted into text.

[1249] Parsing and Document Generation

[1250] The server receives the converted text data and analyzes it using natural language processing. Based on the analysis results, it activates a generative model to generate the first edition of the customer service manual. As a generative model, it uses the transformers library to activate the Japanese GPT-2 model.

[1251] Customization

[1252] The server references past customer service manuals and service guides from the company's database and formats the generated documents into appropriate templates and style guides, thereby ensuring consistency in the documents.

[1253] Corrections and final check

[1254] The user can make corrections to the generated document by issuing additional voice commands. For example, they can say, "Update the second paragraph with the latest trend information." This voice command is also converted into text by the speech_recognition library, and the document is corrected by the server. Finally, the user confirms and sends the document by saying, "There are no problems, so please send it."

[1255] output

[1256] The server generates the final document and presents it to the user, who then confirms it and sends it to the specified location.

[1257] Examples and prompts

[1258] For example, if an employee at a physical store says, "Please create a manual to explain to customers about today's special menu items," the speech is converted into text, and natural language processing is performed based on that text data to generate the manual. The employee then gives additional instructions, such as "Update the second paragraph with new trend information," and finally instructs, "There are no problems, so please submit," and the completed manual is generated.

[1259] (Example of a prompt)

[1260] "Create a manual to explain today's special menu to customers."

[1261] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1262] Step 1:

[1263] The user verbally requests the creation of a document into the microphone. For example, "Create a manual explaining today's special menu to customers." The input is the user's voice data. The output is the captured voice data.

[1264] Step 2:

[1265] The device converts this voice data into text data using a speech recognition engine (speech_recognition library). The input is the captured voice data, specifically voice waveform data. The device obtains text data by inputting the voice waveform data into the voice recognition engine. The output is the converted text data. For example, the voice data "Please create a manual to explain to customers about today's special menu" is converted into text data "Please create a manual to explain to customers about today's special menu."

[1266] Step 3:

[1267] The terminal sends this text data to the server. The input is the converted text data, and the output is the text data sent to the server.

[1268] Step 4:

[1269] The server analyzes the received text data using natural language processing means to understand the user's request. The input is text data, specifically the aforementioned text data "Please create a manual to explain to customers about today's special menu." The server uses natural language processing technology to analyze the request. The output is the analyzed request, for example, "Request to create a manual explaining the special menu."

[1270] Step 5:

[1271] The server invokes the generative model based on the analysis results to generate the first version of the document. The input is the parsed request content, and the output is the generated first version of the document. Specifically, the document is generated using the transformers library and a generative AI model (Japanese GPT-2).

[1272] Step 6:

[1273] The server references previously created documents from the company's database and customizes the generated document. The input is the first version of the generated document and the referenced previous document data. The server uses this information to adjust the format and style of the document. The output is a customized document.

[1274] Step 7:

[1275] The user speaks into the terminal to instruct the device to edit a document. For example, they say, "Update the second paragraph with the new trend information." The input is the voice data of the user's edit instructions. The output is the captured voice data.

[1276] Step 8:

[1277] The terminal converts this voice correction instruction into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "Update the second paragraph with the new trend information."

[1278] Step 9:

[1279] The server receives the text data and modifies the document using the generative model means. The input is the text data of the user's modification instructions, and the output is the modified document. Specifically, the server uses the generative model means to update the document based on the instructions.

[1280] Step 10:

[1281] The user gives a final confirmation instruction. For example, they say, "There's no problem, so send it." The input is the user's voice data of the final confirmation. The output is the captured voice data.

[1282] Step 11:

[1283] The terminal converts this final confirmation voice into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "There are no problems, so please send it."

[1284] Step 12:

[1285] The server generates the final document and sends it to the terminal. The input is the final confirmation instruction text data and the customized / modified document. The server generates the final document based on this. The output is the final document. The terminal reads this final version aloud or displays it in text.

[1286] Step 13:

[1287] The user performs a final check and, if there are no problems, gives a final voice command. The server sends the final document to the specified receiving location. The input is the user's voice command, and the output is the final document sent to the destination.

[1288] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1289] The present invention is a system for generating documents through voice input and adjusting the tone and content of the documents taking into account the user's emotional state. The overall flow of this system and specific embodiments are described below.

[1290] System Overview

[1291] The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit. A user gives an instruction by voice, the instruction is analyzed through natural language processing, and finally a document is generated and output based on the user's emotional state.

[1292] Program processing overview

[1293] Voice input and conversion

[1294] 1. The user speaks into the terminal to request the creation of materials.

[1295] For example: "I'd like you to prepare a proposal for next week."

[1296] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[1297] Audio data: "Please prepare a proposal for next week."

[1298] Converted text data: "Please prepare a proposal for next week."

[1299] Parsing and Document Generation

[1300] 1. The terminal sends the converted text data to the server.

[1301] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[1302] Analysis result: "Creating a proposal"

[1303] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[1304] A first draft proposal is generated.

[1305] Customization and Emotion Recognition

[1306] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[1307] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1308] Example: "Format it into the format of a past marketing proposal."

[1309] 3. The server uses the emotion engine means to recognize emotions from the user's voice. For example, emotions such as "tense" or "relaxed" are recognized from the user's voice.

[1310] 4. The server adjusts the tone and content of the generated document based on the user's emotions.

[1311] Example: If a user is "nervous," adjust the tone of the document to be more relaxing.

[1312] Correction instructions and final version generation

[1313] 1. The user gives instructions to correct the document by voice into the terminal.

[1314] For example: "Delete the third paragraph and add a new point."

[1315] 2. The device converts this voice instruction into text data and sends it to the server.

[1316] Converted text: "Delete the third paragraph and add a new point."

[1317] 3. The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1318] 4. The server modifies the document using the generative model means.

[1319] A revised proposal is generated.

[1320] Check and output

[1321] 1. The user issues a final confirmation instruction to the terminal.

[1322] Example: "Please finalize and submit the report."

[1323] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[1324] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[1325] Example: "No problem, send it."

[1326] 4. The server sends the final document to the receiving location specified by the user.

[1327] For example: a user's email address or a company's shared folder.

[1328] Specific examples

[1329] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats it into an appropriate template. At the same time, an emotion engine is used to analyze the emotions from the user's voice, and if it determines, for example, that the user is nervous, the tone of the document is softened. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[1330] This system not only allows desk workers to create high-quality documents using only their voice without using their eyes, but also provides documents with an appropriate tone and content depending on the user's emotional state, creating a less stressful work environment.

[1331] The processing flow will be explained below.

[1332] Step 1:

[1333] The user gives instructions for creating materials by voice into the terminal.

[1334] For example: "I'd like you to prepare a proposal for next week."

[1335] Step 2:

[1336] The device captures the user's voice with a microphone.

[1337] Audio data: "Please prepare a proposal for next week."

[1338] Step 3:

[1339] The device uses a speech recognition engine to convert the captured voice into text data.

[1340] Converted text data: "Please prepare a proposal for next week."

[1341] Step 4:

[1342] The terminal transmits the text data to the server.

[1343] Step 5:

[1344] The text data received by the server is analyzed using a natural language processing engine.

[1345] Analysis result: Understand the instruction "Write a proposal."

[1346] Step 6:

[1347] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[1348] A first draft proposal is generated.

[1349] Step 7:

[1350] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[1351] Step 8:

[1352] The server applies templates and style guides to the generated initial version to create customized documents.

[1353] The customized proposal draft is completed.

[1354] Step 9:

[1355] The server uses an emotion engine to recognize emotions from the user's voice.

[1356] For example, recognizing emotional states such as "tense," "relaxed," and "excited."

[1357] Step 10:

[1358] The server adjusts the tone and content of the generated document based on the emotion recognition results.

[1359] For example, if the user is "tense," adjust the tone of the document to be softer.

[1360] Step 11:

[1361] The user gives instructions to correct the document by voice into the terminal.

[1362] For example: "Delete the third paragraph and add a new point."

[1363] Step 12:

[1364] The terminal converts the user's voice into text data and sends it to the server.

[1365] Converted text: "Delete the third paragraph and add a new point."

[1366] Step 13:

[1367] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1368] Step 14:

[1369] The server uses the generative model to modify the document.

[1370] A revised proposal is generated.

[1371] Step 15:

[1372] The user issues a final confirmation instruction by voice to the terminal.

[1373] Example: "Please finalize and submit the report."

[1374] Step 16:

[1375] The server generates the final document and sends it to the terminal.

[1376] The final proposal is sent to the device.

[1377] Step 17:

[1378] Your device will read the final document aloud or display it as text.

[1379] Step 18:

[1380] After the user makes a final check, he or she issues a voice command saying, "There are no problems, so send it."

[1381] Step 19:

[1382] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[1383] For example: a user's email address or a company's shared folder.

[1384] This allows users to create high-quality documents using only their voice, without using their eyes, and provides the optimal tone and content according to their emotional state, creating a less stressful work environment.

[1385] Example 2

[1386] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1387] In conventional document creation systems, when users give instructions by voice, they not only convert the voice input into text data, but also lack the functionality to generate and edit documents taking into account the user's emotions and the tone of the document. As a result, even when users give instructions that convey their emotions, these are not easily reflected in the document, resulting in a poor user experience. In addition, the functionality to apply templates by referencing past documents is insufficient, which makes it difficult to efficiently generate documents with a consistent feel.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for accepting voice input and converting the user's voice into character data; natural language processing means for analyzing the character data and understanding the requested content; generative model means for generating a document based on the generated character data; means for customizing the document by referring to previously created documents; means for recognizing emotions from the user's voice and adjusting the tone and content of the document; means for correcting the document in response to voice instructions; and means for checking and outputting the final version. This enables high-quality document generation and correction that reflects the user's voice and emotions, thereby improving the user experience and realizing efficient document creation.

[1389] "Voice input" is a technology that captures what a user says as voice data.

[1390] "Character data" refers to data obtained by converting voice data into characters.

[1391] "Natural language processing means" is a technology that analyzes character data and understands the content of a user's request.

[1392] A "generative model means" is an algorithm that generates a document based on input data.

[1393] "Previously created documents" are documents created in the past that are used as templates or reference materials.

[1394] "Means of customization" refers to techniques for modifying and adjusting generated documents in accordance with specific style guides and templates.

[1395] "Means for recognizing emotions" refers to technology that analyzes and recognizes the emotional state of a user from their voice.

[1396] "Means for adjusting the tone and content of a document" refers to techniques for changing the expression and content of a document based on the user's emotions and needs.

[1397] "Voice instructions" are instructions or commands about a document given by the user through voice.

[1398] "Means for modifying documents" refers to technology that adds, deletes, or modifies existing documents based on voice instructions.

[1399] The "means for checking and outputting the final version" is a technique for outputting a document that has been finally checked by a user.

[1400] MODE FOR CARRYING OUT THE INVENTION

[1401] The present invention provides a system for generating documents through voice input and adjusting the tone and content of the documents in consideration of a user's emotional state. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit.

[1402] Hardware and Software Configuration

[1403] This system is configured using the following hardware and software.

[1404] 1. Device:

[1405] Audio capture device (e.g. microphone)

[1406] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[1407] 2. Server:

[1408] Natural language processing tools (e.g., spaCy, BERT)

[1409] Generative modeling tools (e.g., OpenAI GPT-4)

[1410] Database search methods (e.g., SQL databases)

[1411] Emotion engine means (e.g. IBM Watson Tone Analyzer)

[1412] Program processing flow

[1413] When a user speaks, the speech is captured by the device and converted into text using a speech recognition engine. The text data is then sent to a server, where the user's request is analyzed using natural language processing means. Based on the analysis results, a generative model means is activated to generate a first version of the document. The server then searches for past documents in the company's database and customizes the document by applying appropriate templates and style guides. Next, an emotion engine means is used to recognize emotions from the user's voice and adjust the tone and content of the document based on the emotions.

[1414] If the user makes voice corrections to the document, the voice is converted into text again and sent to the server for analysis and generation. Finally, the user gives a final check, and if there are no problems, the server outputs the final version of the document.

[1415] Explanation with concrete examples

[1416] For example, if a user says to a terminal, "I want you to write a marketing strategy report for next week," the terminal captures this speech and uses a speech recognition engine to convert it into text data saying, "I want you to write a marketing strategy report for next week." This text data is sent to a server and analyzed by natural language processing means. Based on the analysis results, a generative model means creates a first edition of a marketing strategy report. The server then references past marketing reports and applies an appropriate template. At the same time, the emotion engine means recognizes emotions from the user's voice, and if it determines, for example, that the user is nervous, it adjusts the tone of the document to be softer.

[1417] When the user instructs, "Update the second paragraph with the new trend information," the device again captures the voice, converts it into text data, and sends it to the server. The server makes the corrections and generates a document containing the new trend information. Finally, when the user instructs, "There are no problems, so send it," the server sends the final document to the specified recipient (e.g., email address or shared folder).

[1418] Prompt Sentence Examples

[1419] User: "I want a marketing strategy report for next week."

[1420] Device: "Converting speech to text"

[1421] Server: "Generating first version"

[1422] Server: "Applying template by referencing past reports"

[1423] Server: "Analyzing user sentiment"

[1424] Server: "Adjusting the tone of the document"

[1425] User: "Update the second paragraph with new trending information."

[1426] Device: "Converting speech to text"

[1427] Server: "Generating updated document"

[1428] User: "No problem, send it."

[1429] Server: "Sending final version to specified location"

[1430] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1431] Step 1: Capture and convert audio input

[1432] 1. The user gives instructions through a voice input device. Example: "Please prepare a proposal for next week."

[1433] 2. The device captures the user's voice using an audio capture device (e.g., a microphone).

[1434] Input: User's voice data

[1435] 3. The device converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1436] Data processing: Converting voice data into text data

[1437] Output: Text data "Please create a proposal for next week."

[1438] Step 2: Sending text data

[1439] 1. The terminal sends the converted character data to the server.

[1440] Input: Text data "Please create a proposal for next week."

[1441] Output: Character data sent to the server

[1442] Step 3: Analyzing the text data

[1443] 1. The server analyzes the received text data using natural language processing tools (e.g., spaCy, BERT) to understand the user's request.

[1444] Input: Text data "Please create a proposal for next week."

[1445] Data Computing: Analyzing Request Contents Using Natural Language Processing

[1446] Output: Analysis result "Creating a proposal"

[1447] Step 4: Generate the initial document

[1448] 1. The server launches a generative model (e.g., OpenAI GPT-4) based on the analysis results to generate the first version of the document.

[1449] Input: Analysis result "Create a proposal"

[1450] Data Computation: First Edition Document Generation with Document Generation Algorithms

[1451] Output: First edition proposal

[1452] Step 5: Apply and customize the template

[1453] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[1454] Input: Company database

[1455] Data processing: Searching and retrieving past documents

[1456] Output: Templates and style guides

[1457] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1458] Input: First edition proposals, templates and style guides

[1459] Data calculations: Customization based on template application and style guides

[1460] Output: Customized document

[1461] Step 6: Emotional awareness and tone adjustment

[1462] 1. The server recognizes emotions from the user's voice using an emotion engine means (e.g., IBM Watson Tone Analyzer).

[1463] Input: User's voice data

[1464] Data Computing: Sentiment Analysis

[1465] Output: Emotional state (e.g., "I'm nervous")

[1466] 2. The server adjusts the tone and content of the generated document based on the user's emotions.

[1467] Input: Emotional state "Tense", customized document

[1468] Data operations: tone and content adjustment

[1469] Output: Adjusted document

[1470] Step 7: Processing correction instructions

[1471] 1. The user speaks into the device to instruct the device to revise the document. For example, "Delete the third paragraph and add a new main point."

[1472] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[1473] Input: Spoken correction instructions

[1474] Data processing: Converting voice data into text data

[1475] Output: Text data "Delete the third paragraph and add a new point."

[1476] 3. The terminal sends the converted character data to the server.

[1477] Input: Text data "Delete the third paragraph and add a new point."

[1478] Output: Character data sent to the server

[1479] 4. The server again performs natural language processing to analyze the correction instructions.

[1480] Input: Text data "Delete the third paragraph and add a new point."

[1481] Data calculation: Analysis of correction instructions

[1482] Output: Analysis result "Delete the third paragraph and add a new main point"

[1483] 5. The server modifies the document using the generative model.

[1484] Input: Analysis result "Delete third paragraph and add new main point", adjusted document

[1485] Data Computation: Regeneration by Document Correction Algorithms

[1486] Output: The modified document

[1487] Step 8: Final check and output

[1488] 1. The user issues a final confirmation instruction to the terminal. Example: "Please finalize and submit the report."

[1489] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[1490] Input: Final confirmation voice

[1491] Data processing: Converting voice data into text data

[1492] Output: Text data "Please finalize the report and send it."

[1493] 3. The terminal sends the converted character data to the server.

[1494] Input: Text data "Please finalize the report and send it."

[1495] Output: Character data sent to the server

[1496] 4. The server generates the final document and sends it to the terminal.

[1497] Input: Adjusted and amended document

[1498] Output: Final document

[1499] 5. The device will read this final version aloud or display it as text.

[1500] Input: Final document

[1501] Output: Presentation to the user (audio or text display)

[1502] 6. The user performs a final check and, if there are no problems, gives a final voice command. For example, "There are no problems, so please send."

[1503] 7. The server sends the final document to the receiving location specified by the user.

[1504] Input: Final confirmation and sending instructions

[1505] Output: Send the final document to the specified receiving location

[1506] (Application example 2)

[1507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1508] Modern virtual stores are required to efficiently provide diverse product information and quickly and appropriately address customer questions and concerns. However, conventional systems have difficulty responding flexibly to customer emotions, and have lacked effective means to increase customer satisfaction. In particular, they have been unable to adjust the tone and content of documents generated based on voice input to match the user's emotional state, making it difficult to provide personalized service to customers.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1510] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for customizing the document by referring to documents created in the past by the company, means for correcting the document in response to voice instructions, emotion engine means for analyzing the user's emotions from the voice, means for adjusting the tone of the document according to the user's emotional state, and means for checking and outputting the final version. This makes it possible to automate customer service and the generation of product descriptions in a virtual store and provide personalized services according to the customer's emotions.

[1511] "Voice input means" refers to technology for receiving and processing a user's voice.

[1512] "Text data conversion means" refers to a technology that converts the voice captured by the voice input means into text format.

[1513] "Natural language processing means" refers to technology for analyzing text data and understanding the content of user requests.

[1514] "Generative modeling means" refers to technology that automatically generates documents based on analyzed text data.

[1515] "Document customization means" refers to technology that references a company's previously created documents and adjusts the generated documents in accordance with templates and style guides.

[1516] "Voice instruction means" refers to a technology that receives instructions given by voice from a user and modifies a document based on the contents of the instructions.

[1517] The "emotion engine means" refers to a technology that analyzes emotions from the user's voice and identifies the user's emotional state.

[1518] "Tone adjustment" refers to a technique for adjusting the tone and content of a generated document depending on the user's emotional state.

[1519] "Output means" refers to a technique for providing the final generated document to the user in audio or text format.

[1520] The present invention provides a system for generating documents in response to user requests through voice input and adjusting the tone and content of the documents based on the user's emotional state. The system includes a voice input unit, a character data conversion unit, a natural language processing unit, a generative model unit, a document customization unit, a voice instruction unit, an emotion engine unit, a tone adjustment unit, and an output unit.

[1521] System configuration

[1522] The system consists of the following major hardware and software components:

[1523] Hardware: microphone, computer, display device

[1524] software:

[1525] speech_recognition library for implementing speech recognition technology

[1526] Transformers library for natural language processing of text data

[1527] j-hartmann / emotion-english-distilroberta-base model for sentiment analysis

[1528] The rinna / japanese-gpt2-medium model for text generation

[1529] System Operation Overview

[1530] 1. Voice input and conversion

[1531] The user uses a microphone to provide voice input. The device converts the voice data into text using the speech_recognition library. For example, if the user says, "What are the features of this product?", the device converts the voice data into text: "What are the features of this product?"

[1532] 2. Text analysis and first edition generation

[1533] The server receives the converted text data, performs natural language processing using the transformers library, analyzes the user's request, and then generates the first version of the document using the rinna / japanese-gpt2-medium model based on the analysis results.

[1534] 3. Customization and Sentiment Analysis

[1535] The server references the company's past documents and customizes the generated document according to templates and style guides. At the same time, it analyzes the user's voice using the j-hartmann / emotion-english-distilroberta-base model to detect emotions. If the user's voice is recognized as "tense," it adjusts the tone of the document to be more relaxed.

[1536] 4. Correction instructions and final version generation

[1537] The user gives instructions for corrections by voice through a microphone. The device again converts the voice into text, and the server updates the document based on the instructions. For example, if the user instructs the server to "delete the third paragraph and add a new point," the document is updated accordingly.

[1538] 5. Check and final output

[1539] The user then gives a final confirmation, and the server generates the final document and sends it to the device. The device then reads the document aloud or displays it in text. If the user determines there are no problems, the document is sent to their email address, a shared folder, or other location.

[1540] Specific examples

[1541] The user speaks into the device, saying, "Please tell me about the features of this product." The device converts this speech into text data, and the server analyzes the received text data. A product description is generated using the generative AI model rinna / japanese-gpt2-medium, which is then customized based on the company's past documents. At the same time, if the emotion engine detects "tension" in the user's voice, the tone of the text is adjusted to be more relaxed. For example, it could be adjusted to say, "I will explain the features of this product. This product is high quality and can be used safely. Please relax and enjoy it."

[1542] As a concrete example of a prompt, if a user says, "Please tell me the features of this product," the prompt to the generative AI model would be:

[1543] "Please describe the features of this product as follows: This product is high quality, reliable, and very beneficial to the user."

[1544] In this way, the system of the present invention enables personalized document generation based on voice input and in response to the user's emotional state.

[1545] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1546] Step 1:

[1547] The user provides voice input.

[1548] The user speaks a question or request into the microphone, e.g., "What are the features of this product?". Voice data is obtained as input.

[1549] Step 2:

[1550] The terminal converts the voice data into text data.

[1551] The device receives voice data input from the user and converts it into text data using the speech_recognition library. The input is voice data and the output is text data. The converted text data is "Please tell me the features of this product."

[1552] Step 3:

[1553] The server analyzes the text data and understands the request.

[1554] The server receives the converted text data and performs natural language processing using the transformers library, which then analyzes the user's request. The input is text data, and the output is the analyzed request (e.g., "I want to know the product's features").

[1555] Step 4:

[1556] The server generates a document based on the analysis results.

[1557] The server generates the first version of the document using the rinna / japanese-gpt2-medium model based on the parsed request content. The input is the parsed request content, and the output is the generated text. An example of the generated document is: "This product is high quality, reliable, and very useful to users."

[1558] Step 5:

[1559] The server references the company's past documents and customizes the document.

[1560] The server searches the company's database for previously created documents and retrieves the templates and style guides to be applied. The input is the generated text, and the output is the customized document. An example of a customized document is: "This product is high quality, reliable, and very beneficial to users. Please use it with confidence."

[1561] Step 6:

[1562] The server analyzes the user's emotions from the voice.

[1563] The server uses the user's voice data to perform emotion analysis using the j-hartmann / emotion-english-distilroberta-base model. The input is the user's voice data, and the output is the user's emotional state (e.g., "I'm nervous").

[1564] Step 7:

[1565] The server adjusts the tone of the document based on the user's emotional state.

[1566] The server adjusts the tone of the generated document based on the analyzed emotional state. The input is a customized document and the emotional state, and the output is a tone-adjusted document. An example of an adjusted document is: "This product is high quality and reliable, so you can use it with confidence. Please relax and enjoy it."

[1567] Step 8:

[1568] The user gives correction instructions by voice.

[1569] The user issues correction instructions through a microphone, for example, "Delete the third paragraph and add a new main point." The input is voice data, and the output is the voice data of the correction instructions.

[1570] Step 9:

[1571] The device converts the speech back into text and sends it to the server.

[1572] The device converts the voice instructions for correction into text data and sends it to the server. The input is voice data and the output is text data. Converted text data: "Delete the third paragraph and add a new main point."

[1573] Step 10:

[1574] The server updates the document based on the correction instructions.

[1575] The server receives the text data of the correction instructions and updates the document using natural language processing. The input is the text data of the correction instructions and the output is the updated document.

[1576] Step 11:

[1577] The user gives a final confirmation instruction.

[1578] The user uses the microphone to give a final confirmation instruction, e.g., "There are no problems, so send it." The input is voice data, and the output is the voice data of the final confirmation.

[1579] Step 12:

[1580] The server generates and outputs the final document.

[1581] The server generates a final, verified document and sends it to the terminal, which then reads the final document aloud or displays it as text. The input is the updated document, and the output is the final document.

[1582] In this way, a document generation system based on a user's voice input can analyze the user's request, generate an appropriate document according to the user's emotional state, and ultimately provide a personalized document.

[1583] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1585] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1586] [Fourth embodiment]

[1587] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1588] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1589] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1590] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1591] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1593] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1594] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1595] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1596] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1597] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1598] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1599] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1600] The present invention provides a system for generating documents through voice input, thereby reducing eye strain for desk workers. The overall flow of this system and specific embodiments are described below.

[1601] System Overview

[1602] This system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and an output means. A user gives instructions by voice, and the instructions are analyzed through natural language processing, and finally a document is generated and output.

[1603] Program processing overview

[1604] Voice input and conversion

[1605] 1. The user speaks into the terminal to request the creation of materials.

[1606] For example: "I'd like you to prepare a proposal for next week."

[1607] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[1608] Audio data: "Please prepare a proposal for next week."

[1609] Converted text data: "Please prepare a proposal for next week."

[1610] Parsing and Document Generation

[1611] 1. The terminal sends the converted text data to the server.

[1612] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[1613] Analysis result: "Create a proposal for next week"

[1614] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[1615] Customization and Documentation Reference

[1616] 1. The server references previously created documents from the company database and retrieves appropriate templates and style guides.

[1617] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1618] Example: "Format it into the format of a past marketing proposal."

[1619] Correction instructions and final version generation

[1620] 1. The user gives instructions to correct the document by voice into the terminal.

[1621] For example: "Delete the third paragraph and add a new point."

[1622] 2. The device converts this voice instruction into text data and sends it to the server.

[1623] Audio: "Delete the third paragraph and add a new point."

[1624] Converted text: "Delete the third paragraph and add a new point."

[1625] 3. The server receives the text data and modifies it using the generative modeling means.

[1626] Check and output

[1627] 1. The user issues a final confirmation instruction to the terminal.

[1628] Example: "Please finalize and submit the report."

[1629] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[1630] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[1631] Example: "No problem, send it."

[1632] 4. The server sends the final document to the receiving location specified by the user.

[1633] Specific examples

[1634] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[1635] This system allows desk workers to create high-quality documents using only their voice, without looking at a screen, reducing eye strain and supporting a healthier working style.

[1636] The processing flow will be explained below.

[1637] Step 1:

[1638] The user gives instructions for creating materials by voice into the terminal.

[1639] For example: "I'd like you to prepare a proposal for next week."

[1640] Step 2:

[1641] The device captures the user's voice with a microphone.

[1642] Audio data: "Please prepare a proposal for next week."

[1643] Step 3:

[1644] The device uses a speech recognition engine to convert the captured voice into text data.

[1645] Converted text data: "Please prepare a proposal for next week."

[1646] Step 4:

[1647] The terminal transmits the text data to the server.

[1648] Step 5:

[1649] The text data received by the server is analyzed using a natural language processing engine.

[1650] Analysis result: Understand the instruction "Write a proposal."

[1651] Step 6:

[1652] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[1653] A first draft proposal is generated.

[1654] Step 7:

[1655] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[1656] Step 8:

[1657] The server applies templates and style guides to the generated initial version to create customized documents.

[1658] The customized proposal draft is completed.

[1659] Step 9:

[1660] The user gives instructions to correct the document by voice into the terminal.

[1661] For example: "Delete the third paragraph and add a new point."

[1662] Step 10:

[1663] The terminal converts the user's voice into text data and sends it to the server.

[1664] Converted text: "Delete the third paragraph and add a new point."

[1665] Step 11:

[1666] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1667] Step 12:

[1668] The server uses the generative model to modify the document.

[1669] A revised proposal is generated.

[1670] Step 13:

[1671] The user issues a final confirmation instruction by voice to the terminal.

[1672] Example: "Please finalize and submit the report."

[1673] Step 14:

[1674] The server generates the final document and sends it to the terminal.

[1675] The final proposal is sent to the device.

[1676] Step 15:

[1677] Your device will read the final document aloud or display it as text.

[1678] Step 16:

[1679] After the user makes a final confirmation, he or she issues a voice command saying, "There are no problems, so send it."

[1680] Step 17:

[1681] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[1682] For example: a user's email address or a company's shared folder.

[1683] This allows users to create high-quality documents using only their voice, without using their eyes, enabling a healthier way of working.

[1684] Example 1

[1685] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1686] Today's desk workers often suffer from eye strain and stress due to long hours of screen time. Furthermore, document creation takes a lot of time and effort, making it difficult to carry out work efficiently. For this reason, there is a demand for technology that utilizes voice input to efficiently generate documents and reduce eye strain.

[1687] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1688] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for referencing previously created documents and customizing the document, means for correcting the document in response to voice instructions, means for confirming and outputting the final version, means for reading the final version aloud or displaying it in text to prompt the user for confirmation, and means for managing the operation of the entire system based on a program flow divided into processing steps. This allows users to efficiently create, correct, confirm, and output documents using only voice input, reducing eye strain and improving work efficiency.

[1689] "Voice input" is an input method in which the system receives voice data spoken by the user.

[1690] "Character data" refers to text data converted from voice input using voice recognition technology.

[1691] "Natural language processing" is a technology that analyzes text data and understands its content and meaning.

[1692] A "generative model" is a technology that generates new documents or content based on input data.

[1693] "Customization" refers to the process of tailoring a generated document to a particular template or historical document style.

[1694] "Voice instructions" refer to instructions for corrections or operations given by the user using voice.

[1695] The "final version" refers to a document that is completed at the final stage of document creation or revision.

[1696] "Output" refers to the process of displaying, printing, or transmitting the generated document.

[1697] "Read aloud" refers to the process by which the system converts text data into audio and plays it aloud for the user.

[1698] "Program flow" refers to the flow of a program that shows the processing procedures and steps of the entire system.

[1699] "Managing the operation of the entire system" refers to the management activities of controlling each function of the system in an integrated manner and operating it efficiently.

[1700] The present invention provides a system for generating documents through voice input to reduce eye strain for desk workers. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, and an output unit.

[1701] Voice input and conversion

[1702] The user requests the creation of a document by voice, and the device captures this voice using a microphone. The captured voice data is converted into text data by a voice recognition engine (for example, Google Cloud Speech-to-Text API). As a result, the voice data "Please create a proposal for next week" is converted into text data "Please create a proposal for next week."

[1703] Parsing and Document Generation

[1704] The device sends the converted text data to the server. The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the text data and understand the user's request. For example, the analysis result may be "Create a proposal for next week." Based on this analysis result, the server launches a generative model (e.g., OpenAI GPT-3) to generate the first version of the document.

[1705] Customization and Documentation Reference

[1706] The server references past documents from a corporate database (e.g., a MySQL database) and retrieves the appropriate templates and style guides. It then uses these templates and style guides to customize the first version of the document, for example, to match the format of a past marketing proposal.

[1707] Correction instructions and final version generation

[1708] The user gives instructions to edit the document by voice, and the device captures the voice data, converts it into text data, and sends it to the server. For example, the instruction might be, "Delete the third paragraph and add a new main point." The server analyzes this and invokes the generative model to edit the document.

[1709] Check and output

[1710] The user gives a voice command for final confirmation, and the server generates a final version of the document and sends it to the terminal. The terminal reads this final version aloud or displays it as text. The user checks it, and if there are no problems, gives a voice command for final transmission. For example, "There are no problems, so send it." The server sends the final version of the document to the specified receiving location.

[1711] Specific examples

[1712] The specific operation flow is shown below.

[1713] The user says, "I want you to create a marketing strategy report for next week." The device converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats them into an appropriate template. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says, "It's fine, so send it," and the server sends the final version to the specified receiving location.

[1714] This system allows users to efficiently create high-quality documents through voice input and work while reducing eye strain.

[1715] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1716] Step 1: Getting voice input

[1717] The user requests the creation of a document by voice. The input is the user's voice data, and the output is the captured voice data. Specifically, the user speaks into the device's microphone, saying, "I want you to create a proposal for next week."

[1718] Step 2: Capture audio

[1719] The terminal uses a microphone to capture the user's voice. The input is the user's voice data, and the output is the captured voice file. Specifically, the terminal collects voice data in real time.

[1720] Step 3: Convert audio data to text

[1721] The device uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice data into text data. The input is the captured voice data, and the output is text data. Specifically, the device converts voice data such as "Please write a proposal for next week" into text data such as "Please write a proposal for next week."

[1722] Step 4: Sending text data

[1723] The terminal sends the converted text data to the server. The input is text data, and the output is the text data sent to the server. Specifically, the text data is encrypted and transmitted to the server via the Internet.

[1724] Step 5: Parsing the text

[1725] The server uses a natural language processing engine (e.g., Google Natural Language API) to analyze the received text data. The input is the received text data, and the output is the analysis result. Specifically, it analyzes the text "Please create a proposal for next week" and extracts the intent "to create a proposal for next week."

[1726] Step 6: Generate the first version of the document

[1727] Based on the analysis results, the server launches a generative AI model (e.g., OpenAI GPT-3) to generate the first version of the document. The input is the analysis results, and the output is the first version of the document. Specifically, the prompt "Create a proposal for next week" is input into the generative AI model, which then generates the first version of the document.

[1728] Step 7: Referencing past documents

[1729] The server references past documents from the company's database (e.g., MySQL database). The input is a request to reference a past document, and the output is the past document data. Specifically, it uses an SQL query to retrieve templates of past marketing proposals.

[1730] Step 8: Customize the initial document

[1731] The server customizes the first version of the document using the acquired templates and style guides. The input is the first version of the document and past document data, and the output is a customized document. Specifically, the first version of the document is formatted to match the format of past marketing proposals.

[1732] Step 9: Speak correction instructions

[1733] The user gives instructions to edit the document by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Delete the third paragraph and add a new main point."

[1734] Step 10: Convert correction instructions to text

[1735] The terminal captures the voice of the correction instruction and converts it into text data using a voice recognition engine. The input is the captured voice data, and the output is text data. Specifically, the voice data is converted into text and sent to the server.

[1736] Step 11: Modifying the document

[1737] The server analyzes the received text data and uses a generative AI model to correct the document. The input is the text data of the correction instructions, and the output is the corrected document. Specifically, the document is regenerated using the AI ​​model based on the correction instructions.

[1738] Step 12: Speak your final confirmation instructions

[1739] The user gives instructions for final confirmation by voice. The input is the user's voice data, and the output is the captured voice data. The specific action is to say, "Please finalize the report and send it."

[1740] Step 13: Generate and send the final version

[1741] The server generates the final document and sends it to the terminal. The input is the revised document, and the output is the final document. The terminal reads this final version aloud or displays it as text. Specifically, the server generates a final document file and transfers it to the terminal for display or aloud reading.

[1742] Step 14: Send document instructions

[1743] If there are no problems, the user gives the final command for transmission by voice. The input is the user's voice data, and the output is the captured voice data. The specific operation is to say, "There are no problems, so send it."

[1744] Step 15: Final submission of document

[1745] The server sends the final document to the receiving location specified by the user. The input is the final sending instruction and the final document, and the output is the document sent to the receiving location. The specific operation is to upload the final version to a mail server or cloud storage.

[1746] (Application example 1)

[1747] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1748] Employees in brick-and-mortar stores are required to provide fast, high-quality customer service and customer service on a daily basis. However, achieving this requires detailed manuals and guidelines, and creating them takes time and effort. Furthermore, employees may experience eye strain from looking at a screen for long periods of time, which can have a negative impact on their productivity and health. To solve these issues, a system is needed that uses voice input to automatically generate customer service manuals quickly and efficiently.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1750] In this invention, the server includes means for accepting voice input and converting the user's voice into text data, natural language processing means for analyzing the text data and understanding the request content, generative model means for generating documents based on the generated text data, means for referencing past documents and customizing the documents, and means for automatically generating daily work manuals and service guides. This enables store employees to quickly and accurately generate customer service manuals using only voice input, improving productivity and reducing eye strain.

[1751] "Voice input" is the process of receiving a user's voice as digital data and converting it into text data.

[1752] "Natural language processing means" is a technology that analyzes input text data, understands and processes human language, and is used to recognize the content of a user's request.

[1753] "Generative model means" refers to algorithms or models that automatically generate new documents based on analyzed text data.

[1754] "Customization" is the process of referencing previous documents and adjusting the generated document based on them to fit a specific format or style.

[1755] "Voice instruction" refers to the act of a user giving voice instructions to the system to perform a specific operation or correction.

[1756] The "means for modifying a document" refers to a function for editing an existing document and making changes or additions based on the user's voice instructions.

[1757] "Final" refers to the final document after all modifications and customizations have been completed.

[1758] "Means for outputting" refers to a function for presenting the final document to the user or transmitting it to a specified location.

[1759] "Means for automatically generating daily work manuals and service guides" refers to a process for automatically creating and providing instructions and explanations required for daily work and customer service.

[1760] This invention is a system that automatically generates customer service manuals and service guides for brick-and-mortar stores using voice input. The system mainly includes a voice input means, a natural language processing means, a generative model means, a customization means, a voice instruction means, and a final version confirmation and output means.

[1761] System Configuration

[1762] Hardware

[1763] Microphone: A device for capturing audio input.

[1764] Computer or smartphone: a device that processes and stores data.

[1765] software

[1766] speech_recognition library: A library for converting speech data into text.

[1767] Transformers library and general-purpose generative models: A library for natural language processing and text generation. Specifically, it uses the Rinna / Japanese GPT-2 model.

[1768] Data processing and calculation

[1769] Voice input and conversion

[1770] The server captures the voice data that the user speaks into the microphone. This voice data is converted into text data using the speech_recognition library. For example, if the user instructs, "Create a manual to explain to customers about today's special menu," the voice is converted into text.

[1771] Parsing and Document Generation

[1772] The server receives the converted text data and analyzes it using natural language processing. Based on the analysis results, it activates a generative model to generate the first edition of the customer service manual. As a generative model, it uses the transformers library to activate the Japanese GPT-2 model.

[1773] Customization

[1774] The server references past customer service manuals and service guides from the company's database and formats the generated documents into appropriate templates and style guides, thereby ensuring consistency in the documents.

[1775] Corrections and final check

[1776] The user can make corrections to the generated document by issuing additional voice commands. For example, they can say, "Update the second paragraph with the latest trend information." This voice command is also converted into text by the speech_recognition library, and the document is corrected by the server. Finally, the user confirms and sends the document by saying, "There are no problems, so please send it."

[1777] output

[1778] The server generates the final document and presents it to the user, who then confirms it and sends it to the specified location.

[1779] Examples and prompts

[1780] For example, if an employee at a physical store says, "Please create a manual to explain to customers about today's special menu items," the speech is converted into text, and natural language processing is performed based on that text data to generate the manual. The employee then gives additional instructions, such as "Update the second paragraph with new trend information," and finally instructs, "There are no problems, so please submit," and the completed manual is generated.

[1781] (Example of a prompt)

[1782] "Create a manual to explain today's special menu to customers."

[1783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1784] Step 1:

[1785] The user verbally requests the creation of a document into the microphone. For example, "Create a manual explaining today's special menu to customers." The input is the user's voice data. The output is the captured voice data.

[1786] Step 2:

[1787] The device converts this voice data into text data using a speech recognition engine (speech_recognition library). The input is the captured voice data, specifically voice waveform data. The device obtains text data by inputting the voice waveform data into the voice recognition engine. The output is the converted text data. For example, the voice data "Please create a manual to explain to customers about today's special menu" is converted into text data "Please create a manual to explain to customers about today's special menu."

[1788] Step 3:

[1789] The terminal sends this text data to the server. The input is the converted text data, and the output is the text data sent to the server.

[1790] Step 4:

[1791] The server analyzes the received text data using natural language processing means to understand the user's request. The input is text data, specifically the aforementioned text data "Please create a manual to explain to customers about today's special menu." The server uses natural language processing technology to analyze the request. The output is the analyzed request, for example, "Request to create a manual explaining the special menu."

[1792] Step 5:

[1793] The server invokes the generative model based on the analysis results to generate the first version of the document. The input is the parsed request content, and the output is the generated first version of the document. Specifically, the document is generated using the transformers library and a generative AI model (Japanese GPT-2).

[1794] Step 6:

[1795] The server references previously created documents from the company's database and customizes the generated document. The input is the first version of the generated document and the referenced previous document data. The server uses this information to adjust the format and style of the document. The output is a customized document.

[1796] Step 7:

[1797] The user speaks into the terminal to instruct the device to edit a document. For example, they say, "Update the second paragraph with the new trend information." The input is the voice data of the user's edit instructions. The output is the captured voice data.

[1798] Step 8:

[1799] The terminal converts this voice correction instruction into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "Update the second paragraph with the new trend information."

[1800] Step 9:

[1801] The server receives the text data and modifies the document using the generative model means. The input is the text data of the user's modification instructions, and the output is the modified document. Specifically, the server uses the generative model means to update the document based on the instructions.

[1802] Step 10:

[1803] The user gives a final confirmation instruction. For example, they say, "There's no problem, so send it." The input is the user's voice data of the final confirmation. The output is the captured voice data.

[1804] Step 11:

[1805] The terminal converts this final confirmation voice into text data and sends it to the server. The input is the captured voice data, which is voice waveform data. The terminal converts this into text data using a voice recognition engine. The output is the converted text data, for example, "There are no problems, so please send it."

[1806] Step 12:

[1807] The server generates the final document and sends it to the terminal. The input is the final confirmation instruction text data and the customized / modified document. The server generates the final document based on this. The output is the final document. The terminal reads this final version aloud or displays it in text.

[1808] Step 13:

[1809] The user performs a final check and, if there are no problems, gives a final voice command. The server sends the final document to the specified receiving location. The input is the user's voice command, and the output is the final document sent to the destination.

[1810] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1811] The present invention is a system for generating documents through voice input and adjusting the tone and content of the documents taking into account the user's emotional state. The overall flow of this system and specific embodiments are described below.

[1812] System Overview

[1813] The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit. A user gives an instruction by voice, the instruction is analyzed through natural language processing, and finally a document is generated and output based on the user's emotional state.

[1814] Program processing overview

[1815] Voice input and conversion

[1816] 1. The user speaks into the terminal to request the creation of materials.

[1817] For example: "I'd like you to prepare a proposal for next week."

[1818] 2. The device captures this audio and converts it into text data using a speech recognition engine.

[1819] Audio data: "Please prepare a proposal for next week."

[1820] Converted text data: "Please prepare a proposal for next week."

[1821] Parsing and Document Generation

[1822] 1. The terminal sends the converted text data to the server.

[1823] 2. The server analyzes the received text data using natural language processing means to understand the user's request.

[1824] Analysis result: "Creating a proposal"

[1825] 3. The server activates the generative model means based on the analysis results to generate the first version of the document.

[1826] A first draft proposal is generated.

[1827] Customization and Emotion Recognition

[1828] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[1829] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1830] Example: "Format it into the format of a past marketing proposal."

[1831] 3. The server uses the emotion engine means to recognize emotions from the user's voice. For example, emotions such as "tense" or "relaxed" are recognized from the user's voice.

[1832] 4. The server adjusts the tone and content of the generated document based on the user's emotions.

[1833] Example: If a user is "nervous," adjust the tone of the document to be more relaxing.

[1834] Correction instructions and final version generation

[1835] 1. The user gives instructions to correct the document by voice into the terminal.

[1836] For example: "Delete the third paragraph and add a new point."

[1837] 2. The device converts this voice instruction into text data and sends it to the server.

[1838] Converted text: "Delete the third paragraph and add a new point."

[1839] 3. The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1840] 4. The server modifies the document using the generative model means.

[1841] A revised proposal is generated.

[1842] Check and output

[1843] 1. The user issues a final confirmation instruction to the terminal.

[1844] Example: "Please finalize and submit the report."

[1845] 2. The server generates the final document and sends it to the device, which reads it aloud or displays it as text.

[1846] 3. The user checks the information and, if there are no problems, gives final instructions via voice.

[1847] Example: "No problem, send it."

[1848] 4. The server sends the final document to the receiving location specified by the user.

[1849] For example: a user's email address or a company's shared folder.

[1850] Specific examples

[1851] The user speaks into the terminal, "I want you to create a marketing strategy report for next week." The terminal converts this voice into text data and sends it to the server. The server analyzes the received text data and generates the first version of the marketing strategy report. It then refers to the company's past marketing reports and formats it into an appropriate template. At the same time, an emotion engine is used to analyze the emotions from the user's voice, and if it determines, for example, that the user is nervous, the tone of the document is softened. The user gives a voice command such as "Update the second paragraph with new trend information," and the server updates it based on this command. Finally, the user says "There are no problems, so send it," and the server sends the final version to the specified destination.

[1852] This system not only allows desk workers to create high-quality documents using only their voice without using their eyes, but also provides documents with an appropriate tone and content depending on the user's emotional state, creating a less stressful work environment.

[1853] The processing flow will be explained below.

[1854] Step 1:

[1855] The user gives instructions for creating materials by voice into the terminal.

[1856] For example: "I'd like you to prepare a proposal for next week."

[1857] Step 2:

[1858] The device captures the user's voice with a microphone.

[1859] Audio data: "Please prepare a proposal for next week."

[1860] Step 3:

[1861] The device uses a speech recognition engine to convert the captured voice into text data.

[1862] Converted text data: "Please prepare a proposal for next week."

[1863] Step 4:

[1864] The terminal transmits the text data to the server.

[1865] Step 5:

[1866] The text data received by the server is analyzed using a natural language processing engine.

[1867] Analysis result: Understand the instruction "Write a proposal."

[1868] Step 6:

[1869] The server invokes a generative model (e.g., GPT model) based on the analysis results to generate the first version of the document.

[1870] A first draft proposal is generated.

[1871] Step 7:

[1872] The server searches the company's database for previously created documents and retrieves the templates and style guides that should be applied.

[1873] Step 8:

[1874] The server applies templates and style guides to the generated initial version to create customized documents.

[1875] The customized proposal draft is completed.

[1876] Step 9:

[1877] The server uses an emotion engine to recognize emotions from the user's voice.

[1878] For example, recognizing emotional states such as "tense," "relaxed," and "excited."

[1879] Step 10:

[1880] The server adjusts the tone and content of the generated document based on the emotion recognition results.

[1881] For example, if the user is "tense," adjust the tone of the document to be softer.

[1882] Step 11:

[1883] The user gives instructions to correct the document by voice into the terminal.

[1884] For example: "Delete the third paragraph and add a new point."

[1885] Step 12:

[1886] The terminal converts the user's voice into text data and sends it to the server.

[1887] Converted text: "Delete the third paragraph and add a new point."

[1888] Step 13:

[1889] The server receives the text data and performs natural language processing again to analyze the correction instructions.

[1890] Step 14:

[1891] The server uses the generative model to modify the document.

[1892] A revised proposal is generated.

[1893] Step 15:

[1894] The user issues a final confirmation instruction by voice to the terminal.

[1895] Example: "Please finalize and submit the report."

[1896] Step 16:

[1897] The server generates the final document and sends it to the terminal.

[1898] The final proposal is sent to the device.

[1899] Step 17:

[1900] Your device will read the final document aloud or display it as text.

[1901] Step 18:

[1902] After the user makes a final check, he or she issues a voice command saying, "There are no problems, so send it."

[1903] Step 19:

[1904] The server follows the user's instructions and sends the final document to the receiving location specified by the user.

[1905] For example: a user's email address or a company's shared folder.

[1906] This allows users to create high-quality documents using only their voice, without using their eyes, and provides the optimal tone and content according to their emotional state, creating a less stressful work environment.

[1907] Example 2

[1908] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1909] In conventional document creation systems, when users give instructions by voice, they not only convert the voice input into text data, but also lack the functionality to generate and edit documents taking into account the user's emotions and the tone of the document. As a result, even when users give instructions that convey their emotions, these are not easily reflected in the document, resulting in a poor user experience. In addition, the functionality to apply templates by referencing past documents is insufficient, which makes it difficult to efficiently generate documents with a consistent feel.

[1910] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for accepting voice input and converting the user's voice into character data; natural language processing means for analyzing the character data and understanding the requested content; generative model means for generating a document based on the generated character data; means for customizing the document by referring to previously created documents; means for recognizing emotions from the user's voice and adjusting the tone and content of the document; means for correcting the document in response to voice instructions; and means for checking and outputting the final version. This enables high-quality document generation and correction that reflects the user's voice and emotions, thereby improving the user experience and realizing efficient document creation.

[1911] "Voice input" is a technology that captures what a user says as voice data.

[1912] "Character data" refers to data obtained by converting voice data into characters.

[1913] "Natural language processing means" is a technology that analyzes character data and understands the content of a user's request.

[1914] A "generative model means" is an algorithm that generates a document based on input data.

[1915] "Previously created documents" are documents created in the past that are used as templates or reference materials.

[1916] "Means of customization" refers to techniques for modifying and adjusting generated documents in accordance with specific style guides and templates.

[1917] "Means for recognizing emotions" refers to technology that analyzes and recognizes the emotional state of a user from their voice.

[1918] "Means for adjusting the tone and content of a document" refers to techniques for changing the expression and content of a document based on the user's emotions and needs.

[1919] "Voice instructions" are instructions or commands about a document given by the user through voice.

[1920] "Means for modifying documents" refers to technology that adds, deletes, or modifies existing documents based on voice instructions.

[1921] The "means for checking and outputting the final version" is a technique for outputting a document that has been finally checked by a user.

[1922] MODE FOR CARRYING OUT THE INVENTION

[1923] The present invention provides a system for generating documents through voice input and adjusting the tone and content of the documents in consideration of a user's emotional state. The system mainly includes a voice input unit, a natural language processing unit, a generative model unit, a customization unit, a voice instruction unit, an emotion engine unit, and an output unit.

[1924] Hardware and Software Configuration

[1925] This system is configured using the following hardware and software.

[1926] 1. Device:

[1927] Audio capture device (e.g. microphone)

[1928] Speech recognition engine (e.g. Google Cloud Speech-to-Text)

[1929] 2. Server:

[1930] Natural language processing tools (e.g., spaCy, BERT)

[1931] Generative modeling tools (e.g., OpenAI GPT-4)

[1932] Database search methods (e.g., SQL databases)

[1933] Emotion engine means (e.g. IBM Watson Tone Analyzer)

[1934] Program processing flow

[1935] When a user speaks, the speech is captured by the device and converted into text using a speech recognition engine. The text data is then sent to a server, where the user's request is analyzed using natural language processing means. Based on the analysis results, a generative model means is activated to generate a first version of the document. The server then searches for past documents in the company's database and customizes the document by applying appropriate templates and style guides. Next, an emotion engine means is used to recognize emotions from the user's voice and adjust the tone and content of the document based on the emotions.

[1936] If the user makes voice corrections to the document, the voice is converted into text again and sent to the server for analysis and generation. Finally, the user gives a final check, and if there are no problems, the server outputs the final version of the document.

[1937] Explanation with concrete examples

[1938] For example, if a user says to a terminal, "I want you to write a marketing strategy report for next week," the terminal captures this speech and uses a speech recognition engine to convert it into text data saying, "I want you to write a marketing strategy report for next week." This text data is sent to a server and analyzed by natural language processing means. Based on the analysis results, a generative model means creates a first edition of a marketing strategy report. The server then references past marketing reports and applies an appropriate template. At the same time, the emotion engine means recognizes emotions from the user's voice, and if it determines, for example, that the user is nervous, it adjusts the tone of the document to be softer.

[1939] When the user instructs, "Update the second paragraph with the new trend information," the device again captures the voice, converts it into text data, and sends it to the server. The server makes the corrections and generates a document containing the new trend information. Finally, when the user instructs, "There are no problems, so send it," the server sends the final document to the specified recipient (e.g., email address or shared folder).

[1940] Prompt Sentence Examples

[1941] User: "I want a marketing strategy report for next week."

[1942] Device: "Converting speech to text"

[1943] Server: "Generating first version"

[1944] Server: "Applying template by referencing past reports"

[1945] Server: "Analyzing user sentiment"

[1946] Server: "Adjusting the tone of the document"

[1947] User: "Update the second paragraph with new trending information."

[1948] Device: "Converting speech to text"

[1949] Server: "Generating updated document"

[1950] User: "No problem, send it."

[1951] Server: "Sending final version to specified location"

[1952] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1953] Step 1: Capture and convert audio input

[1954] 1. The user gives instructions through a voice input device. Example: "Please prepare a proposal for next week."

[1955] 2. The device captures the user's voice using an audio capture device (e.g., a microphone).

[1956] Input: User's voice data

[1957] 3. The device converts the voice data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1958] Data processing: Converting voice data into text data

[1959] Output: Text data "Please create a proposal for next week."

[1960] Step 2: Sending text data

[1961] 1. The terminal sends the converted character data to the server.

[1962] Input: Text data "Please create a proposal for next week."

[1963] Output: Character data sent to the server

[1964] Step 3: Analyzing the text data

[1965] 1. The server analyzes the received text data using natural language processing tools (e.g., spaCy, BERT) to understand the user's request.

[1966] Input: Text data "Please create a proposal for next week."

[1967] Data Computing: Analyzing Request Contents Using Natural Language Processing

[1968] Output: Analysis result "Creating a proposal"

[1969] Step 4: Generate the initial document

[1970] 1. The server launches a generative model (e.g., OpenAI GPT-4) based on the analysis results to generate the first version of the document.

[1971] Input: Analysis result "Create a proposal"

[1972] Data Computation: First Edition Document Generation with Document Generation Algorithms

[1973] Output: First edition proposal

[1974] Step 5: Apply and customize the template

[1975] 1. The server searches the company's database for previously created documents and obtains the templates and style guides to be applied.

[1976] Input: Company database

[1977] Data processing: Searching and retrieving past documents

[1978] Output: Templates and style guides

[1979] 2. The server customizes the generated first version using the retrieved templates and style guides.

[1980] Input: First edition proposals, templates and style guides

[1981] Data calculations: Customization based on template application and style guides

[1982] Output: Customized document

[1983] Step 6: Emotional awareness and tone adjustment

[1984] 1. The server recognizes emotions from the user's voice using an emotion engine means (e.g., IBM Watson Tone Analyzer).

[1985] Input: User's voice data

[1986] Data Computing: Sentiment Analysis

[1987] Output: Emotional state (e.g., "I'm nervous")

[1988] 2. The server adjusts the tone and content of the generated document based on the user's emotions.

[1989] Input: Emotional state "Tense", customized document

[1990] Data operations: tone and content adjustment

[1991] Output: Adjusted document

[1992] Step 7: Processing correction instructions

[1993] 1. The user speaks into the device to instruct the device to revise the document. For example, "Delete the third paragraph and add a new main point."

[1994] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[1995] Input: Spoken correction instructions

[1996] Data processing: Converting voice data into text data

[1997] Output: Text data "Delete the third paragraph and add a new point."

[1998] 3. The terminal sends the converted character data to the server.

[1999] Input: Text data "Delete the third paragraph and add a new point."

[2000] Output: Character data sent to the server

[2001] 4. The server again performs natural language processing to analyze the correction instructions.

[2002] Input: Text data "Delete the third paragraph and add a new point."

[2003] Data calculation: Analysis of correction instructions

[2004] Output: Analysis result "Delete the third paragraph and add a new main point"

[2005] 5. The server modifies the document using the generative model.

[2006] Input: Analysis result "Delete third paragraph and add new main point", adjusted document

[2007] Data Computation: Regeneration by Document Correction Algorithms

[2008] Output: The modified document

[2009] Step 8: Final check and output

[2010] 1. The user issues a final confirmation instruction to the terminal. Example: "Please finalize and submit the report."

[2011] 2. The device captures the user's voice using a voice capture device and converts the voice into text data using a speech recognition engine.

[2012] Input: Final confirmation voice

[2013] Data processing: Converting voice data into text data

[2014] Output: Text data "Please finalize the report and send it."

[2015] 3. The terminal sends the converted character data to the server.

[2016] Input: Text data "Please finalize the report and send it."

[2017] Output: Character data sent to the server

[2018] 4. The server generates the final document and sends it to the terminal.

[2019] Input: Adjusted and amended document

[2020] Output: Final document

[2021] 5. The device will read this final version aloud or display it as text.

[2022] Input: Final document

[2023] Output: Presentation to the user (audio or text display)

[2024] 6. The user performs a final check and, if there are no problems, gives a final voice command. For example, "There are no problems, so please send."

[2025] 7. The server sends the final document to the receiving location specified by the user.

[2026] Input: Final confirmation and sending instructions

[2027] Output: Send the final document to the specified receiving location

[2028] (Application example 2)

[2029] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2030] Modern virtual stores are required to efficiently provide diverse product information and quickly and appropriately address customer questions and concerns. However, conventional systems have difficulty responding flexibly to customer emotions, and have lacked effective means to increase customer satisfaction. In particular, they have been unable to adjust the tone and content of documents generated based on voice input to match the user's emotional state, making it difficult to provide personalized service to customers.

[2031] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2032] In this invention, the server includes means for accepting voice input and converting the user's voice into character data, natural language processing means for analyzing the character data and understanding the requested content, generative model means for generating a document based on the generated character data, means for customizing the document by referring to documents created in the past by the company, means for correcting the document in response to voice instructions, emotion engine means for analyzing the user's emotions from the voice, means for adjusting the tone of the document according to the user's emotional state, and means for checking and outputting the final version. This makes it possible to automate customer service and the generation of product descriptions in a virtual store and provide personalized services according to the customer's emotions.

[2033] "Voice input means" refers to technology for receiving and processing a user's voice.

[2034] "Text data conversion means" refers to a technology that converts the voice captured by the voice input means into text format.

[2035] "Natural language processing means" refers to technology for analyzing text data and understanding the content of user requests.

[2036] "Generative modeling means" refers to technology that automatically generates documents based on analyzed text data.

[2037] "Document customization means" refers to technology that references a company's previously created documents and adjusts the generated documents in accordance with templates and style guides.

[2038] "Voice instruction means" refers to a technology that receives instructions given by voice from a user and modifies a document based on the contents of the instructions.

[2039] The "emotion engine means" refers to a technology that analyzes emotions from the user's voice and identifies the user's emotional state.

[2040] "Tone adjustment" refers to a technique for adjusting the tone and content of a generated document depending on the user's emotional state.

[2041] "Output means" refers to a technique for providing the final generated document to the user in audio or text format.

[2042] The present invention provides a system for generating documents in response to user requests through voice input and adjusting the tone and content of the documents based on the user's emotional state. The system includes a voice input unit, a character data conversion unit, a natural language processing unit, a generative model unit, a document customization unit, a voice instruction unit, an emotion engine unit, a tone adjustment unit, and an output unit.

[2043] System configuration

[2044] The system consists of the following major hardware and software components:

[2045] Hardware: microphone, computer, display device

[2046] software:

[2047] speech_recognition library for implementing speech recognition technology

[2048] Transformers library for natural language processing of text data

[2049] j-hartmann / emotion-english-distilroberta-base model for sentiment analysis

[2050] The rinna / japanese-gpt2-medium model for text generation

[2051] System Operation Overview

[2052] 1. Voice input and conversion

[2053] The user uses a microphone to provide voice input. The device converts the voice data into text using the speech_recognition library. For example, if the user says, "What are the features of this product?", the device converts the voice data into text: "What are the features of this product?"

[2054] 2. Text analysis and first edition generation

[2055] The server receives the converted text data, performs natural language processing using the transformers library, analyzes the user's request, and then generates the first version of the document using the rinna / japanese-gpt2-medium model based on the analysis results.

[2056] 3. Customization and Sentiment Analysis

[2057] The server references the company's past documents and customizes the generated document according to templates and style guides. At the same time, it analyzes the user's voice using the j-hartmann / emotion-english-distilroberta-base model to detect emotions. If the user's voice is recognized as "tense," it adjusts the tone of the document to be more relaxed.

[2058] 4. Correction instructions and final version generation

[2059] The user gives instructions for corrections by voice through a microphone. The device again converts the voice into text, and the server updates the document based on the instructions. For example, if the user instructs the server to "delete the third paragraph and add a new point," the document is updated accordingly.

[2060] 5. Check and final output

[2061] The user then gives a final confirmation, and the server generates the final document and sends it to the device. The device then reads the document aloud or displays it in text. If the user determines there are no problems, the document is sent to their email address, a shared folder, or other location.

[2062] Specific examples

[2063] The user speaks into the device, saying, "Please tell me about the features of this product." The device converts this speech into text data, and the server analyzes the received text data. A product description is generated using the generative AI model rinna / japanese-gpt2-medium, which is then customized based on the company's past documents. At the same time, if the emotion engine detects "tension" in the user's voice, the tone of the text is adjusted to be more relaxed. For example, it could be adjusted to say, "I will explain the features of this product. This product is high quality and can be used safely. Please relax and enjoy it."

[2064] As a concrete example of a prompt, if a user says, "Please tell me the features of this product," the prompt to the generative AI model would be:

[2065] "Please describe the features of this product as follows: This product is high quality, reliable, and very beneficial to the user."

[2066] In this way, the system of the present invention enables personalized document generation based on voice input and in response to the user's emotional state.

[2067] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2068] Step 1:

[2069] The user provides voice input.

[2070] The user speaks a question or request into the microphone, e.g., "What are the features of this product?". Voice data is obtained as input.

[2071] Step 2:

[2072] The terminal converts the voice data into text data.

[2073] The device receives voice data input from the user and converts it into text data using the speech_recognition library. The input is voice data and the output is text data. The converted text data is "Please tell me the features of this product."

[2074] Step 3:

[2075] The server analyzes the text data and understands the request.

[2076] The server receives the converted text data and performs natural language processing using the transformers library, which then analyzes the user's request. The input is text data, and the output is the analyzed request (e.g., "I want to know the product's features").

[2077] Step 4:

[2078] The server generates a document based on the analysis results.

[2079] The server generates the first version of the document using the rinna / japanese-gpt2-medium model based on the parsed request content. The input is the parsed request content, and the output is the generated text. An example of the generated document is: "This product is high quality, reliable, and very useful to users."

[2080] Step 5:

[2081] The server references the company's past documents and customizes the document.

[2082] The server searches the company's database for previously created documents and retrieves the templates and style guides to be applied. The input is the generated text, and the output is the customized document. An example of a customized document is: "This product is high quality, reliable, and very beneficial to users. Please use it with confidence."

[2083] Step 6:

[2084] The server analyzes the user's emotions from the voice.

[2085] The server uses the user's voice data to perform emotion analysis using the j-hartmann / emotion-english-distilroberta-base model. The input is the user's voice data, and the output is the user's emotional state (e.g., "I'm nervous").

[2086] Step 7:

[2087] The server adjusts the tone of the document based on the user's emotional state.

[2088] The server adjusts the tone of the generated document based on the analyzed emotional state. The input is a customized document and the emotional state, and the output is a tone-adjusted document. An example of an adjusted document is: "This product is high quality and reliable, so you can use it with confidence. Please relax and enjoy it."

[2089] Step 8:

[2090] The user gives correction instructions by voice.

[2091] The user issues correction instructions through a microphone, for example, "Delete the third paragraph and add a new main point." The input is voice data, and the output is the voice data of the correction instructions.

[2092] Step 9:

[2093] The device converts the speech back into text and sends it to the server.

[2094] The device converts the voice instructions for correction into text data and sends it to the server. The input is voice data and the output is text data. Converted text data: "Delete the third paragraph and add a new main point."

[2095] Step 10:

[2096] The server updates the document based on the correction instructions.

[2097] The server receives the text data of the correction instructions and updates the document using natural language processing. The input is the text data of the correction instructions and the output is the updated document.

[2098] Step 11:

[2099] The user gives a final confirmation instruction.

[2100] The user uses the microphone to give a final confirmation instruction, e.g., "There are no problems, so send it." The input is voice data, and the output is the voice data of the final confirmation.

[2101] Step 12:

[2102] The server generates and outputs the final document.

[2103] The server generates a final, verified document and sends it to the terminal, which then reads the final document aloud or displays it as text. The input is the updated document, and the output is the final document.

[2104] In this way, a document generation system based on a user's voice input can analyze the user's request, generate an appropriate document according to the user's emotional state, and ultimately provide a personalized document.

[2105] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2106] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2107] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2108] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2109] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2110] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2111] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2112] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2113] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2114] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2115] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2116] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2117] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2118] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2119] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2120] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2121] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2122] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2123] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2124] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configur...

Claims

1. means for accepting voice input and converting the user's voice into text data; natural language processing means for analyzing the character data and understanding the request content; a generative model means for generating a document based on the generated character data; A means to reference and customize documents from a company's past documentation; means for modifying the document in response to voice instructions; A means to review and print the final version; A system including:

2. 10. The system of claim 1, wherein speech recognition technology is used to convert speech input into text data.

3. The system according to claim 1, wherein character data is analyzed using natural language processing technology, and a document is generated using a generative model.

Images