System

Abstract

To provide a system.SOLUTION: Provided is a system including: means for converting voice data obtained by voice recognition means into text data; means for analyzing the text data by a generative AI so as to extract important information, and generating an abridged sentence on the basis of the extracted important information; and means for executing automatization work on the basis of the generated abridged sentence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In sales activities, tasks such as recording business negotiations, creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information are all done manually, and there is a need to improve the efficiency of these tasks. [Means for solving the problem]

[0005] We provide a means to record business negotiations using voice recognition and convert that voice data into text. Furthermore, we provide a means to summarize and organize the text data using generative AI, and automate tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering documents. This will improve the efficiency of sales activities. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[0029] "Example 2"

[0030] The system of the present invention also includes a means for converting recorded voice data into text. This means converts the voice data into text using voice recognition technology. Specifically, voice recognition services such as Google's (registered trademark) voice recognition API and IBM's Watson (registered trademark) can be used.

[0031] "Example 3"

[0032] Furthermore, the system of the present invention is equipped with a generative AI means for summarizing and organizing textual data. This generative AI means uses natural language processing technology to analyze text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and generative AI-3 can be used.

[0033] "Example 4"

[0034] The system of the present invention also includes a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering them. This means automatically generates minutes based on the summary generated by the generative AI means, provides standard answers to questions, checks homework, downloads necessary materials, and registers them in appropriate locations. Specifically, these tasks can be automated through collaboration with a minutes-creation tool or CRM system.

[0035] The processing flow of each embodiment will be described below.

[0036] "Example 1"

[0037] Step 1: During a sales visit, the sales representative launches an application installed on a mobile information device such as a smartphone or tablet.

[0038] Step 2: The application records the conversation and converts the audio data into text using a speech recognition tool such as Google's speech recognition API or IBM's Watson.

[0039] "Example 2"

[0040] Step 1: The recorded voice data is converted into text using a voice recognition device, and the voice data is converted into text data.

[0041] Step 2: The textual data is summarized and organized by a generative AI tool. This generative AI tool uses natural language processing techniques to analyze the text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and Generative AI-3 are used.

[0042] "Example 3"

[0043] Step 1: Minutes are automatically generated based on the summary generated by the generative AI method. Step 2: Similarly, standard answers to questions are provided and homework items are confirmed.

[0044] Step 3: The necessary documents are downloaded and registered in the appropriate location. This automation is achieved through integration with meeting minutes creation tools and CRM systems.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] In conventional sales activities, accurately recording the details of sales negotiations and then summarizing and analyzing them later was a very time-consuming task. In addition, important points were often overlooked during sales negotiations, and when reviewing the records later, the information was often incomplete. This led to problems such as reduced sales efficiency and a decline in the quality of customer service.

[0048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0049] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for sending speech data to the speech recognition engine in real time, a means for inputting prompts to the generative AI model, and a means for displaying responses from the generative AI model. This makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[0050] "Speech recognition means" is a technology that converts voice into digital data, analyzes the content, and outputs it as text data.

[0051] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text data.

[0052] "Generative AI methods" are artificial intelligence technologies that analyze text data and summarize and organize it.

[0053] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[0054] "Standard answers to questions" is a technology that automatically provides pre-defined answers to frequently asked questions.

[0055] "Confirming homework" is the process of confirming the tasks that came up in business negotiations or meetings and the tasks that need to be done by the next time.

[0056] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[0057] "Registration" is the process of entering and storing data or information in a system.

[0058] "Means for transmitting voice data to a voice recognition engine in real time" refers to a technology that immediately transmits recorded voice data to a voice recognition engine and analyzes it in real time.

[0059] A "means for inputting prompt sentences to a generative AI model" is an interface that allows a user to input instructions or questions to a generative AI model.

[0060] "Means for displaying responses from the generative AI model" refers to technology that displays answers or results returned from the generative AI model to the user.

[0061] MODE FOR CARRYING OUT THE INVENTION

[0062] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[0063] Hardware and software used

[0064] Hardware: Smartphones, tablets, cloud servers (e.g., AWS (registered trademark) EC2)

[0065] Software: Google Cloud Speech-to-Text API, MySQL® database, OpenAI® GPT-3®

[0066] Program processing explanation

[0067] 1. Launch the app

[0068] Users launch the app on their smartphones or tablets during sales calls.

[0069] Example: A user taps the "Sales Assistant" app on the iPhone home screen to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[0070] 2. Start voice recognition

[0071] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[0072] Example: When a user taps the "Start Recording" button in the app, the device's microphone activates and begins recording the business conversation. During recording, the recording time is displayed on the app screen.

[0073] 3. Audio Data Processing

[0074] The device sends the recorded voice data to the voice recognition engine in real time.

[0075] Software used: Google Cloud Speech-to-Text API

[0076] Example: The device sends recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is temporarily stored on the device.

[0077] 4. Saving text data

[0078] The server stores the text data received from the speech recognition engine in a database.

[0079] Hardware used: Cloud server (e.g. AWS EC2)

[0080] Software used: MySQL database

[0081] Example: The server receives text data sent from a device and saves it in a MySQL database on AWS EC2. When saving is complete, the server sends a notification to the device that saving is complete.

[0082] 5. Input to the generative AI model

[0083] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[0084] Software used: OpenAI GPT-3

[0085] Example: A user enters a prompt such as "Please write a summary of this deal" into an input field in an app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[0086] 6. Displaying the results

[0087] The device receives the response from the generative AI model and displays it to the user.

[0088] Example: The device receives a summary and analysis of a business deal from the generative AI model and displays it within the app. The user can check the summary and analysis on the app screen.

[0089] Examples of prompt statements

[0090] "Please write a summary for this opportunity"

[0091] "Analyze the content of the business negotiation and tell us the key points to propose next time."

[0092] The above is an embodiment of the sales assistant system of the present invention. This system makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[0093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0094] Step 1:

[0095] Launching the app

[0096] Users launch the app on their smartphones or tablets during sales calls.

[0097] Specific behavior: The user taps the "Sales Assistant" app on the home screen of their smartphone to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[0098] Input: User taps

[0099] Output: App home screen display

[0100] Step 2:

[0101] Start voice recognition

[0102] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[0103] Specific operation: When the user taps the "Start Recording" button in the app, the device's microphone will be activated and the business conversation will begin to be recorded. During recording, the recording time will be displayed on the app screen.

[0104] Input: User taps

[0105] Output: Start recording, display recording time

[0106] Step 3:

[0107] Audio data processing

[0108] The device sends the recorded voice data to the voice recognition engine in real time.

[0109] Software used: Google Cloud Speech-to-Text API

[0110] How it works: The device sends the recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data, which is then temporarily stored on the device.

[0111] Input: Recorded audio data

[0112] Output: Text data

[0113] Step 4:

[0114] Saving text data

[0115] The server stores the text data received from the speech recognition engine in a database.

[0116] Hardware used: Cloud server (e.g. AWS EC2)

[0117] Software used: MySQL database

[0118] Specific operation: The server receives the text data sent from the device and saves it in a MySQL database on AWS EC2. Once saving is complete, the server sends a notification to the device.

[0119] Input: Text data

[0120] Output: Saving to database, notification of completion of saving

[0121] Step 5:

[0122] Input to generative AI models

[0123] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[0124] Software used: OpenAI GPT-3

[0125] Specific operation: The user enters the prompt "Please write a summary of this deal" into the input field in the app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[0126] Input: prompt, text data

[0127] Output: Send to generative AI model

[0128] Step 6:

[0129] Displaying the results

[0130] The device receives the response from the generative AI model and displays it to the user.

[0131] Specific operation: The device receives the summary and analysis results of the business negotiation received from the generative AI model and displays them within the app. The user can check the summary and analysis results on the app screen.

[0132] Input: Response from a generative AI model

[0133] Output: Display of summary and analysis results

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Conventional sales assistant systems were capable of recording and transcribing sales negotiations, but lacked the functionality to analyze the content of sales negotiations and automatically generate next actions and proposals. As a result, sales staff had to analyze the content themselves after the negotiation and consider the next steps, which was time-consuming and labor-intensive. In addition, summarizing and organizing the content of sales negotiations had to be done manually, which was inefficient. A system that could solve these problems and make sales activities more efficient was needed.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, and a means for analyzing the content of business negotiations using a generative AI model and automatically generating next actions and proposals. This makes it possible to consistently automate the process from recording the content of business negotiations to analyzing it and proposing the next action.

[0139] "Speech recognition means" is a technology that acquires voice data and converts it into text data.

[0140] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[0141] "Generative AI methods" are technologies that summarize and organize text data, and then use generative AI models to analyze and make suggestions.

[0142] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[0143] "Standard Answers to Questions" is a feature that automatically provides pre-set answers to frequently asked questions.

[0144] "Homework review" is the process of tracking and reviewing issues and next steps that arise from a business meeting or negotiation.

[0145] "Download materials" is a function that allows you to obtain necessary materials from the Internet or an internal network.

[0146] "Registration" is the process of storing and managing acquired data and information in a system.

[0147] A "generative AI model" is an artificial intelligence technology that generates new information and suggestions based on text data.

[0148] "Analyzing the content of business negotiations" is the process of analyzing the content discussed during business negotiations and extracting important points and next actions.

[0149] "Means for automatically generating next actions and proposals" refers to technology that automatically generates next actions and proposals based on the content of business negotiations.

[0150] The following system configuration will be described as an embodiment of the present invention.

[0151] System Configuration

[0152] This system includes speech recognition means, text conversion means, generative AI means, minutes creation, standard answers to questions, homework confirmation, document download and registration, analysis of business negotiation content using generative AI models, and means for automatically generating next actions and proposals. These means are realized as applications installed on mobile information terminals such as smartphones and tablets.

[0153] Hardware and software used

[0154] Hardware: Smartphone (iOS or ANDROID (registered trademark)), tablet

[0155] Software: Python (registered trademark), speech recognition library, transformers library

[0156] Processing flow

[0157] 1. Voice recording: The user starts the smartphone app and records the details of the business negotiation. The details of the business negotiation are captured as voice data using voice recognition.

[0158] 2. Speech recognition: The server uses a speech recognition library to convert the recorded voice data into text data.

[0159] 3. Texting: The audio data is converted into text format through textual processing.

[0160] 4. Data Summarization and Organizing: Summarize and organize text data using generative AI methods.

[0161] 5. Analyze the content of sales negotiations: Using generative AI models, analyze the content of sales negotiations and automatically generate next actions and proposals.

[0162] 6. Automation functions: Automate functions such as creating minutes, standard answers to questions, checking homework, downloading materials, and registering.

[0163] Specific examples

[0164] For example, if a customer says, "I'd like to discuss the introduction of a new product" during a sales negotiation, the user can launch an app on their smartphone and record the content of the negotiation. The recorded voice data is converted into text data using a speech recognition tool. The text data is then summarized and organized using a generative AI tool. The content of the sales negotiation is analyzed using a generative AI model, and a proposal such as "As a next step regarding the introduction of the new product, I suggest scheduling a demonstration" is automatically generated.

[0165] Prompt Sentence Examples

[0166] Negotiation details: We would like to discuss the introduction of a new product.

[0167] Next actions and suggestions:

[0168] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] The user launches the smartphone app and records the contents of the business meeting. The input is the audio data of the business meeting, and the output is the recorded audio file. Specifically, the user presses the record button on the app and then presses the stop button when the business meeting is over.

[0172] Step 2:

[0173] The server converts the recorded audio file into text data using a speech recognition means. The input is the audio file and the output is text data. Specifically, the server analyzes the audio data using a speech recognition library and generates corresponding text.

[0174] Step 3:

[0175] The server uses generative AI tools to summarize and organize text data. The input is text data, and the output is summarized text data. Specifically, the server uses a generative AI model to extract important parts of the text and create a summary.

[0176] Step 4:

[0177] The server uses a generative AI model to analyze the content of the business negotiation and automatically generate the next action and proposal. The input is summarized text data, and the output is text data containing the next action and proposal. Specifically, the server inputs a prompt sentence into the generative AI model and obtains the generated text.

[0178] Step 5:

[0179] The server automates functions such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering. The input is text data of the business negotiations, and the output is automatically generated minutes, proposals, and confirmation items. Specifically, the server executes the processes corresponding to each function and generates and saves the necessary information.

[0180] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[0181] Example 2

[0182] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] In conventional sales assistant systems, the process of recording voice data, converting it to text, summarizing, and organizing it was often done manually, resulting in inefficiency. Furthermore, it was difficult to use highly accurate voice recognition technology to convert voice data into text, resulting in low quality text data. Furthermore, automation of tasks such as creating meeting minutes and standard responses to questions based on the generated text data was insufficient, placing a heavy burden on users.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0185] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for transmitting recorded speech data from a terminal to the server, a means for the server to call a speech recognition service and convert the speech data into text data, a means for transmitting the converted text data to the terminal and displaying it, and a means for automating the creation of minutes, standard answers to questions, confirmation of homework items, downloading materials, registration, etc. This makes it possible to consistently and efficiently perform processes from recording speech data to converting it into text, summarizing, organizing, and even automating various tasks.

[0186] "Speech recognition means" refers to technology that analyzes voice data and converts it into corresponding text data.

[0187] The "means for converting text" refers to a technology for recording voice data obtained by a voice recognition means as text data.

[0188] "Generative AI methods" refers to artificial intelligence techniques for summarizing and organizing written data.

[0189] "Means for transmitting recorded voice data from a terminal to a server" refers to a technology for uploading voice data recorded on a terminal to a server via a network.

[0190] "Means for the server to call a voice recognition service and convert voice data into text data" refers to a technology in which the server uses an external voice recognition service to convert voice data into text data.

[0191] "Means for transmitting the converted text data to the terminal and displaying it" refers to a technology for transmitting the text data converted by the server to the terminal and displaying the text data on the terminal.

[0192] "Means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering" refers to technology that uses generated text data to automatically perform tasks such as creating minutes, answering questions, checking homework, downloading materials, and registering materials.

[0193] MODE FOR CARRYING OUT THE INVENTION

[0194] This invention is a system that efficiently converts voice data into text data and automates various tasks based on the text data. This system includes a voice recognition means, a text generation means, a generative AI means, a means for transmitting recorded voice data from a terminal to a server, a means for the server to call a voice recognition service and convert the voice data into text data, a means for transmitting the converted text data to a terminal and displaying it, and a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering.

[0195] Hardware and software used

[0196] Hardware: smartphones, PCs, servers

[0197] Software: Speech recognition APIs (e.g., general speech recognition services), generative AI models

[0198] Specific operation of the system

[0199] 1. A user records audio data using a recording app on a smartphone or computer. For example, to record the contents of a meeting, the user presses the record button on their smartphone to record the audio of the meeting.

[0200] 2. The device sends the recorded audio data to the server. Specifically, the recording app uploads the audio data to the server as an HTTP request.

[0201] 3. The server temporarily stores the received voice data and prepares it for the next process.

[0202] 4. The server calls the speech recognition service and converts the voice data into text data. Specifically, the server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text.

[0203] 5. The voice recognition service analyzes the received voice data and converts it into text data.

[0204] 6. The server receives the converted text data and sends it to the user's device. Specifically, the server returns the text data to the user's device as an HTTP response.

[0205] 7. The device displays the received text data to the user. For example, a smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[0206] Specific examples

[0207] For example, when a user records the contents of a meeting and converts it into text, the specific operation is as follows.

[0208] 1. The user launches the recording app on their smartphone and presses the record button to record the audio of the meeting.

[0209] 2. The device sends the recorded audio data to the server. For example, a recording app uploads a file called "MeetingRecording.wav" to the server as an HTTP request.

[0210] 3. The server receives the "Meeting Recording.wav" and temporarily stores it.

[0211] 4. The server calls the speech recognition service and sends "Meeting Recording.wav" to the API.

[0212] 5. The voice recognition service analyzes the voice data and generates text data such as "The contents of the meeting have been transcribed into text."

[0213] 6. The server receives the generated text data and sends it to the user's smartphone.

[0214] 7. The device displays the received text data on the app screen, allowing the user to check the contents of the meeting as text.

[0215] Prompt Sentence Examples

[0216] Examples of prompts to be input to a generative AI model include:

[0217] "Please convert the recorded audio data of the meeting into text data. The audio data can be obtained from the following URL. Please use a general speech recognition service for the conversion."

[0218] In this way, a system can be constructed that can efficiently convert recorded voice data into text data.

[0219] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0220] Step 1:

[0221] Users record audio data using a recording app on their smartphone or computer.

[0222] Input: Audio from meetings, business meetings, etc.

[0223] Output: Recorded audio data file (e.g., MeetingRecording.wav)

[0224] Specific operation: The user launches the recording app, presses the record button to record the audio of the meeting, and when the recording is finished, presses the save button to save the audio data.

[0225] Step 2:

[0226] The terminal transmits the audio data to the server after recording is completed.

[0227] Input: Recorded audio data file (e.g., MeetingRecording.wav)

[0228] Output: Audio data uploaded to the server

[0229] Specific operation: The recording app uploads the audio data to the server as an HTTP request. The user presses the send button to send the audio data to the server.

[0230] Step 3:

[0231] The server temporarily stores the received voice data in preparation for the next process.

[0232] Input: Uploaded audio data

[0233] Output: Audio data stored on the server

[0234] Specific operation: The server temporarily stores the received voice data in storage in preparation for calling the voice recognition service.

[0235] Step 4:

[0236] The server invokes a speech recognition service to convert the speech data into text data.

[0237] Input: Audio data stored on the server

[0238] Output: Audio data sent to the speech recognition service

[0239] Specific operation: The server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text. For example, it creates an API request, attaches the voice data, and sends it.

[0240] Step 5:

[0241] The voice recognition service analyzes the received voice data and converts it into text data.

[0242] Input: Audio data sent to the speech recognition service

[0243] Output: Converted text data

[0244] What it does: A speech recognition service analyzes audio data and generates corresponding text data, for example, by converting speech to text using a speech recognition algorithm.

[0245] Step 6:

[0246] The server receives the converted text data and sends it to the user's terminal.

[0247] Input: Converted text data

[0248] Output: Text data sent to the user's device

[0249] Specific operation: The server receives the text data returned from the speech recognition service and returns it to the user's device as an HTTP response.

[0250] Step 7:

[0251] The terminal displays the received text data to the user.

[0252] Input: Text data sent from the server

[0253] Output: Text data displayed on the terminal

[0254] Specific operation: The smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[0255] (Application example 2)

[0256] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] While systems using conventional speech recognition technology were able to convert recorded audio data into text, they lacked the ability to generate and display subtitles for video content in real time. This made it difficult to immediately understand the audio content when watching video content, limiting the viewing experience, especially when watching content in a different language.

[0258] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for generating subtitles for video content in real time, and a display means for displaying subtitles. This makes it possible to recognize the voice of video content in real time and instantly display it as subtitles.

[0259] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[0260] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text format.

[0261] "Generative AI methods" are artificial intelligence techniques for summarizing and organizing text data.

[0262] "Means for generating subtitles for video content in real time" refers to technology that analyzes the audio data of video in real time and displays it as subtitles.

[0263] The "display means for displaying subtitles" is a technique for displaying generated subtitles superimposed on video frames.

[0264] The following system configuration will be described as an embodiment of the present invention.

[0265] System Configuration

[0266] This system includes a speech recognition means, a text generation means, a generative AI means, a means for generating subtitles in real time for video content, and a display means for displaying the subtitles.

[0267] Voice recognition means

[0268] The server uses a technology to analyze voice data and convert it into text data as a means of speech recognition, using a common speech recognition API (such as Google's speech recognition API or IBM's Watson).

[0269] Means of writing

[0270] The speech data obtained by the speech recognition means is converted into text format, and this converted text data is used for subsequent processing.

[0271] Generative AI means

[0272] Generative AI methods use artificial intelligence techniques to summarize and organize text data, including natural language processing (NLP) models. For example, generative AI models such as GPT-3 can be used.

[0273] A means of generating subtitles for video content in real time

[0274] The server uses technology to analyze the audio data of the video in real time and display it as subtitles. It generates subtitles in real time based on the text data obtained by the speech recognition means.

[0275] Display means for displaying subtitles

[0276] We use a technique to overlay the generated subtitles onto the video frames. Specifically, we use an image processing library such as OpenCV to add subtitles to the video frames.

[0277] Specific examples

[0278] For example, consider an application that generates English subtitles in real time while a user is watching a Japanese movie. This application is installed on a smartphone or head-mounted display, and while the user is watching the video, it converts the recorded audio data into text data in real time and displays it as subtitles.

[0279] Prompt Sentence Examples

[0280] "Create an app that generates English subtitles in real time while watching a Japanese movie. Use Google's Speech Recognition API for speech recognition and Google Translate API for translation. The subtitles should be displayed in the video frame."

[0281] In this way, the mode for carrying out the invention can be concretely shown.

[0282] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0283] Step 1:

[0284] The server acquires the audio data of the video content. Specifically, it extracts the audio track from the video file and saves it as audio data. The input is the video file, and the output is the audio data.

[0285] Step 2:

[0286] The server converts the voice data into text data using a voice recognition means. Specifically, it calls a voice recognition API, analyzes the voice data, and generates text data. The input is voice data, and the output is text data.

[0287] Step 3:

[0288] The server uses generative AI tools to summarize and organize text data. Specifically, it uses a natural language processing model to summarize the text data and extract the necessary information. The input is text data, and the output is summarized text data.

[0289] Step 4:

[0290] The server generates subtitles in real time based on the summarized text data. Specifically, it converts the generated text data into a subtitle format and adds a timestamp. The input is the summarized text data, and the output is the subtitle data.

[0291] Step 5:

[0292] The server overlays subtitle data onto video frames and displays them in real time using an image processing library. The input is subtitle data and video frames, and the output is video frames with added subtitles.

[0293] Step 6:

[0294] The user watches video frames with added subtitles. Specifically, they use a smartphone or a head-mounted display to watch videos with subtitles displayed in real time. The input is the video frames with added subtitles, and the output is the viewing experience.

[0295] Example 3

[0296] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0297] In conventional sales assistant systems, the transcription, summarization, and organization of voice data was often done manually, which was time-consuming and labor-intensive. Furthermore, tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering were not automated, resulting in inefficiency and a high likelihood of human error. Furthermore, there was a lack of a way to extract important information and execute automated tasks based on that information, which led to a decline in the accuracy and efficiency of the work.

[0298] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0299] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summaries. This enables the automatic textualization, summarization, and organization of speech data, and automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, improving the efficiency and accuracy of tasks.

[0300] "Speech recognition means" is a technology for analyzing voice data and converting it into text data.

[0301] The "text conversion means" is a technology that executes a process of converting the voice data obtained by the voice recognition means into text data.

[0302] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries based on that information.

[0303] "Minutes writing" is the process of recording the contents of a meeting or conference and creating a document that summarizes it.

[0304] "Standard answers to questions" is the process of providing predefined answers to questions posed by users.

[0305] "Homework review" is the process of reviewing action items and tasks decided upon in meetings and tracking their progress.

[0306] "Downloading materials" is the process of automatically obtaining required materials from the Internet or internal systems.

[0307] "Registration" is the process of storing and managing generated data and documents in an appropriate location.

[0308] "Means for analyzing text data and extracting important information" refers to technology that analyzes text data and extracts important keywords and phrases from it.

[0309] "Means for performing automated tasks based on generated summaries" refers to technology that automatically performs tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information based on generated summaries.

[0310] MODE FOR CARRYING OUT THE INVENTION

[0311] This invention is a system including a speech recognition means, a text generation means, a generative AI means, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summary.

[0312] 1. Generating the system program

[0313] The server uses a speech recognition means to acquire voice data and generates a program that converts it into text data. Next, it uses a generative AI means to analyze the text data, extract important information, and generate a program that generates a summary based on that information. Furthermore, it generates a program that automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering based on the generated summary.

[0314] 2. Explanation of program processing

[0315] The server uses voice recognition software to convert the voice data provided by the user into text data. Specifically, voice recognition software such as Google Speech-to-Text can be used. The converted text data is sent to the server.

[0316] Next, the server uses natural language processing models such as BERT and Generative AI-3 as generative AI tools to analyze the text data. As a result of the analysis, important information is extracted and a summary is generated. This summary is then stored on the server.

[0317] Furthermore, the server performs the following automated tasks based on the generated summary:

[0318] Automatic generation of meeting minutes

[0319] Providing standard answers to questions

[0320] Checking homework

[0321] Download materials

[0322] Data registration

[0323] These tasks are automated through integration with meeting minutes-taking tools and CRM systems.

[0324] 3. Examples of concrete examples and prompts

[0325] Examples:

[0326] When a user creates minutes of a meeting, the audio data of the meeting is converted into text and the text data is sent to the server. The server then analyzes the text data using BERT to extract important topics and decisions. The server then generates a summary based on the extracted information and automatically generates the minutes.

[0327] Example prompt sentence:

[0328] "Please summarize the text data from the following meeting and prepare minutes:

[0329] 1. Meeting start and end times

[0330] 2. Participant's name

[0331] 3. Agenda and details

[0332] 4. Decisions and action items

[0333] By inputting this prompt sentence into the generative AI model, the server automatically generates a summary sentence and creates the minutes. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0334] Program processing flow

[0335] Step 1: Input audio data

[0336] Users record audio data of meetings and business negotiations, and the recorded audio data is saved on the device.

[0337] Specific behavior:

[0338] The user starts the recording application and records the audio of a meeting or business negotiation.

[0339] Once the recording is complete, the audio data is saved on the device.

[0340] Input: Audio data from meetings and business negotiations

[0341] Output: Audio data stored on the device

[0342] Step 2: Converting audio data to text

[0343] The device uses voice recognition software to convert the recorded voice data into text data, which is then sent to the server.

[0344] Specific behavior:

[0345] The device launches voice recognition software (e.g., Google Speech-to-Text).

[0346] The audio data is input into the software and converted into text data.

[0347] The converted text data is sent to the server.

[0348] Input: Audio data stored on the device

[0349] Output: Text data sent to the server

[0350] Step 3: Analyzing the text data

[0351] The server uses generative AI tools (e.g., BERT or Generative AI-3) to analyze the received text data. As a result of the analysis, important information is extracted.

[0352] Specific behavior:

[0353] The server activates the generative AI means.

[0354] Text data is input into a generative AI tool and analyzed.

[0355] Extract important keywords and phrases.

[0356] Input: Text data sent to the server

[0357] Output: Extracted important information

[0358] Step 4: Generate a summary

[0359] The server generates a summary based on the extracted important information, and the generated summary is stored on the server.

[0360] Specific behavior:

[0361] The server inputs the extracted information into a summary generation model.

[0362] A summary sentence is generated using a summary sentence generation model.

[0363] The generated summary is stored on the server.

[0364] Input: Extracted important information

[0365] Output: Generated summary

[0366] Step 5: Execute the automation

[0367] The server performs the following automated tasks based on the generated summary:

[0368] 1. Automatic generation of meeting minutes

[0369] 2. Providing standard answers to questions

[0370] 3. Check homework

[0371] 4. Download materials

[0372] 5. Data Registration

[0373] Specific behavior:

[0374] The server automatically generates minutes based on the summary and sends them to the user.

[0375] The server generates and provides standard answers to questions from users based on the summary text.

[0376] The server checks the homework items based on the summary and notifies the user.

[0377] The server automatically downloads the necessary materials and provides them to the user.

[0378] The server registers the generated data in the appropriate location (e.g., a CRM system).

[0379] Input: Generated summary

[0380] Output: Execution results of automated tasks

[0381] (Application example 3)

[0382] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0383] In conventional sales assistant systems and factory work efficiency systems, the texting, summarization, and organization of voice data was often done manually, resulting in problems with reduced work efficiency. Furthermore, they lacked functions such as summarizing and organizing work instructions, automatically recording work progress, providing standard responses to worker questions, and automatically downloading and registering necessary parts and materials, and as a result, work efficiency was not sufficiently improved. This increased the burden on workers and led to issues with reduced overall work efficiency.

[0384] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0385] In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading and registering materials, a means for summarizing and organizing work instructions, a means for automatically recording work progress, a means for providing standard answers to questions from workers, and a means for automatically downloading and registering necessary parts and materials. This enables efficient textualization, summarization, and organization of voice data, and further automates the summarization and organization of work instructions, the automatic recording of work progress, the standard answers to questions from workers, and the automatic downloading and registration of necessary parts and materials, thereby improving overall work efficiency.

[0386] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[0387] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[0388] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries.

[0389] Minute-taking is the process of recording the contents of a meeting or discussion so that it can be referenced later.

[0390] "Standard Answers to Questions" is a feature that provides pre-prepared answers to frequently asked questions.

[0391] "Checking homework" is the process of checking the progress of assignments and tasks given in meetings and consultations.

[0392] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[0393] "Registration" is the process of storing acquired data and materials in an appropriate location.

[0394] "Means for summarizing and organizing work instructions" refers to a technique for concisely summarizing and organizing the contents of work instructions.

[0395] "Means for automatically recording work progress" refers to technology that automatically records the progress of work.

[0396] The "means for providing standard answers to questions from workers" is a technique for providing answers prepared in advance to questions from workers.

[0397] "Means for automatically downloading and registering necessary parts and materials" refers to technology that automatically obtains the parts and materials necessary for the work and stores them in the appropriate location.

[0398] To implement the present invention, the following system configuration and program are required.

[0399] System Configuration

[0400] The system consists of the following main components:

[0401] 1. Speech recognition: This involves analyzing voice data and converting it into text using technology such as speech recognition software or APIs (e.g., Google Speech-to-Text API).

[0402] 2. Text conversion means: A process is carried out to convert the voice data obtained by the voice recognition means into text format.

[0403] 3. Generative AI: Analyzes text data using natural language processing techniques, extracts important information, and generates summaries. Specifically, generative AI models such as BERT and GPT-3 are used.

[0404] 4. Means of automating tasks such as taking minutes, providing standard answers to questions, checking homework, downloading materials, and registering: To achieve these functions, we will integrate with CRM systems and minute-taking tools.

[0405] 5. Summarize and organize work orders: Use software to summarize and organize work orders concisely.

[0406] 6. Means of automatic recording of work progress: Use a system to automatically record the progress of work.

[0407] 7. Means of providing standard answers to worker questions: Use an FAQ system to provide pre-prepared answers to worker questions.

[0408] 8. Means of automatic download and registration of required parts and materials: Use a download manager to automatically obtain required parts and materials and save them in the appropriate location.

[0409] Program processing explanation

[0410] The server uses speech recognition to convert voice data into text data. Next, it uses generative AI to analyze the text data, extract important information, and generate a summary. Based on this summary, it automates tasks such as creating meeting minutes, providing standard answers to questions, confirming homework assignments, downloading and registering materials, and more. It also summarizes and organizes work instructions, automatically records work progress, provides standard answers to questions from workers, and automatically downloads and registers required parts and materials.

[0411] Specifically, the following hardware and software are used.

[0412] Hardware: Server equipped with a high-performance GPU

[0413] Software: Python 3.x, Transformers library, PyTorch, Google Speech-to-Text API, CRM system, meeting minutes tool, FAQ system, download manager

[0414] Specific examples

[0415] For example, a factory work instruction manual contains multiple steps and important points to note. By using this system, workers can quickly grasp important information and proceed with their work efficiently.

[0416] Example prompt for a generative AI model:

[0417] Work order text:

[0418] 1. Attach part A.

[0419] 2. When installing part B, use a torque wrench.

[0420] 3. Once completed, quality inspection is carried out.

[0421] By inputting this prompt, the generative AI model extracts important information and generates a summary.

[0422] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0423] Step 1:

[0424] The server converts the voice data into text data using a voice recognition means.

[0425] Input: Audio data

[0426] Data processing: Using voice recognition software (e.g., Google Speech-to-Text API), we analyze the voice data and convert it into text data.

[0427] Output: Text data

[0428] Step 2:

[0429] The server uses generative AI tools to analyze the text data, extract important information, and generate a summary.

[0430] Input: Text data

[0431] Data Computing: Using generative AI models such as BERT and GPT-3 to extract important information from text data and generate summaries.

[0432] Output: Summary

[0433] Step 3:

[0434] The server automatically generates minutes based on the generated summary.

[0435] Input: Abstract

[0436] Data processing: Organize the summaries according to the format and save them as minutes.

[0437] Output: Meeting minutes

[0438] Step 4:

[0439] The server provides standard answers to questions posed by the user.

[0440] Input: User question

[0441] Data Computing: Use an FAQ system to find and provide standard answers to questions.

[0442] Output: Standard answer

[0443] Step 5:

[0444] The server checks the homework items.

[0445] Input: List of homework items

[0446] Data processing: Check the progress of homework items and update the list.

[0447] Output: Updated list of homework items

[0448] Step 6:

[0449] The server automatically downloads the necessary materials and registers them in the appropriate locations.

[0450] Input: URL or identification information of the material

[0451] Data processing: Use the download manager to obtain the materials and save them in the specified folder.

[0452] Output: Saved material

[0453] Step 7:

[0454] The server summarizes and organizes the work orders.

[0455] Input: Work order text

[0456] Data Computation: Using generative AI models to concise and organize work instructions.

[0457] Output: Summarized work instructions

[0458] Step 8:

[0459] The server automatically records the progress of the work.

[0460] Input: Work progress data

[0461] Data processing: Work progress is automatically recorded and saved in a database.

[0462] Output: Recorded work progress data

[0463] Step 9:

[0464] The server provides standard answers to questions from workers.

[0465] Input: Worker Question

[0466] Data Computing: Use an FAQ system to find and provide standard answers to questions.

[0467] Output: Standard answer

[0468] Step 10:

[0469] The server automatically downloads the necessary parts and materials and registers them in the appropriate location.

[0470] Input: Part or material identification information

[0471] Data processing: Use the download manager to obtain parts and materials and save them in the specified folder.

[0472] Output: Saved parts and documents

[0473] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0474] "Example 1"

[0475] In one embodiment of the present invention, the speech recognition means includes an emotion engine that recognizes the user's emotion. This emotion engine analyzes the emotion from the user's speech and transmits the result along with the speech data to the generative AI means. Specifically, when the user utters a speech that indicates an emotion such as joy, anger, or sadness, the emotion engine recognizes the emotion and transmits it along with the speech data to the generative AI means.

[0476] "Example 2"

[0477] The generative AI means receives the voice data and emotion data and summarizes and organizes the written data taking them into account. Specifically, if the user makes a sound that indicates joy, the generative AI means summarizes and organizes the written data taking that information into account. This allows for the generation of more appropriate minutes and answers to questions that reflect the user's emotions.

[0478] "Example 3"

[0479] The automation means takes into account the summary, organized data, and emotional data received from the generative AI means to create minutes, provide standard answers to questions, confirm homework, download materials, register, etc. Specifically, if a user makes a sound that indicates anger, that information is taken into account when creating minutes or providing answers to questions. This allows for more appropriate task automation that reflects the user's emotions.

[0480] The processing flow of each embodiment will be described below.

[0481] "Example 1"

[0482] Step 1: The user speaks.

[0483] Step 2: The voice recognition means recognizes the user's voice, and the emotion engine analyzes the emotion from the voice.

[0484] Step 3: The emotion engine sends the analysis results along with the voice data to the generative AI means. "Example 2"

[0485] Step 1: The generative AI means receives the voice data and emotion data.

[0486] Step 2: The generative AI tool summarizes and organizes the written data, taking into account the audio data and emotion data.

[0487] Step 3: The generative AI method sends the summarized and organized data to the automated method.

[0488] "Example 3"

[0489] Step 1: The automated method receives the summarized and organized data and sentiment data.

[0490] Step 2: The automated means takes into account the received data and performs tasks such as creating meeting minutes, providing standard answers to questions, checking homework items, downloading materials, registering, etc.

[0491] Example 1

[0492] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0493] Conventional sales assistant systems provide basic functions such as recording, transcribing, summarizing, and organizing sales negotiations, but lack the ability to analyze users' emotions and provide advanced suggestions and feedback based on those emotions. This makes it difficult to automatically suggest specific actions to improve the quality of sales negotiations.

[0494] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering, an emotion analysis means for analyzing the user's emotions, and a means for transmitting the emotion data obtained by the emotion analysis means to the generative AI means. This enables advanced suggestions and feedback based on the user's emotions.

[0495] "Speech recognition means" is a technology for acquiring speech as digital data and analyzing its content.

[0496] "Means of converting text" refers to technology that converts voice data into text data.

[0497] "Generative AI means" refers to artificial intelligence technology that performs processes such as summarizing, organizing, and proposing based on input data.

[0498] "Minutes-taking" is the process of recording the contents of a meeting or business negotiation so that it can be referenced later.

[0499] "Canonical Answers to Questions" is the process of providing pre-prepared answers to frequently asked questions.

[0500] "Homework review" is the process of reviewing assignments and tasks that came up in a business meeting or negotiation and tracking their progress.

[0501] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[0502] "Registration" is the process of entering and storing data or information in a system.

[0503] "Emotion analysis means" is a technology that analyzes the speaker's emotions from voice data and obtains the results.

[0504] "Emotion data" is information about the speaker's emotions obtained by emotion analysis means.

[0505] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales visit and records the details of the business negotiation. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[0506] Hardware and software used

[0507] Hardware

[0508] Smartphones (e.g., smartphones, Android devices)

[0509] Tablets (e.g. iPad (registered trademark), Android tablets)

[0510] software

[0511] Sales assistant apps (e.g. iOS apps, Android apps)

[0512] Sentiment analysis tools (e.g., IBM Watson Tone Analyzer, Microsoft® Azure® Emotion API)

[0513] Generative AI means (e.g. OpenAI GPT-3, Google BERT)

[0514] Program processing

[0515] 1. Launch the app

[0516] A user launches the sales assistant app on their smartphone or tablet.

[0517] Specific action: The user taps the Sales Assistant app icon on the smartphone home screen.

[0518] 2. Start voice recognition

[0519] After the app is launched, the device starts the voice recognition function to record the details of the business meeting.

[0520] Specific operation: When the user presses the "Start Recording" button in the app, the device's microphone begins recording the audio of the business meeting.

[0521] 3. Performing sentiment analysis

[0522] The terminal analyzes the recorded voice data in real time and recognizes the user's emotions using emotion analysis means.

[0523] Specific behavior: When a user says, "This product is really great!", the device's emotion analysis means (e.g., IBM Watson Tone Analyzer) recognizes the emotion of "joy."

[0524] 4. Data transmission

[0525] The device transmits the analyzed emotion data and voice data to the generative AI means.

[0526] Specific operation: After the emotion analysis means recognizes the emotion of "joy," the device sends that information along with the voice data to a generative AI means (e.g., OpenAI GPT-3) on the cloud.

[0527] 5. Processing by generative AI models

[0528] Based on the voice and emotion data received by the server, the generative AI model summarizes the content of the business negotiation and suggests the next action.

[0529] Specific operation: The server analyzes the received data, and in response to a statement such as "This product is really great!", the generative AI model generates a summary and suggestion such as "The user is very satisfied with the product, so we suggest providing additional product information on their next visit."

[0530] Specific examples

[0531] Prompt Sentence Examples

[0532] "When a user makes a sales call, they launch the app and record the details of the sales meeting. The recorded voice data is analyzed by the emotion analysis means to recognize the user's emotions. For example, if the user says, 'This product is truly amazing!', the emotion analysis means recognizes the emotion of 'joy' and sends that information along with the voice data to the generative AI model. Based on the received data, the generative AI model summarizes the details of the sales meeting and suggests the next action."

[0533] In this way, the sales assistant system of the present invention can improve the quality of sales negotiations by recording the user's voice, analyzing their emotions, and sending them to the generative AI model.

[0534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0535] Step 1:

[0536] A user launches the sales assistant app on a smartphone or tablet. The input is the user tapping the app icon. The output is the app launching and the main screen being displayed.

[0537] Step 2:

[0538] After the app launches, the device starts the voice recognition function to record the business negotiations. The input is the user pressing the "Start Recording" button in the app. The output is the device's microphone starting to record the business negotiations.

[0539] Step 3:

[0540] The device analyzes the recorded voice data in real time and recognizes the user's emotions using an emotion analysis means. The input is the recorded voice data. The output is generated by the emotion analysis means, which recognizes the user's emotions (e.g., joy, anger, sadness). In concrete terms, if a user says, "This product is really great!", the emotion analysis means recognizes the emotion of "joy."

[0541] Step 4:

[0542] The terminal sends the analyzed emotion data and voice data to the generative AI means. The emotion data and voice data obtained by the emotion analysis means are input. As output, these data are sent to the generative AI means on the cloud. In concrete terms, after the emotion analysis means recognizes the emotion of "joy," the terminal sends that information along with the voice data to the generative AI means.

[0543] Step 5:

[0544] Based on the voice data and emotion data received by the server, the generative AI model summarizes the content of the business negotiation and suggests the next action. The input is the voice data and emotion data received by the server. As output, the generative AI model generates a summary of the content of the business negotiation and a suggestion for the next action. Specifically, in response to a statement such as "This product is truly amazing!", the server generates a summary and suggestion such as "Because the user is very satisfied with the product, we suggest providing additional product information on your next visit."

[0545] (Application example 1)

[0546] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0547] Conventional sales assistant systems offer basic functions such as recording, transcribing, summarizing, and organizing sales negotiations, but they have the problem of not being able to analyze customer emotions in real time and respond accordingly. This has led to a decline in the quality of customer service and a lack of improvement in customer satisfaction.

[0548] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, confirming homework, downloading materials, and registering, an emotion engine for analyzing user emotions, and a means for transmitting emotion data analyzed by the emotion engine to the generative AI means. This makes it possible to analyze customer emotions in real time and respond based on the results.

[0549] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[0550] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text format.

[0551] "Generative AI tools" are technologies that summarize and organize written data, and automate tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information.

[0552] An "emotion engine" is a technology that analyzes emotions from a user's voice and outputs the results as data.

[0553] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[0554] A "sales assistant system" is a system that supports sales activities by recording the contents of sales negotiations during sales visits and performing voice recognition, text conversion, summarization, organization, emotion analysis, etc.

[0555] As an embodiment of the present invention, a customer assistance system for a brick-and-mortar store will be described as an example. This system includes a speech recognition unit, a text generation unit, a generative AI unit, an emotion engine, and an emotion data transmission unit.

[0556] System configuration

[0557] 1. Voice recognition means:

[0558] Using an application installed on a smartphone or smart glasses.

[0559] Record conversations with customers and obtain audio data.

[0560] 2. Means of writing:

[0561] The voice data obtained by the voice recognition means is converted into text data.

[0562] Specifically, we use a Python speech recognition library.

[0563] 3. Generative AI means:

[0564] It summarizes and organizes written data, and automates tasks such as creating minutes, providing standard answers to questions, checking homework assignments, downloading materials, and registering.

[0565] Specifically, we use the transformers library to perform sentiment analysis.

[0566] 4. Emotion Engine:

[0567] Emotions are analyzed from the user's voice and the results are output as data.

[0568] The analyzed emotional data is sent to the generative AI means.

[0569] 5. Means of sending emotional data:

[0570] The emotion data analyzed by the emotion engine is sent to the generative AI means.

[0571] This allows the generative AI means to respond based on the customer's emotions.

[0572] Processing flow

[0573] The server first records the customer's voice using a voice recognition means. Next, the recorded voice data is converted into text data using a text conversion means. After that, the generative AI means summarizes and organizes the text data and automates the necessary information. Furthermore, the emotion engine analyzes the user's emotions and sends the emotion data to the generative AI means. This enables the generative AI means to respond based on the customer's emotions.

[0574] Specific examples

[0575] For example, if a customer says, "I'd like to know more about this product," the voice recognition means records this statement, and the text translation means converts it into text data. The emotion engine analyzes this text data and determines that the customer's emotion is "interest." Based on this information, the generative AI means provides the staff with appropriate responses and additional information.

[0576] Prompt Sentence Examples

[0577] Customers say: I want to know more about this product

[0578] Emotion: Interest

[0579] In this way, customer service can be more effectively provided in physical stores.

[0580] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0581] Step 1:

[0582] The server uses a voice recognition means to record the customer's voice. Specifically, it starts an application installed on a smartphone or smart glasses and records the conversation with the customer. The input is the customer's voice data, and the output is a recorded voice file.

[0583] Step 2:

[0584] The server converts the recorded voice data into text data using a text-to-speech conversion method. Specifically, it uses a Python speech recognition library to convert the voice data into text. The input is an audio file, and the output is text data.

[0585] Step 3:

[0586] The server uses generative AI tools to summarize and organize text data. Specifically, it uses the transformers library to analyze, summarize, and organize text data. The input is text data, and the output is summarized text data.

[0587] Step 4:

[0588] The server analyzes the user's emotions using an emotion engine. Specifically, it analyzes emotions from text data using the emotion analysis model in the transformers library. The input is text data, and the output is emotion data.

[0589] Step 5:

[0590] The server transmits the emotion data analyzed by the emotion engine to the generative AI means. The input is the emotion data, and the output is the emotion data transmitted to the generative AI means.

[0591] Step 6:

[0592] The server uses generative AI to respond based on the customer's emotions. Specifically, it generates appropriate responses and additional information based on emotion data and text data. The inputs are emotion data and text data, and the output is responses and additional information.

[0593] Step 7:

[0594] The terminal presents the response methods and additional information provided by the generative AI means to the staff. Specifically, it displays them on the display of a smartphone or smart glasses. The input is the response methods and additional information, and the output is what is presented to the staff.

[0595] Example 2

[0596] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0597] Conventional voice recognition systems simply convert voice data into text data, and do not summarize or organize the data to reflect the user's emotions, which can result in inappropriate minutes and answers to questions. Furthermore, the process of recording voice data, generating text data, summarizing, and organizing it is often done manually, resulting in inefficiencies. Furthermore, there is a need to process data quickly even during sales visits or while traveling, which is difficult to achieve with current systems.

[0598] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0599] In this invention, the server includes a speech recognition unit, a means for converting speech data obtained by the speech recognition unit into text, a generative AI unit for summarizing and organizing the textual data, an emotion analysis unit, and a means for summarizing and organizing the textual data in consideration of the emotion data obtained by the emotion analysis unit. This enables appropriate minutes and answers to questions that reflect the user's emotions. Furthermore, the process from recording speech data to generating, summarizing, and organizing text data is automated and can be performed efficiently. Furthermore, data can be processed quickly even when visiting or on the move.

[0600] "Speech recognition means" is a technology that analyzes voice data and converts it into corresponding text data.

[0601] The "text conversion means" is a technology that executes a process of converting the voice data obtained by the voice recognition means into text data.

[0602] "Generative AI means" refers to artificial intelligence technology for summarizing and organizing text data, and in particular has the ability to generate appropriate sentences by taking into account the user's emotions.

[0603] "Emotion analysis means" is a technology that analyzes a user's emotions from text data or voice data and acquires emotion data.

[0604] "Means for summarizing and organizing" refers to a technology that uses generative AI means to carry out the process of summarizing and organizing text based on text data and emotional data.

[0605] "Minutes taking" is the process of recording the contents of a meeting or discussion and creating a document for future reference.

[0606] "Canonical Answers to Questions" is the process of providing pre-canned answers to frequently asked questions.

[0607] "Homework review" is the process of reviewing action items and tasks decided upon in meetings and tracking their progress.

[0608] "Downloading materials" is the process of obtaining necessary materials and documents from the Internet or an internal network.

[0609] "Registration" is the process of adding new data or information to the system.

[0610] This invention is a system that combines speech recognition technology and generative AI technology, which converts a user's speech data into text data, and then summarizes and organizes it while taking into account emotional data, thereby generating appropriate minutes and answers to questions. Specific embodiments of this system are described below.

[0611] First, the user uses the device to record audio data. The device is equipped with a recording function, allowing high-quality recording of audio from meetings, interviews, etc. Once recording is complete, the device sends the recorded audio data to the server. An HTTP request is used for transmission, and the audio data is uploaded to the server.

[0612] The server sends the received voice data to a voice recognition service. Specifically, it uses voice recognition services such as Google's voice recognition API or IBM's Watson. These services analyze the voice data and generate corresponding text data. For example, if a user says, "I had a lot of fun today," the voice recognition service generates the text data, "I had a lot of fun today."

[0613] Next, the server uses an emotion analysis tool to obtain emotion data from the text data. The emotion analysis tool analyzes the text data and identifies the user's emotion. For example, emotion data of "joy" is obtained from text data such as "I had a lot of fun today."

[0614] The server uses generative AI means to summarize and organize the text data and emotion data. The generative AI means takes the user's emotions into consideration to generate more appropriate minutes and answers to questions. For example, based on the text data "Today was a lot of fun" and the emotion data "joy," it generates a summary that reads, "The user really enjoyed today's activities."

[0615] The generated text data is sent from the server to the terminal. The terminal displays the received text data to the user. The user can check the generated summary and minutes on the terminal screen.

[0616] For example, the following prompts can be fed to a generative AI model:

[0617] Prompt Sentence Examples

[0618] Audio data: "I had a lot of fun today."

[0619] Emotion data: "Joy"

[0620] Generated text: "The user enjoyed today's activity very much."

[0621] Using this prompt, the generative AI model can take into account the speech and emotion data to generate an appropriate summary and organized text.

[0622] This system allows users to efficiently convert voice data into text data and quickly generate appropriate summaries and minutes that reflect the emotions of the participants, making it extremely convenient for accurately recording the contents of meetings and interviews and for later reference.

[0623] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] The user records the audio data.

[0626] A user can record voice data using the recording function of a device. For example, they can record speech during a meeting or the contents of an interview. The input is the user's voice, and the output is the recorded voice data.

[0627] Step 2:

[0628] The device sends the recorded audio data to the server.

[0629] The device sends the recorded audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. The input is the recorded audio data, and the output is the audio data sent to the server.

[0630] Step 3:

[0631] The server uses a voice recognition service to convert the voice data into text data.

[0632] The server sends the received voice data to a voice recognition service. Specifically, it uses a general voice recognition API. The voice recognition service analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data. For example, voice data saying "I had a lot of fun today" is converted into text data saying "I had a lot of fun today."

[0633] Step 4:

[0634] The server acquires emotion data using an emotion analysis tool.

[0635] The server sends the text data to an emotion analysis tool and obtains emotion data. The emotion analysis tool analyzes the text data and identifies the user's emotion. The input is text data and the output is emotion data. For example, the emotion data obtained from the text data "I had a lot of fun today" is "joy."

[0636] Step 5:

[0637] The server uses generative AI tools to summarize and organize text data and emotion data.

[0638] The server uses generative AI means to summarize and organize the text data and emotional data. The generative AI means takes the user's emotions into consideration to generate more appropriate minutes and answers to questions. The input is text data and emotional data, and the output is summarized and organized text data. For example, based on the text data "Today was a lot of fun" and the emotional data "joy," it generates the summary "The user really enjoyed today's activities."

[0639] Step 6:

[0640] The server transmits the generated text data to the terminal.

[0641] The server sends the generated text data to the terminal. Specifically, it returns the generated text data to the terminal as an HTTP response. The input is the summarized and organized text data, and the output is the text data sent to the terminal.

[0642] Step 7:

[0643] The user checks the generated text data.

[0644] The user checks the text data generated by the terminal. For example, the user can check a summary or minutes displayed in a terminal application. The input is the text data sent to the terminal, and the output is the text data checked by the user.

[0645] (Application example 2)

[0646] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0647] Customer service in brick-and-mortar stores requires immediate understanding of customer needs and emotions and the ability to respond appropriately. However, conventional methods require store clerks to manually record conversations with customers and analyze them later, making it difficult to respond in real time. It is also difficult to accurately grasp customer emotions, which can lead to inappropriate responses. A system that can solve these problems and improve customer satisfaction is needed.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0649] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for recording conversations with customers and converting them into text and performing sentiment analysis in real time, and a means for instantly grasping customer needs and emotions and supporting appropriate responses. This makes it possible to streamline customer service operations in brick-and-mortar stores and grasp customer needs and emotions in real time.

[0650] "Speech recognition means" is a technology for converting voice data into text data.

[0651] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into text data.

[0652] "Generative AI methods" are artificial intelligence techniques for summarizing and organizing text data.

[0653] "Sentiment analysis" is a technology that analyzes emotions from text data and identifies those emotions.

[0654] A "customer service support system" is a system for supporting customer service operations in physical stores.

[0655] "Real-time text conversion" is the process of instantly converting audio data into text data.

[0656] "Means of understanding customer needs and emotions" refers to techniques for analyzing and understanding customer needs and emotions through conversations with customers.

[0657] "Means to support appropriate responses" are support technologies that enable optimal responses based on the customer's needs and emotions.

[0658] As an embodiment of the present invention, a specific method for constructing a customer service support system will be described below.

[0659] First, the server uses a speech recognition tool to record conversations with customers in physical stores and convert them into text data in real time. A common speech recognition API can be used as the speech recognition tool, such as Google's speech recognition API or IBM's Watson.

[0660] Next, the server uses a generative AI method to summarize and organize the text data. This generative AI method has the ability to analyze the content of the text data, extract important information, and summarize it. Specifically, it uses an AI model that uses natural language processing technology. For example, it can use the Transformers library.

[0661] Furthermore, the server analyzes customer sentiment from the text data using a sentiment analysis means. As the sentiment analysis means, a sentiment analysis model can be used. For example, a Sentiment Analysis Pipeline can be used.

[0662] This allows the server to instantly understand the customer's needs and feelings and provide appropriate support, such as providing information about products and services that the customer is interested in, or taking measures to resolve customer dissatisfaction.

[0663] For example, if a store clerk records a conversation with a customer and the application converts it into text and performs sentiment analysis in real time, if the customer says, "I'd like to know more about this product," the application can convert the remark into text and use sentiment analysis to determine that the customer is interested, prompting the store clerk to respond appropriately.

[0664] Examples of prompts for generative AI models include:

[0665] Record your conversations with customers and follow these steps:

[0666] 1. Convert the audio data into text data.

[0667] 2. Analyze the sentiment of text data.

[0668] 3. Summarize text data.

[0669] Output the results in the following format:

[0670] Transcript: [Text data]

[0671] Emotion: [Emotion data]

[0672] Summary: [Summary data]

[0673] In this way, customer service operations in physical stores can be made more efficient and customer satisfaction can be improved.

[0674] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0675] Step 1:

[0676] The server uses a voice recognition means to record conversations with customers in the physical store.

[0677] Input: Voice data of conversation with customer

[0678] Output: Recorded audio data file

[0679] Specific operation: The server acquires audio data through a microphone and saves it as an audio file.

[0680] Step 2:

[0681] The server converts the recorded voice data into text data using a voice recognition means.

[0682] Input: Recorded audio data file

[0683] Output: Text data

[0684] Specific operation: The server calls a speech recognition API and converts the speech data into text data. For example, it uses Google's speech recognition API.

[0685] Step 3:

[0686] The server uses generative AI tools to summarize and organize the text data.

[0687] Input: Text data

[0688] Output: Summarized text data

[0689] How it works: The server uses an AI model that uses natural language processing techniques to analyze the content of the text data, extract important information, and summarize it, for example, using the Transformers library.

[0690] Step 4:

[0691] The server uses a sentiment analysis means to analyze the sentiment of the customer from the text data.

[0692] Input: Text data

[0693] Output: Emotion data

[0694] Specific operation: The server uses a sentiment analysis model to analyze the sentiment of the text data, for example, using the Sentiment Analysis Pipeline.

[0695] Step 5:

[0696] The server uses the summarized text data and emotional data to understand the customer's needs and emotions and supports appropriate responses.

[0697] Input: Summarized text data, emotion data

[0698] Output: Response instruction data

[0699] Specific Actions: The server analyzes the summarized text data and emotion data and generates specific instructions for the store clerk based on the customer's needs and emotions.

[0700] Step 6:

[0701] The server transmits the generated response instruction data to the terminal and notifies the store clerk.

[0702] Input: Response instruction data

[0703] Output: Notification to store clerk

[0704] Specific operation: The server sends the response instruction data to the terminal and notifies the store clerk, for example, by displaying the notification on a smartphone or smart glasses.

[0705] In this way, it becomes possible to streamline customer service operations in physical stores and understand customer needs and emotions in real time.

[0706] Example 3

[0707] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0708] In conventional sales assistant systems, the transcription, summarization, and organization of voice data is often done manually, which is inefficient. Also, it is difficult to improve user satisfaction because it is not possible to automate tasks that take the user's emotions into account. Furthermore, there is an issue of a heavy workload because tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering are not automated.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0710] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, and a means for automating tasks taking into account user emotion data. This enables efficient textualization, summarization, and organization of speech data, realizing task automation that reflects the user's emotions. Furthermore, because tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering are automated, the workload is reduced.

[0711] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[0712] The "means for converting text" is a technique for recording voice data obtained by a voice recognition means as text data.

[0713] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries based on that information.

[0714] The "minutes creation method" is a technology that automatically creates a document that records the contents of a meeting based on a summary generated by a generative AI method.

[0715] "Standard answering means for questions" is a technology that provides standard answers to questions posed by users based on summary sentences generated by generative AI means.

[0716] The "homework confirmation method" is a technology that confirms unresolved issues and action items in meetings and work based on summaries generated by generative AI methods.

[0717] "Material downloading means" refers to technology that automatically downloads necessary materials.

[0718] The "registration means" is a technology that automatically registers the summary text and minutes generated by the generative AI means in the appropriate location.

[0719] "Means that take emotion data into consideration" refers to technology that analyzes emotions from user voice data and automates tasks by reflecting those emotions.

[0720] MODE FOR CARRYING OUT THE INVENTION

[0721] This invention is a system that includes a speech recognition means, a text generation means, a generative AI means, a minutes creation means, a standard answer means for questions, a homework confirmation means, a material download means, a registration means, and a means for considering emotional data. Specific embodiments of this system are described below.

[0722] System configuration

[0723] The server uses voice recognition software (for example, a voice recognition API) as a means of voice recognition. When a user uploads voice data from a terminal, the server receives this voice data and converts it into text data using the voice recognition software.

[0724] Next, the server uses a natural language processing model (e.g., BERT or Generative AI-3) as a generative AI tool to analyze the converted text data and extract important information. Based on the extracted information, the generative AI model generates a summary.

[0725] Based on the generated summary, the server automatically generates minutes using a minutes-creation tool. The server then formats the minutes according to the format in cooperation with the minutes-creation tool. Furthermore, the server uses a standard answering tool to provide standard answers to questions from users.

[0726] The server also uses a homework confirmation means to confirm unresolved issues and action items in meetings and business, a material download means to automatically download necessary materials, and a registration means to automatically register the generated summary and minutes in appropriate locations.

[0727] Furthermore, the server uses a means to consider emotional data to analyze the user's voice data and automate tasks based on that emotion. For example, if the user makes a sound that indicates anger, the server can use that information to add notes to meeting minutes or adjust responses to questions.

[0728] Specific examples

[0729] As a concrete example, consider a scenario in which a user uploads audio data from a meeting. After the meeting, the user uploads the recorded audio file from their device to the server. The server converts the audio data into text data using a speech recognition API. The converted text data is analyzed using the BERT model to extract important information. Based on the extracted information, the generative AI-3 model generates a summary. The generated summary is formatted as minutes using a minutes-taking tool and registered in the CRM system. Finally, the server extracts the user's emotional data from the audio data, and if any voice indicating anger is included, adds a note to the minutes based on that information.

[0730] Example prompt sentence:

[0731] "Please prepare minutes of yesterday's meeting. The audio can be downloaded from the link below. Please summarize key decisions and action items."

[0732] In this way, the server automatically creates minutes based on the user's instructions and organizes and provides the necessary information. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[0733] Step 1:

[0734] The user uploads the audio data.

[0735] After the meeting, the user uploads the recorded audio file from their device to the server. Specifically, the user drags and drops the audio file into a designated upload form. The input is the audio file, and the output is the audio data stored on the server.

[0736] Step 2:

[0737] The server converts the voice data into text data.

[0738] The server receives the uploaded voice data and converts it into text data using voice recognition software (e.g., a voice recognition API). The input is voice data, and the output is text data. Specifically, the voice recognition API analyzes the voice data and generates corresponding text.

[0739] Step 3:

[0740] The server analyzes the text data and extracts important information.

[0741] The server inputs the converted text data into a natural language processing model (e.g., BERT) to extract important information. The input is text data, and the output is data containing important information. Specifically, the BERT model analyzes the text data and identifies meeting agendas, decisions, action items, etc.

[0742] Step 4:

[0743] The server generates a summary based on the extracted information.

[0744] The server generates a summary using a generative AI model (e.g., Generative AI-3) based on the extracted important information. The input is data containing important information, and the output is a summary. Specifically, the Generative AI-3 model summarizes the important information and creates a concise, easy-to-understand summary.

[0745] Step 5:

[0746] The server creates minutes based on the summary.

[0747] The server automatically generates minutes based on the generated summary. The input is the summary, and the output is the minutes. Specifically, it works in conjunction with a minutes-creation tool to format the minutes according to the format.

[0748] Step 6:

[0749] The server registers the minutes in the CRM system.

[0750] The server registers the generated meeting minutes in the CRM system. The input is the meeting minutes, and the output is the data registered in the CRM system. Specifically, the minutes are uploaded to the CRM system via API and linked to related project and customer information.

[0751] Step 7:

[0752] The server automates tasks by taking into account the user's emotional data.

[0753] The server extracts the user's emotional data from the voice data and automates tasks by reflecting that emotion. The input is the voice data, and the output is the task results that take the emotional data into account. Specifically, if the user makes a sound that indicates anger, the server can use that information to add notes to the minutes or adjust responses to questions.

[0754] (Application example 3)

[0755] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0756] Conventional sales assistant systems provide functions such as voice recognition, text generation, summarization, and organization, but they do not support the automation of work instructions and reports within factories. Therefore, in order to improve work efficiency within factories, functions such as real-time summarization of work content, reporting to managers, standard answers to questions from workers, and automatic downloading and registration of necessary materials are required. This makes it a challenge to improve work efficiency within factories.

[0757] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0758] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, and a generative AI means for summarizing and organizing the textual data, thereby enabling the automation of tasks such as creating minutes, providing standard answers to questions, checking homework, downloading and registering materials, automating work instructions and reports within the factory, summarizing work content in real time and reporting it to a manager, providing standard answers to questions from workers, and automatically downloading and registering necessary materials.

[0759] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[0760] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[0761] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries.

[0762] Minute-taking is the process of recording the contents of a meeting or discussion so that it can be referenced later.

[0763] "Canonical Answers to Questions" is the process of providing pre-made answers to frequently asked questions.

[0764] "Confirming homework" is the process of confirming assignments and tasks given in meetings and discussions and managing progress.

[0765] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[0766] "Registration" is the process of storing and managing acquired materials and data in an appropriate location.

[0767] "Means for automating work instructions and reports within a factory" refers to technology that gives instructions on work within a factory and automatically reports the results.

[0768] "Means for summarizing work content in real time and reporting to managers" refers to technology that summarizes work content performed within a factory in real time and immediately conveys that information to managers.

[0769] The "means for providing standard answers to questions from workers" is a technology that automatically provides pre-prepared answers to questions from workers.

[0770] "Means for automatically downloading and registering necessary materials" refers to technology that automatically obtains necessary materials from the Internet or an internal network and stores them in an appropriate location.

[0771] A system for carrying out this invention includes a speech recognition means, a text generation means, a generative AI means, a means for creating minutes, providing standard answers to questions, checking homework items, downloading and registering materials, a means for automating work instructions and reports within a factory, a means for summarizing work content in real time and reporting it to a manager, a means for providing standard answers to questions from workers, and a means for automatically downloading and registering necessary materials.

[0772] Hardware and Software Configuration

[0773] Hardware: Smartphones and tablets used by robots and workers in factories.

[0774] Software: Python, Transformers library, OpenAI API.

[0775] Data processing and calculation

[0776] 1. Speech recognition: Recognizes speech from factory workers and robots in real time and converts it into text data. A speech recognition engine is used for this purpose.

[0777] 2. Text conversion: The voice data obtained by the voice recognition means is converted into text format, and the work content is saved as text data.

[0778] 3. Generative AI methods: Using natural language processing models such as BERT and Generative AI-3, we analyze text data, extract important information, and generate summaries.

[0779] 4. Minutes creation: Minutes are automatically generated based on the summary text generated by generative AI tools.

[0780] 5. Standard answers to questions: Provide pre-prepared standard answers to questions posed by workers. This is done using OpenAI's generative AI.

[0781] 6. Check homework: Check the assignments and tasks given in meetings and discussions and manage progress.

[0782] 7. Material download and registration: Automatically download necessary materials from the Internet or internal network and store them in the appropriate location.

[0783] 8. Automating work instructions and reports within the factory: Give instructions on work within the factory and automatically report the results.

[0784] 9. Real-time summary and reporting of work: Summarize the work carried out in the factory in real time and immediately convey this information to management.

[0785] 10. Providing standard answers to worker questions: Automatically provide pre-prepared answers to worker questions.

[0786] 11. Automatic download and registration of required materials: Automatically retrieve required materials from the Internet or internal network and save them in the appropriate location.

[0787] Specific examples

[0788] Summary of work: Summarize the text as "Today's work will be maintenance on Line 1 and quality checks on Line 2" and report it to the manager.

[0789] Answering the Question: Provide a standard answer to the question, "What is the maintenance procedure for Line 1?"

[0790] Download and register the materials: Download the "Maintenance Manual" and register it in the appropriate location.

[0791] Prompt Sentence Examples

[0792] Summary generation: Summary: Today's work involves maintenance of Line 1 and quality check of Line 2.

[0793] Standard Answer: Question: What is the maintenance procedure for Line 1?

[0794] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0795] Step 1:

[0796] The voice recognition means is activated to recognize voices emitted by workers and robots in the factory in real time.

[0797] Input: Voice data from workers and robots.

[0798] Data processing: Converting voice data into text data using a speech recognition engine.

[0799] Output: Text data.

[0800] Step 2:

[0801] The text data obtained by the speech recognition means is converted into a text format using a sentence conversion means.

[0802] Input: Text data from a speech recognizer.

[0803] Data processing: Format text data and save it as text.

[0804] Output: Prose-encoded text data.

[0805] Step 3:

[0806] Using generative AI methods, we analyze written text data, extract important information, and generate a summary.

[0807] Input: Documented text data.

[0808] Data processing: Using natural language processing models such as BERT and generative AI-3, we analyze text data and generate summaries.

[0809] Output: Summary statement.

[0810] Step 4:

[0811] Minutes are automatically generated based on the summary text generated by generative AI tools.

[0812] Input: Abstract text.

[0813] Data processing: Based on the summary text, the data is formatted into minutes.

[0814] Output: Meeting minutes.

[0815] Step 5:

[0816] Provide pre-prepared standard answers to questions from workers.

[0817] Input: Question from worker.

[0818] Data processing: Use OpenAI's generative AI to generate standard answers to questions.

[0819] Output: Standard answer.

[0820] Step 6:

[0821] Check the issues and tasks raised in meetings and discussions and manage progress.

[0822] Input: Details of the meeting or discussion.

[0823] Data processing: List issues and tasks and manage progress.

[0824] Output: Issue or task progress.

[0825] Step 7:

[0826] Automatically download necessary materials from the Internet or internal network and save them in the appropriate location.

[0827] Input: Name or URL of the resource.

[0828] Data processing: Download the materials and save them in the specified location.

[0829] Output: The downloaded material.

[0830] Step 8:

[0831] It gives instructions on work to be done within the factory and automatically reports the results.

[0832] Input: Work instruction content.

[0833] Data processing: Create work instructions in text format and communicate them to workers and robots.

[0834] Output: Work order report.

[0835] Step 9:

[0836] It summarizes the work being done in the factory in real time and immediately conveys that information to managers.

[0837] Input: Text data of the work content.

[0838] Data processing: Summarizing the work and formatting it for reporting to management.

[0839] Output: Report to administrator.

[0840] Step 10:

[0841] The system automatically provides pre-prepared answers to questions from workers.

[0842] Input: Question from worker.

[0843] Data processing: Generate and provide standard answers to questions.

[0844] Output: Standard answer.

[0845] Step 11:

[0846] Automatically retrieve necessary materials from the Internet or internal network and store them in the appropriate location.

[0847] Input: Name or URL of the resource.

[0848] Data processing: Download the materials and save them in the specified location.

[0849] Output: The downloaded material.

[0850] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0851] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0852] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0853] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0854] [Second embodiment]

[0855] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0856] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0857] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0858] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0859] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0860] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0861] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0862] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0863] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0864] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0865] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0866] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0867] "Example 1"

[0868] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[0869] "Example 2"

[0870] The system of the present invention also includes a means for converting recorded voice data into text. This means converts voice data into text using voice recognition technology. Specifically, it can use voice recognition services such as Google's voice recognition API or IBM's Watson.

[0871] "Example 3"

[0872] Furthermore, the system of the present invention is equipped with a generative AI means for summarizing and organizing textual data. This generative AI means uses natural language processing technology to analyze text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and generative AI-3 can be used.

[0873] "Example 4"

[0874] The system of the present invention also includes a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering them. This means automatically generates minutes based on the summary generated by the generative AI means, provides standard answers to questions, checks homework, downloads necessary materials, and registers them in appropriate locations. Specifically, these tasks can be automated through collaboration with a minutes-creation tool or CRM system.

[0875] The processing flow of each embodiment will be described below.

[0876] "Example 1"

[0877] Step 1: During a sales visit, the sales representative launches an application installed on a mobile information device such as a smartphone or tablet.

[0878] Step 2: The application records the conversation and converts the audio data into text using a speech recognition tool such as Google's speech recognition API or IBM's Watson.

[0879] "Example 2"

[0880] Step 1: The recorded voice data is converted into text using a voice recognition device, and the voice data is converted into text data.

[0881] Step 2: The textual data is summarized and organized by a generative AI tool. This generative AI tool uses natural language processing techniques to analyze the text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and Generative AI-3 are used.

[0882] "Example 3"

[0883] Step 1: Minutes are automatically generated based on the summary generated by the generative AI method. Step 2: Similarly, standard answers to questions are provided and homework items are confirmed.

[0884] Step 3: The necessary documents are downloaded and registered in the appropriate location. This automation is achieved through integration with meeting minutes creation tools and CRM systems.

[0885] Example 1

[0886] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0887] In conventional sales activities, accurately recording the details of sales negotiations and then summarizing and analyzing them later was a very time-consuming task. In addition, important points were often overlooked during sales negotiations, and when reviewing the records later, the information was often incomplete. This led to problems such as reduced sales efficiency and a decline in the quality of customer service.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0889] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for sending speech data to the speech recognition engine in real time, a means for inputting prompts to the generative AI model, and a means for displaying responses from the generative AI model. This makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[0890] "Speech recognition means" is a technology that converts voice into digital data, analyzes the content, and outputs it as text data.

[0891] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text data.

[0892] "Generative AI methods" are artificial intelligence technologies that analyze text data and summarize and organize it.

[0893] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[0894] "Standard answers to questions" is a technology that automatically provides pre-defined answers to frequently asked questions.

[0895] "Confirming homework" is the process of confirming the tasks that came up in business negotiations or meetings and the tasks that need to be done by the next time.

[0896] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[0897] "Registration" is the process of entering and storing data or information in a system.

[0898] "Means for transmitting voice data to a voice recognition engine in real time" refers to a technology that immediately transmits recorded voice data to a voice recognition engine and analyzes it in real time.

[0899] A "means for inputting prompt sentences to a generative AI model" is an interface that allows a user to input instructions or questions to a generative AI model.

[0900] "Means for displaying responses from the generative AI model" refers to technology that displays answers or results returned from the generative AI model to the user.

[0901] MODE FOR CARRYING OUT THE INVENTION

[0902] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[0903] Hardware and software used

[0904] Hardware: Smartphones, tablets, cloud servers (e.g. AWS EC2)

[0905] Software: Google Cloud Speech-to-Text API, MySQL database, OpenAI GPT-3

[0906] Program processing explanation

[0907] 1. Launch the app

[0908] Users launch the app on their smartphones or tablets during sales calls.

[0909] Example: A user taps the "Sales Assistant" app from the home screen of their smartphone to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[0910] 2. Start voice recognition

[0911] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[0912] Example: When a user taps the "Start Recording" button in the app, the device's microphone activates and begins recording the business conversation. During recording, the recording time is displayed on the app screen.

[0913] 3. Audio Data Processing

[0914] The device sends the recorded voice data to the voice recognition engine in real time.

[0915] Software used: Google Cloud Speech-to-Text API

[0916] Example: The device sends recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is temporarily stored on the device.

[0917] 4. Saving text data

[0918] The server stores the text data received from the speech recognition engine in a database.

[0919] Hardware used: Cloud server (e.g. AWS EC2)

[0920] Software used: MySQL database

[0921] Example: The server receives text data sent from a device and saves it in a MySQL database on AWS EC2. When saving is complete, the server sends a notification to the device that saving is complete.

[0922] 5. Input to the generative AI model

[0923] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[0924] Software used: OpenAI GPT-3

[0925] Example: A user enters a prompt such as "Please write a summary of this deal" into an input field in an app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[0926] 6. Displaying the results

[0927] The device receives the response from the generative AI model and displays it to the user.

[0928] Example: The device receives a summary and analysis of a business deal from the generative AI model and displays it within the app. The user can check the summary and analysis on the app screen.

[0929] Examples of prompt statements

[0930] "Please write a summary for this opportunity"

[0931] "Analyze the content of the business negotiation and tell us the key points to propose next time."

[0932] The above is an embodiment of the sales assistant system of the present invention. This system makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[0933] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0934] Step 1:

[0935] Launching the app

[0936] Users launch the app on their smartphones or tablets during sales calls.

[0937] Specific behavior: The user taps the "Sales Assistant" app on the home screen of their smartphone to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[0938] Input: User taps

[0939] Output: App home screen display

[0940] Step 2:

[0941] Start voice recognition

[0942] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[0943] Specific operation: When the user taps the "Start Recording" button in the app, the device's microphone will be activated and the business conversation will begin to be recorded. During recording, the recording time will be displayed on the app screen.

[0944] Input: User taps

[0945] Output: Start recording, display recording time

[0946] Step 3:

[0947] Audio data processing

[0948] The device sends the recorded voice data to the voice recognition engine in real time.

[0949] Software used: Google Cloud Speech-to-Text API

[0950] How it works: The device sends the recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data, which is then temporarily stored on the device.

[0951] Input: Recorded audio data

[0952] Output: Text data

[0953] Step 4:

[0954] Saving text data

[0955] The server stores the text data received from the speech recognition engine in a database.

[0956] Hardware used: Cloud server (e.g. AWS EC2)

[0957] Software used: MySQL database

[0958] Specific operation: The server receives the text data sent from the device and saves it in a MySQL database on AWS EC2. Once saving is complete, the server sends a notification to the device.

[0959] Input: Text data

[0960] Output: Saving to database, notification of completion of saving

[0961] Step 5:

[0962] Input to generative AI models

[0963] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[0964] Software used: OpenAI GPT-3

[0965] Specific operation: The user enters the prompt "Please write a summary of this deal" into the input field in the app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[0966] Input: prompt, text data

[0967] Output: Send to generative AI model

[0968] Step 6:

[0969] Displaying the results

[0970] The device receives the response from the generative AI model and displays it to the user.

[0971] Specific operation: The device receives the summary and analysis results of the business negotiation received from the generative AI model and displays them within the app. The user can check the summary and analysis results on the app screen.

[0972] Input: Response from a generative AI model

[0973] Output: Display of summary and analysis results

[0974] (Application example 1)

[0975] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0976] Conventional sales assistant systems were capable of recording and transcribing sales negotiations, but lacked the functionality to analyze the content of sales negotiations and automatically generate next actions and proposals. As a result, sales staff had to analyze the content themselves after the negotiation and consider the next steps, which was time-consuming and labor-intensive. In addition, summarizing and organizing the content of sales negotiations had to be done manually, which was inefficient. A system that could solve these problems and make sales activities more efficient was needed.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0978] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, and a means for analyzing the content of business negotiations using a generative AI model and automatically generating next actions and proposals. This makes it possible to consistently automate the process from recording the content of business negotiations to analyzing it and proposing the next action.

[0979] "Speech recognition means" is a technology that acquires voice data and converts it into text data.

[0980] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[0981] "Generative AI methods" are technologies that summarize and organize text data, and then use generative AI models to analyze and make suggestions.

[0982] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[0983] "Standard Answers to Questions" is a feature that automatically provides pre-set answers to frequently asked questions.

[0984] "Homework review" is the process of tracking and reviewing issues and next steps that arise from a business meeting or negotiation.

[0985] "Download materials" is a function that allows you to obtain necessary materials from the Internet or an internal network.

[0986] "Registration" is the process of storing and managing acquired data and information in a system.

[0987] A "generative AI model" is an artificial intelligence technology that generates new information and suggestions based on text data.

[0988] "Analyzing the content of business negotiations" is the process of analyzing the content discussed during business negotiations and extracting important points and next actions.

[0989] "Means for automatically generating next actions and proposals" refers to technology that automatically generates next actions and proposals based on the content of business negotiations.

[0990] The following system configuration will be described as an embodiment of the present invention.

[0991] System Configuration

[0992] This system includes speech recognition means, text conversion means, generative AI means, minutes creation, standard answers to questions, homework confirmation, document download and registration, analysis of business negotiation content using generative AI models, and means for automatically generating next actions and proposals. These means are realized as applications installed on mobile information terminals such as smartphones and tablets.

[0993] Hardware and software used

[0994] Hardware: Smartphone (iOS or Android), tablet

[0995] Software: Python, speech recognition library, transformers library

[0996] Processing flow

[0997] 1. Voice recording: The user starts the smartphone app and records the details of the business negotiation. The details of the business negotiation are captured as voice data using voice recognition.

[0998] 2. Speech recognition: The server uses a speech recognition library to convert the recorded voice data into text data.

[0999] 3. Texting: The audio data is converted into text format through textual processing.

[1000] 4. Data Summarization and Organizing: Summarize and organize text data using generative AI methods.

[1001] 5. Analyze the content of sales negotiations: Using generative AI models, analyze the content of sales negotiations and automatically generate next actions and proposals.

[1002] 6. Automation functions: Automate functions such as creating minutes, standard answers to questions, checking homework, downloading materials, and registering.

[1003] Specific examples

[1004] For example, if a customer says, "I'd like to discuss the introduction of a new product" during a sales negotiation, the user can launch an app on their smartphone and record the content of the negotiation. The recorded voice data is converted into text data using a speech recognition tool. The text data is then summarized and organized using a generative AI tool. The content of the sales negotiation is analyzed using a generative AI model, and a proposal such as "As a next step regarding the introduction of the new product, I suggest scheduling a demonstration" is automatically generated.

[1005] Prompt Sentence Examples

[1006] Negotiation details: We would like to discuss the introduction of a new product.

[1007] Next actions and suggestions:

[1008] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[1009] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1010] Step 1:

[1011] The user launches the smartphone app and records the contents of the business meeting. The input is the audio data of the business meeting, and the output is the recorded audio file. Specifically, the user presses the record button on the app and then presses the stop button when the business meeting is over.

[1012] Step 2:

[1013] The server converts the recorded audio file into text data using a speech recognition means. The input is the audio file and the output is text data. Specifically, the server analyzes the audio data using a speech recognition library and generates corresponding text.

[1014] Step 3:

[1015] The server uses generative AI tools to summarize and organize text data. The input is text data, and the output is summarized text data. Specifically, the server uses a generative AI model to extract important parts of the text and create a summary.

[1016] Step 4:

[1017] The server uses a generative AI model to analyze the content of the business negotiation and automatically generate the next action and proposal. The input is summarized text data, and the output is text data containing the next action and proposal. Specifically, the server inputs a prompt sentence into the generative AI model and obtains the generated text.

[1018] Step 5:

[1019] The server automates functions such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering. The input is text data of the business negotiations, and the output is automatically generated minutes, proposals, and confirmation items. Specifically, the server executes the processes corresponding to each function and generates and saves the necessary information.

[1020] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[1021] Example 2

[1022] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1023] In conventional sales assistant systems, the process of recording voice data, converting it to text, summarizing, and organizing it was often done manually, resulting in inefficiency. Furthermore, it was difficult to use highly accurate voice recognition technology to convert voice data into text, resulting in low quality text data. Furthermore, automation of tasks such as creating meeting minutes and standard responses to questions based on the generated text data was insufficient, placing a heavy burden on users.

[1024] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1025] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for transmitting recorded speech data from a terminal to the server, a means for the server to call a speech recognition service and convert the speech data into text data, a means for transmitting the converted text data to the terminal and displaying it, and a means for automating the creation of minutes, standard answers to questions, confirmation of homework items, downloading materials, registration, etc. This makes it possible to consistently and efficiently perform processes from recording speech data to converting it into text, summarizing, organizing, and even automating various tasks.

[1026] "Speech recognition means" refers to technology that analyzes voice data and converts it into corresponding text data.

[1027] The "means for converting text" refers to a technology for recording voice data obtained by a voice recognition means as text data.

[1028] "Generative AI methods" refers to artificial intelligence techniques for summarizing and organizing written data.

[1029] "Means for transmitting recorded voice data from a terminal to a server" refers to a technology for uploading voice data recorded on a terminal to a server via a network.

[1030] "Means for the server to call a voice recognition service and convert voice data into text data" refers to a technology in which the server uses an external voice recognition service to convert voice data into text data.

[1031] "Means for transmitting the converted text data to the terminal and displaying it" refers to a technology for transmitting the text data converted by the server to the terminal and displaying the text data on the terminal.

[1032] "Means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering" refers to technology that uses generated text data to automatically perform tasks such as creating minutes, answering questions, checking homework, downloading materials, and registering materials.

[1033] MODE FOR CARRYING OUT THE INVENTION

[1034] This invention is a system that efficiently converts voice data into text data and automates various tasks based on the text data. This system includes a voice recognition means, a text generation means, a generative AI means, a means for transmitting recorded voice data from a terminal to a server, a means for the server to call a voice recognition service and convert the voice data into text data, a means for transmitting the converted text data to a terminal and displaying it, and a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering.

[1035] Hardware and software used

[1036] Hardware: smartphones, PCs, servers

[1037] Software: Speech recognition APIs (e.g., general speech recognition services), generative AI models

[1038] Specific operation of the system

[1039] 1. A user records audio data using a recording app on a smartphone or computer. For example, to record the contents of a meeting, the user presses the record button on their smartphone to record the audio of the meeting.

[1040] 2. The device sends the recorded audio data to the server. Specifically, the recording app uploads the audio data to the server as an HTTP request.

[1041] 3. The server temporarily stores the received voice data and prepares it for the next process.

[1042] 4. The server calls the speech recognition service and converts the voice data into text data. Specifically, the server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text.

[1043] 5. The voice recognition service analyzes the received voice data and converts it into text data.

[1044] 6. The server receives the converted text data and sends it to the user's device. Specifically, the server returns the text data to the user's device as an HTTP response.

[1045] 7. The device displays the received text data to the user. For example, a smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[1046] Specific examples

[1047] For example, when a user records the contents of a meeting and converts it into text, the specific operation is as follows.

[1048] 1. The user launches the recording app on their smartphone and presses the record button to record the audio of the meeting.

[1049] 2. The device sends the recorded audio data to the server. For example, a recording app uploads a file called "MeetingRecording.wav" to the server as an HTTP request.

[1050] 3. The server receives the "Meeting Recording.wav" and temporarily stores it.

[1051] 4. The server calls the speech recognition service and sends "Meeting Recording.wav" to the API.

[1052] 5. The voice recognition service analyzes the voice data and generates text data such as "The contents of the meeting have been transcribed into text."

[1053] 6. The server receives the generated text data and sends it to the user's smartphone.

[1054] 7. The device displays the received text data on the app screen, allowing the user to check the contents of the meeting as text.

[1055] Prompt Sentence Examples

[1056] Examples of prompts to be input to a generative AI model include:

[1057] "Please convert the recorded audio data of the meeting into text data. The audio data can be obtained from the following URL. Please use a general speech recognition service for the conversion."

[1058] In this way, a system can be constructed that can efficiently convert recorded voice data into text data.

[1059] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1060] Step 1:

[1061] Users record audio data using a recording app on their smartphone or computer.

[1062] Input: Audio from meetings, business meetings, etc.

[1063] Output: Recorded audio data file (e.g., MeetingRecording.wav)

[1064] Specific operation: The user launches the recording app, presses the record button to record the audio of the meeting, and when the recording is finished, presses the save button to save the audio data.

[1065] Step 2:

[1066] The terminal transmits the audio data to the server after recording is completed.

[1067] Input: Recorded audio data file (e.g., MeetingRecording.wav)

[1068] Output: Audio data uploaded to the server

[1069] Specific operation: The recording app uploads the audio data to the server as an HTTP request. The user presses the send button to send the audio data to the server.

[1070] Step 3:

[1071] The server temporarily stores the received voice data in preparation for the next process.

[1072] Input: Uploaded audio data

[1073] Output: Audio data stored on the server

[1074] Specific operation: The server temporarily stores the received voice data in storage in preparation for calling the voice recognition service.

[1075] Step 4:

[1076] The server invokes a speech recognition service to convert the speech data into text data.

[1077] Input: Audio data stored on the server

[1078] Output: Audio data sent to the speech recognition service

[1079] Specific operation: The server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text. For example, it creates an API request, attaches the voice data, and sends it.

[1080] Step 5:

[1081] The voice recognition service analyzes the received voice data and converts it into text data.

[1082] Input: Audio data sent to the speech recognition service

[1083] Output: Converted text data

[1084] What it does: A speech recognition service analyzes audio data and generates corresponding text data, for example, by converting speech to text using a speech recognition algorithm.

[1085] Step 6:

[1086] The server receives the converted text data and sends it to the user's terminal.

[1087] Input: Converted text data

[1088] Output: Text data sent to the user's device

[1089] Specific operation: The server receives the text data returned from the speech recognition service and returns it to the user's device as an HTTP response.

[1090] Step 7:

[1091] The terminal displays the received text data to the user.

[1092] Input: Text data sent from the server

[1093] Output: Text data displayed on the terminal

[1094] Specific operation: The smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[1095] (Application example 2)

[1096] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1097] While systems using conventional speech recognition technology were able to convert recorded audio data into text, they lacked the ability to generate and display subtitles for video content in real time. This made it difficult to immediately understand the audio content when watching video content, limiting the viewing experience, especially when watching content in a different language.

[1098] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for generating subtitles for video content in real time, and a display means for displaying subtitles. This makes it possible to recognize the voice of video content in real time and instantly display it as subtitles.

[1099] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1100] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text format.

[1101] "Generative AI methods" are artificial intelligence techniques for summarizing and organizing text data.

[1102] "Means for generating subtitles for video content in real time" refers to technology that analyzes the audio data of video in real time and displays it as subtitles.

[1103] The "display means for displaying subtitles" is a technique for displaying generated subtitles superimposed on video frames.

[1104] The following system configuration will be described as an embodiment of the present invention.

[1105] System Configuration

[1106] This system includes a speech recognition means, a text generation means, a generative AI means, a means for generating subtitles in real time for video content, and a display means for displaying the subtitles.

[1107] Voice recognition means

[1108] The server uses a technology to analyze voice data and convert it into text data as a means of speech recognition, using a common speech recognition API (such as Google's speech recognition API or IBM's Watson).

[1109] Means of writing

[1110] The speech data obtained by the speech recognition means is converted into text format, and this converted text data is used for subsequent processing.

[1111] Generative AI means

[1112] Generative AI methods use artificial intelligence techniques to summarize and organize text data, including natural language processing (NLP) models. For example, generative AI models such as GPT-3 can be used.

[1113] A means of generating subtitles for video content in real time

[1114] The server uses technology to analyze the audio data of the video in real time and display it as subtitles. It generates subtitles in real time based on the text data obtained by the speech recognition means.

[1115] Display means for displaying subtitles

[1116] We use a technique to overlay the generated subtitles onto the video frames. Specifically, we use an image processing library such as OpenCV to add subtitles to the video frames.

[1117] Specific examples

[1118] For example, consider an application that generates English subtitles in real time while a user is watching a Japanese movie. This application is installed on a smartphone or head-mounted display, and while the user is watching the video, it converts the recorded audio data into text data in real time and displays it as subtitles.

[1119] Prompt Sentence Examples

[1120] "Create an app that generates English subtitles in real time while watching a Japanese movie. Use Google's Speech Recognition API for speech recognition and Google Translate API for translation. The subtitles should be displayed in the video frame."

[1121] In this way, the mode for carrying out the invention can be concretely shown.

[1122] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1123] Step 1:

[1124] The server acquires the audio data of the video content. Specifically, it extracts the audio track from the video file and saves it as audio data. The input is the video file, and the output is the audio data.

[1125] Step 2:

[1126] The server converts the voice data into text data using a voice recognition means. Specifically, it calls a voice recognition API, analyzes the voice data, and generates text data. The input is voice data, and the output is text data.

[1127] Step 3:

[1128] The server uses generative AI tools to summarize and organize text data. Specifically, it uses a natural language processing model to summarize the text data and extract the necessary information. The input is text data, and the output is summarized text data.

[1129] Step 4:

[1130] The server generates subtitles in real time based on the summarized text data. Specifically, it converts the generated text data into a subtitle format and adds a timestamp. The input is the summarized text data, and the output is the subtitle data.

[1131] Step 5:

[1132] The server overlays subtitle data onto video frames and displays them in real time using an image processing library. The input is subtitle data and video frames, and the output is video frames with added subtitles.

[1133] Step 6:

[1134] The user watches video frames with added subtitles. Specifically, they use a smartphone or a head-mounted display to watch videos with subtitles displayed in real time. The input is the video frames with added subtitles, and the output is the viewing experience.

[1135] Example 3

[1136] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] In conventional sales assistant systems, the transcription, summarization, and organization of voice data was often done manually, which was time-consuming and labor-intensive. Furthermore, tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering were not automated, resulting in inefficiency and a high likelihood of human error. Furthermore, there was a lack of a way to extract important information and execute automated tasks based on that information, which led to a decline in the accuracy and efficiency of the work.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1139] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summaries. This enables the automatic textualization, summarization, and organization of speech data, and automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, improving the efficiency and accuracy of tasks.

[1140] "Speech recognition means" is a technology for analyzing voice data and converting it into text data.

[1141] The "text conversion means" is a technology that executes a process of converting the voice data obtained by the voice recognition means into text data.

[1142] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries based on that information.

[1143] "Minutes writing" is the process of recording the contents of a meeting or conference and creating a document that summarizes it.

[1144] "Standard answers to questions" is the process of providing predefined answers to questions posed by users.

[1145] "Homework review" is the process of reviewing action items and tasks decided upon in meetings and tracking their progress.

[1146] "Downloading materials" is the process of automatically obtaining required materials from the Internet or internal systems.

[1147] "Registration" is the process of storing and managing generated data and documents in an appropriate location.

[1148] "Means for analyzing text data and extracting important information" refers to technology that analyzes text data and extracts important keywords and phrases from it.

[1149] "Means for performing automated tasks based on generated summaries" refers to technology that automatically performs tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information based on generated summaries.

[1150] MODE FOR CARRYING OUT THE INVENTION

[1151] This invention is a system including a speech recognition means, a text generation means, a generative AI means, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summary.

[1152] 1. Generating the system program

[1153] The server uses a speech recognition means to acquire voice data and generates a program that converts it into text data. Next, it uses a generative AI means to analyze the text data, extract important information, and generate a program that generates a summary based on that information. Furthermore, it generates a program that automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering based on the generated summary.

[1154] 2. Explanation of program processing

[1155] The server uses voice recognition software to convert the voice data provided by the user into text data. Specifically, voice recognition software such as Google Speech-to-Text can be used. The converted text data is sent to the server.

[1156] Next, the server uses natural language processing models such as BERT and Generative AI-3 as generative AI tools to analyze the text data. As a result of the analysis, important information is extracted and a summary is generated. This summary is then stored on the server.

[1157] Furthermore, the server performs the following automated tasks based on the generated summary:

[1158] Automatic generation of meeting minutes

[1159] Providing standard answers to questions

[1160] Checking homework

[1161] Download materials

[1162] Data registration

[1163] These tasks are automated through integration with meeting minutes-taking tools and CRM systems.

[1164] 3. Examples of concrete examples and prompts

[1165] Examples:

[1166] When a user creates minutes of a meeting, the audio data of the meeting is converted into text and the text data is sent to the server. The server then analyzes the text data using BERT to extract important topics and decisions. The server then generates a summary based on the extracted information and automatically generates the minutes.

[1167] Example prompt sentence:

[1168] "Please summarize the text data from the following meeting and prepare minutes:

[1169] 1. Meeting start and end times

[1170] 2. Participant's name

[1171] 3. Agenda and details

[1172] 4. Decisions and action items

[1173] By inputting this prompt sentence into the generative AI model, the server automatically generates a summary sentence and creates the minutes. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1174] Program processing flow

[1175] Step 1: Input audio data

[1176] Users record audio data of meetings and business negotiations, and the recorded audio data is saved on the device.

[1177] Specific behavior:

[1178] The user starts the recording application and records the audio of a meeting or business negotiation.

[1179] Once the recording is complete, the audio data is saved on the device.

[1180] Input: Audio data from meetings and business negotiations

[1181] Output: Audio data stored on the device

[1182] Step 2: Converting audio data to text

[1183] The device uses voice recognition software to convert the recorded voice data into text data, which is then sent to the server.

[1184] Specific behavior:

[1185] The device launches voice recognition software (e.g., Google Speech-to-Text).

[1186] The audio data is input into the software and converted into text data.

[1187] The converted text data is sent to the server.

[1188] Input: Audio data stored on the device

[1189] Output: Text data sent to the server

[1190] Step 3: Analyzing the text data

[1191] The server uses generative AI tools (e.g., BERT or Generative AI-3) to analyze the received text data. As a result of the analysis, important information is extracted.

[1192] Specific behavior:

[1193] The server activates the generative AI means.

[1194] Text data is input into a generative AI tool and analyzed.

[1195] Extract important keywords and phrases.

[1196] Input: Text data sent to the server

[1197] Output: Extracted important information

[1198] Step 4: Generate a summary

[1199] The server generates a summary based on the extracted important information, and the generated summary is stored on the server.

[1200] Specific behavior:

[1201] The server inputs the extracted information into a summary generation model.

[1202] A summary sentence is generated using a summary sentence generation model.

[1203] The generated summary is stored on the server.

[1204] Input: Extracted important information

[1205] Output: Generated summary

[1206] Step 5: Execute the automation

[1207] The server performs the following automated tasks based on the generated summary:

[1208] 1. Automatic generation of meeting minutes

[1209] 2. Providing standard answers to questions

[1210] 3. Check homework

[1211] 4. Download materials

[1212] 5. Data Registration

[1213] Specific behavior:

[1214] The server automatically generates minutes based on the summary and sends them to the user.

[1215] The server generates and provides standard answers to questions from users based on the summary text.

[1216] The server checks the homework items based on the summary and notifies the user.

[1217] The server automatically downloads the necessary materials and provides them to the user.

[1218] The server registers the generated data in the appropriate location (e.g., a CRM system).

[1219] Input: Generated summary

[1220] Output: Execution results of automated tasks

[1221] (Application example 3)

[1222] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1223] In conventional sales assistant systems and factory work efficiency systems, the texting, summarization, and organization of voice data was often done manually, resulting in problems with reduced work efficiency. Furthermore, they lacked functions such as summarizing and organizing work instructions, automatically recording work progress, providing standard responses to worker questions, and automatically downloading and registering necessary parts and materials, and as a result, work efficiency was not sufficiently improved. This increased the burden on workers and led to issues with reduced overall work efficiency.

[1224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1225] In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading and registering materials, a means for summarizing and organizing work instructions, a means for automatically recording work progress, a means for providing standard answers to questions from workers, and a means for automatically downloading and registering necessary parts and materials. This enables efficient textualization, summarization, and organization of voice data, and further automates the summarization and organization of work instructions, the automatic recording of work progress, the standard answers to questions from workers, and the automatic downloading and registration of necessary parts and materials, thereby improving overall work efficiency.

[1226] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1227] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[1228] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries.

[1229] Minute-taking is the process of recording the contents of a meeting or discussion so that it can be referenced later.

[1230] "Standard Answers to Questions" is a feature that provides pre-prepared answers to frequently asked questions.

[1231] "Checking homework" is the process of checking the progress of assignments and tasks given in meetings and consultations.

[1232] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[1233] "Registration" is the process of storing acquired data and materials in an appropriate location.

[1234] "Means for summarizing and organizing work instructions" refers to a technique for concisely summarizing and organizing the contents of work instructions.

[1235] "Means for automatically recording work progress" refers to technology that automatically records the progress of work.

[1236] The "means for providing standard answers to questions from workers" is a technique for providing answers prepared in advance to questions from workers.

[1237] "Means for automatically downloading and registering necessary parts and materials" refers to technology that automatically obtains the parts and materials necessary for the work and stores them in the appropriate location.

[1238] To implement the present invention, the following system configuration and program are required.

[1239] System Configuration

[1240] The system consists of the following main components:

[1241] 1. Speech recognition: This involves analyzing voice data and converting it into text using technology such as speech recognition software or APIs (e.g., Google Speech-to-Text API).

[1242] 2. Text conversion means: A process is carried out to convert the voice data obtained by the voice recognition means into text format.

[1243] 3. Generative AI: Analyzes text data using natural language processing techniques, extracts important information, and generates summaries. Specifically, generative AI models such as BERT and GPT-3 are used.

[1244] 4. Means of automating tasks such as taking minutes, providing standard answers to questions, checking homework, downloading materials, and registering: To achieve these functions, we will integrate with CRM systems and minute-taking tools.

[1245] 5. Summarize and organize work orders: Use software to summarize and organize work orders concisely.

[1246] 6. Means of automatic recording of work progress: Use a system to automatically record the progress of work.

[1247] 7. Means of providing standard answers to worker questions: Use an FAQ system to provide pre-prepared answers to worker questions.

[1248] 8. Means of automatic download and registration of required parts and materials: Use a download manager to automatically obtain required parts and materials and save them in the appropriate location.

[1249] Program processing explanation

[1250] The server uses speech recognition to convert voice data into text data. Next, it uses generative AI to analyze the text data, extract important information, and generate a summary. Based on this summary, it automates tasks such as creating meeting minutes, providing standard answers to questions, confirming homework assignments, downloading and registering materials, and more. It also summarizes and organizes work instructions, automatically records work progress, provides standard answers to questions from workers, and automatically downloads and registers required parts and materials.

[1251] Specifically, the following hardware and software are used.

[1252] Hardware: Server equipped with a high-performance GPU

[1253] Software: Python 3.x, Transformers library, PyTorch, Google Speech-to-Text API, CRM system, meeting minutes tool, FAQ system, download manager

[1254] Specific examples

[1255] For example, a factory work instruction manual contains multiple steps and important points to note. By using this system, workers can quickly grasp important information and proceed with their work efficiently.

[1256] Example prompt for a generative AI model:

[1257] Work order text:

[1258] 1. Attach part A.

[1259] 2. When installing part B, use a torque wrench.

[1260] 3. Once completed, quality inspection is carried out.

[1261] By inputting this prompt, the generative AI model extracts important information and generates a summary.

[1262] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1263] Step 1:

[1264] The server converts the voice data into text data using a voice recognition means.

[1265] Input: Audio data

[1266] Data processing: Using voice recognition software (e.g., Google Speech-to-Text API), we analyze the voice data and convert it into text data.

[1267] Output: Text data

[1268] Step 2:

[1269] The server uses generative AI tools to analyze the text data, extract important information, and generate a summary.

[1270] Input: Text data

[1271] Data Computing: Using generative AI models such as BERT and GPT-3 to extract important information from text data and generate summaries.

[1272] Output: Summary

[1273] Step 3:

[1274] The server automatically generates minutes based on the generated summary.

[1275] Input: Abstract

[1276] Data processing: Organize the summaries according to the format and save them as minutes.

[1277] Output: Meeting minutes

[1278] Step 4:

[1279] The server provides standard answers to questions posed by the user.

[1280] Input: User question

[1281] Data Computing: Use an FAQ system to find and provide standard answers to questions.

[1282] Output: Standard answer

[1283] Step 5:

[1284] The server checks the homework items.

[1285] Input: List of homework items

[1286] Data processing: Check the progress of homework items and update the list.

[1287] Output: Updated list of homework items

[1288] Step 6:

[1289] The server automatically downloads the necessary materials and registers them in the appropriate locations.

[1290] Input: URL or identification information of the material

[1291] Data processing: Use the download manager to obtain the materials and save them in the specified folder.

[1292] Output: Saved material

[1293] Step 7:

[1294] The server summarizes and organizes the work orders.

[1295] Input: Work order text

[1296] Data Computation: Using generative AI models to concise and organize work instructions.

[1297] Output: Summarized work instructions

[1298] Step 8:

[1299] The server automatically records the progress of the work.

[1300] Input: Work progress data

[1301] Data processing: Work progress is automatically recorded and saved in a database.

[1302] Output: Recorded work progress data

[1303] Step 9:

[1304] The server provides standard answers to questions from workers.

[1305] Input: Worker Question

[1306] Data Computing: Use an FAQ system to find and provide standard answers to questions.

[1307] Output: Standard answer

[1308] Step 10:

[1309] The server automatically downloads the necessary parts and materials and registers them in the appropriate location.

[1310] Input: Part or material identification information

[1311] Data processing: Use the download manager to obtain parts and materials and save them in the specified folder.

[1312] Output: Saved parts and documents

[1313] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1314] "Example 1"

[1315] In one embodiment of the present invention, the speech recognition means includes an emotion engine that recognizes the user's emotion. This emotion engine analyzes the emotion from the user's speech and transmits the result along with the speech data to the generative AI means. Specifically, when the user utters a speech that indicates an emotion such as joy, anger, or sadness, the emotion engine recognizes the emotion and transmits it along with the speech data to the generative AI means.

[1316] "Example 2"

[1317] The generative AI means receives the voice data and emotion data and summarizes and organizes the written data taking them into account. Specifically, if the user makes a sound that indicates joy, the generative AI means summarizes and organizes the written data taking that information into account. This allows for the generation of more appropriate minutes and answers to questions that reflect the user's emotions.

[1318] "Example 3"

[1319] The automation means takes into account the summary, organized data, and emotional data received from the generative AI means to create minutes, provide standard answers to questions, confirm homework, download materials, register, etc. Specifically, if a user makes a sound that indicates anger, that information is taken into account when creating minutes or providing answers to questions. This allows for more appropriate task automation that reflects the user's emotions.

[1320] The processing flow of each embodiment will be described below.

[1321] "Example 1"

[1322] Step 1: The user speaks.

[1323] Step 2: The voice recognition means recognizes the user's voice, and the emotion engine analyzes the emotion from the voice.

[1324] Step 3: The emotion engine sends the analysis results along with the voice data to the generative AI means. "Example 2"

[1325] Step 1: The generative AI means receives the voice data and emotion data.

[1326] Step 2: The generative AI tool summarizes and organizes the written data, taking into account the audio data and emotion data.

[1327] Step 3: The generative AI method sends the summarized and organized data to the automated method.

[1328] "Example 3"

[1329] Step 1: The automated method receives the summarized and organized data and sentiment data.

[1330] Step 2: The automated means takes into account the received data and performs tasks such as creating meeting minutes, providing standard answers to questions, checking homework items, downloading materials, registering, etc.

[1331] Example 1

[1332] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1333] Conventional sales assistant systems provide basic functions such as recording, transcribing, summarizing, and organizing sales negotiations, but lack the ability to analyze users' emotions and provide advanced suggestions and feedback based on those emotions. This makes it difficult to automatically suggest specific actions to improve the quality of sales negotiations.

[1334] The identification process by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering, an emotion analysis means for analyzing the user's emotions, and a means for transmitting the emotion data obtained by the emotion analysis means to the generative AI means. This enables advanced suggestions and feedback based on the user's emotions.

[1335] "Speech recognition means" is a technology for acquiring speech as digital data and analyzing its content.

[1336] "Means of converting text" refers to technology that converts voice data into text data.

[1337] "Generative AI means" refers to artificial intelligence technology that performs processes such as summarizing, organizing, and proposing based on input data.

[1338] "Minutes-taking" is the process of recording the contents of a meeting or business negotiation so that it can be referenced later.

[1339] "Canonical Answers to Questions" is the process of providing pre-prepared answers to frequently asked questions.

[1340] "Homework review" is the process of reviewing assignments and tasks that came up in a business meeting or negotiation and tracking their progress.

[1341] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[1342] "Registration" is the process of entering and storing data or information in a system.

[1343] "Emotion analysis means" is a technology that analyzes the speaker's emotions from voice data and obtains the results.

[1344] "Emotion data" is information about the speaker's emotions obtained by emotion analysis means.

[1345] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales visit and records the details of the business negotiation. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[1346] Hardware and software used

[1347] Hardware

[1348] Smartphones (e.g., smartphones, Android devices)

[1349] Tablets (e.g. iPad, Android tablets)

[1350] software

[1351] Sales assistant apps (e.g. iOS apps, Android apps)

[1352] Emotion analysis tools (e.g. IBM Watson Tone Analyzer, Microsoft Azure Emotion API)

[1353] Generative AI means (e.g. OpenAI GPT-3, Google BERT)

[1354] Program processing

[1355] 1. Launch the app

[1356] A user launches the sales assistant app on their smartphone or tablet.

[1357] Specific action: The user taps the Sales Assistant app icon on the smartphone home screen.

[1358] 2. Start voice recognition

[1359] After the app is launched, the device starts the voice recognition function to record the details of the business meeting.

[1360] Specific operation: When the user presses the "Start Recording" button in the app, the device's microphone begins recording the audio of the business meeting.

[1361] 3. Performing sentiment analysis

[1362] The terminal analyzes the recorded voice data in real time and recognizes the user's emotions using emotion analysis means.

[1363] Specific behavior: When a user says, "This product is really great!", the device's emotion analysis means (e.g., IBM Watson Tone Analyzer) recognizes the emotion of "joy."

[1364] 4. Data transmission

[1365] The device transmits the analyzed emotion data and voice data to the generative AI means.

[1366] Specific operation: After the emotion analysis means recognizes the emotion of "joy," the device sends that information along with the voice data to a generative AI means (e.g., OpenAI GPT-3) on the cloud.

[1367] 5. Processing by generative AI models

[1368] Based on the voice and emotion data received by the server, the generative AI model summarizes the content of the business negotiation and suggests the next action.

[1369] Specific operation: The server analyzes the received data, and in response to a statement such as "This product is really great!", the generative AI model generates a summary and suggestion such as "The user is very satisfied with the product, so we suggest providing additional product information on their next visit."

[1370] Specific examples

[1371] Prompt Sentence Examples

[1372] "When a user makes a sales call, they launch the app and record the details of the sales meeting. The recorded voice data is analyzed by the emotion analysis means to recognize the user's emotions. For example, if the user says, 'This product is truly amazing!', the emotion analysis means recognizes the emotion of 'joy' and sends that information along with the voice data to the generative AI model. Based on the received data, the generative AI model summarizes the details of the sales meeting and suggests the next action."

[1373] In this way, the sales assistant system of the present invention can improve the quality of sales negotiations by recording the user's voice, analyzing their emotions, and sending them to the generative AI model.

[1374] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1375] Step 1:

[1376] A user launches the sales assistant app on a smartphone or tablet. The input is the user tapping the app icon. The output is the app launching and the main screen being displayed.

[1377] Step 2:

[1378] After the app launches, the device starts the voice recognition function to record the business negotiations. The input is the user pressing the "Start Recording" button in the app. The output is the device's microphone starting to record the business negotiations.

[1379] Step 3:

[1380] The device analyzes the recorded voice data in real time and recognizes the user's emotions using an emotion analysis means. The input is the recorded voice data. The output is generated by the emotion analysis means, which recognizes the user's emotions (e.g., joy, anger, sadness). In concrete terms, if a user says, "This product is really great!", the emotion analysis means recognizes the emotion of "joy."

[1381] Step 4:

[1382] The terminal sends the analyzed emotion data and voice data to the generative AI means. The emotion data and voice data obtained by the emotion analysis means are input. As output, these data are sent to the generative AI means on the cloud. In concrete terms, after the emotion analysis means recognizes the emotion of "joy," the terminal sends that information along with the voice data to the generative AI means.

[1383] Step 5:

[1384] Based on the voice data and emotion data received by the server, the generative AI model summarizes the content of the business negotiation and suggests the next action. The input is the voice data and emotion data received by the server. As output, the generative AI model generates a summary of the content of the business negotiation and a suggestion for the next action. Specifically, in response to a statement such as "This product is truly amazing!", the server generates a summary and suggestion such as "Because the user is very satisfied with the product, we suggest providing additional product information on your next visit."

[1385] (Application example 1)

[1386] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1387] Conventional sales assistant systems offer basic functions such as recording, transcribing, summarizing, and organizing sales negotiations, but they have the problem of not being able to analyze customer emotions in real time and respond accordingly. This has led to a decline in the quality of customer service and a lack of improvement in customer satisfaction.

[1388] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, confirming homework, downloading materials, and registering, an emotion engine for analyzing user emotions, and a means for transmitting emotion data analyzed by the emotion engine to the generative AI means. This makes it possible to analyze customer emotions in real time and respond based on the results.

[1389] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1390] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text format.

[1391] "Generative AI tools" are technologies that summarize and organize written data, and automate tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information.

[1392] An "emotion engine" is a technology that analyzes emotions from a user's voice and outputs the results as data.

[1393] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[1394] A "sales assistant system" is a system that supports sales activities by recording the contents of sales negotiations during sales visits and performing voice recognition, text conversion, summarization, organization, emotion analysis, etc.

[1395] As an embodiment of the present invention, a customer assistance system for a brick-and-mortar store will be described as an example. This system includes a speech recognition unit, a text generation unit, a generative AI unit, an emotion engine, and an emotion data transmission unit.

[1396] System configuration

[1397] 1. Voice recognition means:

[1398] Using an application installed on a smartphone or smart glasses.

[1399] Record conversations with customers and obtain audio data.

[1400] 2. Means of writing:

[1401] The voice data obtained by the voice recognition means is converted into text data.

[1402] Specifically, we use a Python speech recognition library.

[1403] 3. Generative AI means:

[1404] It summarizes and organizes written data, and automates tasks such as creating minutes, providing standard answers to questions, checking homework assignments, downloading materials, and registering.

[1405] Specifically, we use the transformers library to perform sentiment analysis.

[1406] 4. Emotion Engine:

[1407] Emotions are analyzed from the user's voice and the results are output as data.

[1408] The analyzed emotional data is sent to the generative AI means.

[1409] 5. Means of sending emotional data:

[1410] The emotion data analyzed by the emotion engine is sent to the generative AI means.

[1411] This allows the generative AI means to respond based on the customer's emotions.

[1412] Processing flow

[1413] The server first records the customer's voice using a voice recognition means. Next, the recorded voice data is converted into text data using a text conversion means. After that, the generative AI means summarizes and organizes the text data and automates the necessary information. Furthermore, the emotion engine analyzes the user's emotions and sends the emotion data to the generative AI means. This enables the generative AI means to respond based on the customer's emotions.

[1414] Specific examples

[1415] For example, if a customer says, "I'd like to know more about this product," the voice recognition means records this statement, and the text translation means converts it into text data. The emotion engine analyzes this text data and determines that the customer's emotion is "interest." Based on this information, the generative AI means provides the staff with appropriate responses and additional information.

[1416] Prompt Sentence Examples

[1417] Customers say: I want to know more about this product

[1418] Emotion: Interest

[1419] In this way, customer service can be more effectively provided in physical stores.

[1420] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1421] Step 1:

[1422] The server uses a voice recognition means to record the customer's voice. Specifically, it starts an application installed on a smartphone or smart glasses and records the conversation with the customer. The input is the customer's voice data, and the output is a recorded voice file.

[1423] Step 2:

[1424] The server converts the recorded voice data into text data using a text-to-speech conversion method. Specifically, it uses a Python speech recognition library to convert the voice data into text. The input is an audio file, and the output is text data.

[1425] Step 3:

[1426] The server uses generative AI tools to summarize and organize text data. Specifically, it uses the transformers library to analyze, summarize, and organize text data. The input is text data, and the output is summarized text data.

[1427] Step 4:

[1428] The server analyzes the user's emotions using an emotion engine. Specifically, it analyzes emotions from text data using the emotion analysis model in the transformers library. The input is text data, and the output is emotion data.

[1429] Step 5:

[1430] The server transmits the emotion data analyzed by the emotion engine to the generative AI means. The input is the emotion data, and the output is the emotion data transmitted to the generative AI means.

[1431] Step 6:

[1432] The server uses generative AI to respond based on the customer's emotions. Specifically, it generates appropriate responses and additional information based on emotion data and text data. The inputs are emotion data and text data, and the output is responses and additional information.

[1433] Step 7:

[1434] The terminal presents the response methods and additional information provided by the generative AI means to the staff. Specifically, it displays them on the display of a smartphone or smart glasses. The input is the response methods and additional information, and the output is what is presented to the staff.

[1435] Example 2

[1436] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1437] Conventional voice recognition systems simply convert voice data into text data, and do not summarize or organize the data to reflect the user's emotions, which can result in inappropriate minutes and answers to questions. Furthermore, the process of recording voice data, generating text data, summarizing, and organizing it is often done manually, resulting in inefficiencies. Furthermore, there is a need to process data quickly even during sales visits or while traveling, which is difficult to achieve with current systems.

[1438] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1439] In this invention, the server includes a speech recognition unit, a means for converting speech data obtained by the speech recognition unit into text, a generative AI unit for summarizing and organizing the textual data, an emotion analysis unit, and a means for summarizing and organizing the textual data in consideration of the emotion data obtained by the emotion analysis unit. This enables appropriate minutes and answers to questions that reflect the user's emotions. Furthermore, the process from recording speech data to generating, summarizing, and organizing text data is automated and can be performed efficiently. Furthermore, data can be processed quickly even when visiting or on the move.

[1440] "Speech recognition means" is a technology that analyzes voice data and converts it into corresponding text data.

[1441] The "text conversion means" is a technology that executes a process of converting the voice data obtained by the voice recognition means into text data.

[1442] "Generative AI means" refers to artificial intelligence technology for summarizing and organizing text data, and in particular has the ability to generate appropriate sentences by taking into account the user's emotions.

[1443] "Emotion analysis means" is a technology that analyzes a user's emotions from text data or voice data and acquires emotion data.

[1444] "Means for summarizing and organizing" refers to a technology that uses generative AI means to carry out the process of summarizing and organizing text based on text data and emotional data.

[1445] "Minutes taking" is the process of recording the contents of a meeting or discussion and creating a document for future reference.

[1446] "Canonical Answers to Questions" is the process of providing pre-canned answers to frequently asked questions.

[1447] "Homework review" is the process of reviewing action items and tasks decided upon in meetings and tracking their progress.

[1448] "Downloading materials" is the process of obtaining necessary materials and documents from the Internet or an internal network.

[1449] "Registration" is the process of adding new data or information to the system.

[1450] This invention is a system that combines speech recognition technology and generative AI technology, which converts a user's speech data into text data, and then summarizes and organizes it while taking into account emotional data, thereby generating appropriate minutes and answers to questions. Specific embodiments of this system are described below.

[1451] First, the user uses the device to record audio data. The device is equipped with a recording function, allowing high-quality recording of audio from meetings, interviews, etc. Once recording is complete, the device sends the recorded audio data to the server. An HTTP request is used for transmission, and the audio data is uploaded to the server.

[1452] The server sends the received voice data to a voice recognition service. Specifically, it uses voice recognition services such as Google's voice recognition API or IBM's Watson. These services analyze the voice data and generate corresponding text data. For example, if a user says, "I had a lot of fun today," the voice recognition service generates the text data, "I had a lot of fun today."

[1453] Next, the server uses an emotion analysis tool to obtain emotion data from the text data. The emotion analysis tool analyzes the text data and identifies the user's emotion. For example, emotion data of "joy" is obtained from text data such as "I had a lot of fun today."

[1454] The server uses generative AI means to summarize and organize the text data and emotion data. The generative AI means takes the user's emotions into consideration to generate more appropriate minutes and answers to questions. For example, based on the text data "Today was a lot of fun" and the emotion data "joy," it generates a summary that reads, "The user really enjoyed today's activities."

[1455] The generated text data is sent from the server to the terminal. The terminal displays the received text data to the user. The user can check the generated summary and minutes on the terminal screen.

[1456] For example, the following prompts can be fed to a generative AI model:

[1457] Prompt Sentence Examples

[1458] Audio data: "I had a lot of fun today."

[1459] Emotion data: "Joy"

[1460] Generated text: "The user enjoyed today's activity very much."

[1461] Using this prompt, the generative AI model can take into account the speech and emotion data to generate an appropriate summary and organized text.

[1462] This system allows users to efficiently convert voice data into text data and quickly generate appropriate summaries and minutes that reflect the emotions of the participants, making it extremely convenient for accurately recording the contents of meetings and interviews and for later reference.

[1463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1464] Step 1:

[1465] The user records the audio data.

[1466] A user can record voice data using the recording function of a device. For example, they can record speech during a meeting or the contents of an interview. The input is the user's voice, and the output is the recorded voice data.

[1467] Step 2:

[1468] The device sends the recorded audio data to the server.

[1469] The device sends the recorded audio data to the server. Specifically, the device application uploads the audio data to the server as an HTTP request. The input is the recorded audio data, and the output is the audio data sent to the server.

[1470] Step 3:

[1471] The server uses a voice recognition service to convert the voice data into text data.

[1472] The server sends the received voice data to a voice recognition service. Specifically, it uses a general voice recognition API. The voice recognition service analyzes the voice data and generates corresponding text data. The input is voice data, and the output is text data. For example, voice data saying "I had a lot of fun today" is converted into text data saying "I had a lot of fun today."

[1473] Step 4:

[1474] The server acquires emotion data using an emotion analysis tool.

[1475] The server sends the text data to an emotion analysis tool and obtains emotion data. The emotion analysis tool analyzes the text data and identifies the user's emotion. The input is text data and the output is emotion data. For example, the emotion data obtained from the text data "I had a lot of fun today" is "joy."

[1476] Step 5:

[1477] The server uses generative AI tools to summarize and organize text data and emotion data.

[1478] The server uses generative AI means to summarize and organize the text data and emotional data. The generative AI means takes the user's emotions into consideration to generate more appropriate minutes and answers to questions. The input is text data and emotional data, and the output is summarized and organized text data. For example, based on the text data "Today was a lot of fun" and the emotional data "joy," it generates the summary "The user really enjoyed today's activities."

[1479] Step 6:

[1480] The server transmits the generated text data to the terminal.

[1481] The server sends the generated text data to the terminal. Specifically, it returns the generated text data to the terminal as an HTTP response. The input is the summarized and organized text data, and the output is the text data sent to the terminal.

[1482] Step 7:

[1483] The user checks the generated text data.

[1484] The user checks the text data generated by the terminal. For example, the user can check a summary or minutes displayed in a terminal application. The input is the text data sent to the terminal, and the output is the text data checked by the user.

[1485] (Application example 2)

[1486] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1487] Customer service in brick-and-mortar stores requires immediate understanding of customer needs and emotions and the ability to respond appropriately. However, conventional methods require store clerks to manually record conversations with customers and analyze them later, making it difficult to respond in real time. It is also difficult to accurately grasp customer emotions, which can lead to inappropriate responses. A system that can solve these problems and improve customer satisfaction is needed.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1489] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for recording conversations with customers and converting them into text and performing sentiment analysis in real time, and a means for instantly grasping customer needs and emotions and supporting appropriate responses. This makes it possible to streamline customer service operations in brick-and-mortar stores and grasp customer needs and emotions in real time.

[1490] "Speech recognition means" is a technology for converting voice data into text data.

[1491] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into text data.

[1492] "Generative AI methods" are artificial intelligence techniques for summarizing and organizing text data.

[1493] "Sentiment analysis" is a technology that analyzes emotions from text data and identifies those emotions.

[1494] A "customer service support system" is a system for supporting customer service operations in physical stores.

[1495] "Real-time text conversion" is the process of instantly converting audio data into text data.

[1496] "Means of understanding customer needs and emotions" refers to techniques for analyzing and understanding customer needs and emotions through conversations with customers.

[1497] "Means to support appropriate responses" are support technologies that enable optimal responses based on the customer's needs and emotions.

[1498] As an embodiment of the present invention, a specific method for constructing a customer service support system will be described below.

[1499] First, the server uses a speech recognition tool to record conversations with customers in physical stores and convert them into text data in real time. A common speech recognition API can be used as the speech recognition tool, such as Google's speech recognition API or IBM's Watson.

[1500] Next, the server uses a generative AI method to summarize and organize the text data. This generative AI method has the ability to analyze the content of the text data, extract important information, and summarize it. Specifically, it uses an AI model that uses natural language processing technology. For example, it can use the Transformers library.

[1501] Furthermore, the server analyzes customer sentiment from the text data using a sentiment analysis means. As the sentiment analysis means, a sentiment analysis model can be used. For example, a Sentiment Analysis Pipeline can be used.

[1502] This allows the server to instantly understand the customer's needs and feelings and provide appropriate support, such as providing information about products and services that the customer is interested in, or taking measures to resolve customer dissatisfaction.

[1503] For example, if a store clerk records a conversation with a customer and the application converts it into text and performs sentiment analysis in real time, if the customer says, "I'd like to know more about this product," the application can convert the remark into text and use sentiment analysis to determine that the customer is interested, prompting the store clerk to respond appropriately.

[1504] Examples of prompts for generative AI models include:

[1505] Record your conversations with customers and follow these steps:

[1506] 1. Convert the audio data into text data.

[1507] 2. Analyze the sentiment of text data.

[1508] 3. Summarize text data.

[1509] Output the results in the following format:

[1510] Transcript: [Text data]

[1511] Emotion: [Emotion data]

[1512] Summary: [Summary data]

[1513] In this way, customer service operations in physical stores can be made more efficient and customer satisfaction can be improved.

[1514] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1515] Step 1:

[1516] The server uses a voice recognition means to record conversations with customers in the physical store.

[1517] Input: Voice data of conversation with customer

[1518] Output: Recorded audio data file

[1519] Specific operation: The server acquires audio data through a microphone and saves it as an audio file.

[1520] Step 2:

[1521] The server converts the recorded voice data into text data using a voice recognition means.

[1522] Input: Recorded audio data file

[1523] Output: Text data

[1524] Specific operation: The server calls a speech recognition API and converts the speech data into text data. For example, it uses Google's speech recognition API.

[1525] Step 3:

[1526] The server uses generative AI tools to summarize and organize the text data.

[1527] Input: Text data

[1528] Output: Summarized text data

[1529] How it works: The server uses an AI model that uses natural language processing techniques to analyze the content of the text data, extract important information, and summarize it, for example, using the Transformers library.

[1530] Step 4:

[1531] The server uses a sentiment analysis means to analyze the sentiment of the customer from the text data.

[1532] Input: Text data

[1533] Output: Emotion data

[1534] Specific operation: The server uses a sentiment analysis model to analyze the sentiment of the text data, for example, using the Sentiment Analysis Pipeline.

[1535] Step 5:

[1536] The server uses the summarized text data and emotional data to understand the customer's needs and emotions and supports appropriate responses.

[1537] Input: Summarized text data, emotion data

[1538] Output: Response instruction data

[1539] Specific Actions: The server analyzes the summarized text data and emotion data and generates specific instructions for the store clerk based on the customer's needs and emotions.

[1540] Step 6:

[1541] The server transmits the generated response instruction data to the terminal and notifies the store clerk.

[1542] Input: Response instruction data

[1543] Output: Notification to store clerk

[1544] Specific operation: The server sends the response instruction data to the terminal and notifies the store clerk, for example, by displaying the notification on a smartphone or smart glasses.

[1545] In this way, it becomes possible to streamline customer service operations in physical stores and understand customer needs and emotions in real time.

[1546] Example 3

[1547] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1548] In conventional sales assistant systems, the transcription, summarization, and organization of voice data is often done manually, which is inefficient. Also, it is difficult to improve user satisfaction because it is not possible to automate tasks that take the user's emotions into account. Furthermore, there is an issue of a heavy workload because tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering are not automated.

[1549] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1550] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, and a means for automating tasks taking into account user emotion data. This enables efficient textualization, summarization, and organization of speech data, realizing task automation that reflects the user's emotions. Furthermore, because tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering are automated, the workload is reduced.

[1551] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1552] The "means for converting text" is a technique for recording voice data obtained by a voice recognition means as text data.

[1553] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries based on that information.

[1554] The "minutes creation method" is a technology that automatically creates a document that records the contents of a meeting based on a summary generated by a generative AI method.

[1555] "Standard answering means for questions" is a technology that provides standard answers to questions posed by users based on summary sentences generated by generative AI means.

[1556] The "homework confirmation method" is a technology that confirms unresolved issues and action items in meetings and work based on summaries generated by generative AI methods.

[1557] "Material downloading means" refers to technology that automatically downloads necessary materials.

[1558] The "registration means" is a technology that automatically registers the summary text and minutes generated by the generative AI means in the appropriate location.

[1559] "Means that take emotion data into consideration" refers to technology that analyzes emotions from user voice data and automates tasks by reflecting those emotions.

[1560] MODE FOR CARRYING OUT THE INVENTION

[1561] This invention is a system that includes a speech recognition means, a text generation means, a generative AI means, a minutes creation means, a standard answer means for questions, a homework confirmation means, a material download means, a registration means, and a means for considering emotional data. Specific embodiments of this system are described below.

[1562] System configuration

[1563] The server uses voice recognition software (for example, a voice recognition API) as a means of voice recognition. When a user uploads voice data from a terminal, the server receives this voice data and converts it into text data using the voice recognition software.

[1564] Next, the server uses a natural language processing model (e.g., BERT or Generative AI-3) as a generative AI tool to analyze the converted text data and extract important information. Based on the extracted information, the generative AI model generates a summary.

[1565] Based on the generated summary, the server automatically generates minutes using a minutes-creation tool. The server then formats the minutes according to the format in cooperation with the minutes-creation tool. Furthermore, the server uses a standard answering tool to provide standard answers to questions from users.

[1566] The server also uses a homework confirmation means to confirm unresolved issues and action items in meetings and business, a material download means to automatically download necessary materials, and a registration means to automatically register the generated summary and minutes in appropriate locations.

[1567] Furthermore, the server uses a means to consider emotional data to analyze the user's voice data and automate tasks based on that emotion. For example, if the user makes a sound that indicates anger, the server can use that information to add notes to meeting minutes or adjust responses to questions.

[1568] Specific examples

[1569] As a concrete example, consider a scenario in which a user uploads audio data from a meeting. After the meeting, the user uploads the recorded audio file from their device to the server. The server converts the audio data into text data using a speech recognition API. The converted text data is analyzed using the BERT model to extract important information. Based on the extracted information, the generative AI-3 model generates a summary. The generated summary is formatted as minutes using a minutes-taking tool and registered in the CRM system. Finally, the server extracts the user's emotional data from the audio data, and if any voice indicating anger is included, adds a note to the minutes based on that information.

[1570] Example prompt sentence:

[1571] "Please prepare minutes of yesterday's meeting. The audio can be downloaded from the link below. Please summarize key decisions and action items."

[1572] In this way, the server automatically creates minutes based on the user's instructions and organizes and provides the necessary information. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[1573] Step 1:

[1574] The user uploads the audio data.

[1575] After the meeting, the user uploads the recorded audio file from their device to the server. Specifically, the user drags and drops the audio file into a designated upload form. The input is the audio file, and the output is the audio data stored on the server.

[1576] Step 2:

[1577] The server converts the voice data into text data.

[1578] The server receives the uploaded voice data and converts it into text data using voice recognition software (e.g., a voice recognition API). The input is voice data, and the output is text data. Specifically, the voice recognition API analyzes the voice data and generates corresponding text.

[1579] Step 3:

[1580] The server analyzes the text data and extracts important information.

[1581] The server inputs the converted text data into a natural language processing model (e.g., BERT) to extract important information. The input is text data, and the output is data containing important information. Specifically, the BERT model analyzes the text data and identifies meeting agendas, decisions, action items, etc.

[1582] Step 4:

[1583] The server generates a summary based on the extracted information.

[1584] The server generates a summary using a generative AI model (e.g., Generative AI-3) based on the extracted important information. The input is data containing important information, and the output is a summary. Specifically, the Generative AI-3 model summarizes the important information and creates a concise, easy-to-understand summary.

[1585] Step 5:

[1586] The server creates minutes based on the summary.

[1587] The server automatically generates minutes based on the generated summary. The input is the summary, and the output is the minutes. Specifically, it works in conjunction with a minutes-creation tool to format the minutes according to the format.

[1588] Step 6:

[1589] The server registers the minutes in the CRM system.

[1590] The server registers the generated meeting minutes in the CRM system. The input is the meeting minutes, and the output is the data registered in the CRM system. Specifically, the minutes are uploaded to the CRM system via API and linked to related project and customer information.

[1591] Step 7:

[1592] The server automates tasks by taking into account the user's emotional data.

[1593] The server extracts the user's emotional data from the voice data and automates tasks by reflecting that emotion. The input is the voice data, and the output is the task results that take the emotional data into account. Specifically, if the user makes a sound that indicates anger, the server can use that information to add notes to the minutes or adjust responses to questions.

[1594] (Application example 3)

[1595] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1596] Conventional sales assistant systems provide functions such as voice recognition, text generation, summarization, and organization, but they do not support the automation of work instructions and reports within factories. Therefore, in order to improve work efficiency within factories, functions such as real-time summarization of work content, reporting to managers, standard answers to questions from workers, and automatic downloading and registration of necessary materials are required. This makes it a challenge to improve work efficiency within factories.

[1597] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1598] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, and a generative AI means for summarizing and organizing the textual data, thereby enabling the automation of tasks such as creating minutes, providing standard answers to questions, checking homework, downloading and registering materials, automating work instructions and reports within the factory, summarizing work content in real time and reporting it to a manager, providing standard answers to questions from workers, and automatically downloading and registering necessary materials.

[1599] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1600] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[1601] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries.

[1602] Minute-taking is the process of recording the contents of a meeting or discussion so that it can be referenced later.

[1603] "Canonical Answers to Questions" is the process of providing pre-made answers to frequently asked questions.

[1604] "Confirming homework" is the process of confirming assignments and tasks given in meetings and discussions and managing progress.

[1605] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[1606] "Registration" is the process of storing and managing acquired materials and data in an appropriate location.

[1607] "Means for automating work instructions and reports within a factory" refers to technology that gives instructions on work within a factory and automatically reports the results.

[1608] "Means for summarizing work content in real time and reporting to managers" refers to technology that summarizes work content performed within a factory in real time and immediately conveys that information to managers.

[1609] The "means for providing standard answers to questions from workers" is a technology that automatically provides pre-prepared answers to questions from workers.

[1610] "Means for automatically downloading and registering necessary materials" refers to technology that automatically obtains necessary materials from the Internet or an internal network and stores them in an appropriate location.

[1611] A system for carrying out this invention includes a speech recognition means, a text generation means, a generative AI means, a means for creating minutes, providing standard answers to questions, checking homework items, downloading and registering materials, a means for automating work instructions and reports within a factory, a means for summarizing work content in real time and reporting it to a manager, a means for providing standard answers to questions from workers, and a means for automatically downloading and registering necessary materials.

[1612] Hardware and Software Configuration

[1613] Hardware: Smartphones and tablets used by robots and workers in factories.

[1614] Software: Python, Transformers library, OpenAI API.

[1615] Data processing and calculation

[1616] 1. Speech recognition: Recognizes speech from factory workers and robots in real time and converts it into text data. A speech recognition engine is used for this purpose.

[1617] 2. Text conversion: The voice data obtained by the voice recognition means is converted into text format, and the work content is saved as text data.

[1618] 3. Generative AI methods: Using natural language processing models such as BERT and Generative AI-3, we analyze text data, extract important information, and generate summaries.

[1619] 4. Minutes creation: Minutes are automatically generated based on the summary text generated by generative AI tools.

[1620] 5. Standard answers to questions: Provide pre-prepared standard answers to questions posed by workers. This is done using OpenAI's generative AI.

[1621] 6. Check homework: Check the assignments and tasks given in meetings and discussions and manage progress.

[1622] 7. Material download and registration: Automatically download necessary materials from the Internet or internal network and store them in the appropriate location.

[1623] 8. Automating work instructions and reports within the factory: Give instructions on work within the factory and automatically report the results.

[1624] 9. Real-time summary and reporting of work: Summarize the work carried out in the factory in real time and immediately convey this information to management.

[1625] 10. Providing standard answers to worker questions: Automatically provide pre-prepared answers to worker questions.

[1626] 11. Automatic download and registration of required materials: Automatically retrieve required materials from the Internet or internal network and save them in the appropriate location.

[1627] Specific examples

[1628] Summary of work: Summarize the text as "Today's work will be maintenance on Line 1 and quality checks on Line 2" and report it to the manager.

[1629] Answering the Question: Provide a standard answer to the question, "What is the maintenance procedure for Line 1?"

[1630] Download and register the materials: Download the "Maintenance Manual" and register it in the appropriate location.

[1631] Prompt Sentence Examples

[1632] Summary generation: Summary: Today's work involves maintenance of Line 1 and quality check of Line 2.

[1633] Standard Answer: Question: What is the maintenance procedure for Line 1?

[1634] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1635] Step 1:

[1636] The voice recognition means is activated to recognize voices emitted by workers and robots in the factory in real time.

[1637] Input: Voice data from workers and robots.

[1638] Data processing: Converting voice data into text data using a speech recognition engine.

[1639] Output: Text data.

[1640] Step 2:

[1641] The text data obtained by the speech recognition means is converted into a text format using a sentence conversion means.

[1642] Input: Text data from a speech recognizer.

[1643] Data processing: Format text data and save it as text.

[1644] Output: Prose-encoded text data.

[1645] Step 3:

[1646] Using generative AI methods, we analyze written text data, extract important information, and generate a summary.

[1647] Input: Documented text data.

[1648] Data processing: Using natural language processing models such as BERT and generative AI-3, we analyze text data and generate summaries.

[1649] Output: Summary statement.

[1650] Step 4:

[1651] Minutes are automatically generated based on the summary text generated by generative AI tools.

[1652] Input: Abstract text.

[1653] Data processing: Based on the summary text, the data is formatted into minutes.

[1654] Output: Meeting minutes.

[1655] Step 5:

[1656] Provide pre-prepared standard answers to questions from workers.

[1657] Input: Question from worker.

[1658] Data processing: Use OpenAI's generative AI to generate standard answers to questions.

[1659] Output: Standard answer.

[1660] Step 6:

[1661] Check the issues and tasks raised in meetings and discussions and manage progress.

[1662] Input: Details of the meeting or discussion.

[1663] Data processing: List issues and tasks and manage progress.

[1664] Output: Issue or task progress.

[1665] Step 7:

[1666] Automatically download necessary materials from the Internet or internal network and save them in the appropriate location.

[1667] Input: Name or URL of the resource.

[1668] Data processing: Download the materials and save them in the specified location.

[1669] Output: The downloaded material.

[1670] Step 8:

[1671] It gives instructions on work to be done within the factory and automatically reports the results.

[1672] Input: Work instruction content.

[1673] Data processing: Create work instructions in text format and communicate them to workers and robots.

[1674] Output: Work order report.

[1675] Step 9:

[1676] It summarizes the work being done in the factory in real time and immediately conveys that information to managers.

[1677] Input: Text data of the work content.

[1678] Data processing: Summarizing the work and formatting it for reporting to management.

[1679] Output: Report to administrator.

[1680] Step 10:

[1681] The system automatically provides pre-prepared answers to questions from workers.

[1682] Input: Question from worker.

[1683] Data processing: Generate and provide standard answers to questions.

[1684] Output: Standard answer.

[1685] Step 11:

[1686] Automatically retrieve necessary materials from the Internet or internal network and store them in the appropriate location.

[1687] Input: Name or URL of the resource.

[1688] Data processing: Download the materials and save them in the specified location.

[1689] Output: The downloaded material.

[1690] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1691] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1693] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1694] [Third embodiment]

[1695] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1696] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1697] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1698] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1699] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1701] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1702] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1703] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1704] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1705] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1706] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1707] "Example 1"

[1708] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[1709] "Example 2"

[1710] The system of the present invention also includes a means for converting recorded voice data into text. This means converts voice data into text using voice recognition technology. Specifically, it can use voice recognition services such as Google's voice recognition API or IBM's Watson.

[1711] "Example 3"

[1712] Furthermore, the system of the present invention is equipped with a generative AI means for summarizing and organizing textual data. This generative AI means uses natural language processing technology to analyze text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and generative AI-3 can be used.

[1713] "Example 4"

[1714] The system of the present invention also includes a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering them. This means automatically generates minutes based on the summary generated by the generative AI means, provides standard answers to questions, checks homework, downloads necessary materials, and registers them in appropriate locations. Specifically, these tasks can be automated through collaboration with a minutes-creation tool or CRM system.

[1715] The processing flow of each embodiment will be described below.

[1716] "Example 1"

[1717] Step 1: During a sales visit, the sales representative launches an application installed on a mobile information device such as a smartphone or tablet.

[1718] Step 2: The application records the conversation and converts the audio data into text using a speech recognition tool such as Google's speech recognition API or IBM's Watson.

[1719] "Example 2"

[1720] Step 1: The recorded voice data is converted into text using a voice recognition device, and the voice data is converted into text data.

[1721] Step 2: The textual data is summarized and organized by a generative AI tool. This generative AI tool uses natural language processing techniques to analyze the text data, extract important information, and generate a summary based on that information. Specifically, natural language processing models such as BERT and Generative AI-3 are used.

[1722] "Example 3"

[1723] Step 1: Minutes are automatically generated based on the summary generated by the generative AI method. Step 2: Similarly, standard answers to questions are provided and homework items are confirmed.

[1724] Step 3: The necessary documents are downloaded and registered in the appropriate location. This automation is achieved through integration with meeting minutes creation tools and CRM systems.

[1725] Example 1

[1726] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1727] In conventional sales activities, accurately recording the details of sales negotiations and then summarizing and analyzing them later was a very time-consuming task. In addition, important points were often overlooked during sales negotiations, and when reviewing the records later, the information was often incomplete. This led to problems such as reduced sales efficiency and a decline in the quality of customer service.

[1728] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1729] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for sending speech data to the speech recognition engine in real time, a means for inputting prompts to the generative AI model, and a means for displaying responses from the generative AI model. This makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[1730] "Speech recognition means" is a technology that converts voice into digital data, analyzes the content, and outputs it as text data.

[1731] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text data.

[1732] "Generative AI methods" are artificial intelligence technologies that analyze text data and summarize and organize it.

[1733] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[1734] "Standard answers to questions" is a technology that automatically provides pre-defined answers to frequently asked questions.

[1735] "Confirming homework" is the process of confirming the tasks that came up in business negotiations or meetings and the tasks that need to be done by the next time.

[1736] "Downloading materials" is the process of obtaining necessary materials from the Internet or an internal network.

[1737] "Registration" is the process of entering and storing data or information in a system.

[1738] "Means for transmitting voice data to a voice recognition engine in real time" refers to a technology that immediately transmits recorded voice data to a voice recognition engine and analyzes it in real time.

[1739] A "means for inputting prompt sentences to a generative AI model" is an interface that allows a user to input instructions or questions to a generative AI model.

[1740] "Means for displaying responses from the generative AI model" refers to technology that displays answers or results returned from the generative AI model to the user.

[1741] MODE FOR CARRYING OUT THE INVENTION

[1742] The sales assistant system of the present invention is equipped with a voice recognition unit that starts an application during a sales call and records the details of the business negotiations. This voice recognition unit is realized as an application installed on a mobile information terminal such as a smartphone or tablet.

[1743] Hardware and software used

[1744] Hardware: Smartphones, tablets, cloud servers (e.g. AWS EC2)

[1745] Software: Google Cloud Speech-to-Text API, MySQL database, OpenAI GPT-3

[1746] Program processing explanation

[1747] 1. Launch the app

[1748] Users launch the app on their smartphones or tablets during sales calls.

[1749] Example: A user taps the "Sales Assistant" app from the home screen of their smartphone to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[1750] 2. Start voice recognition

[1751] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[1752] Example: When a user taps the "Start Recording" button in the app, the device's microphone activates and begins recording the business conversation. During recording, the recording time is displayed on the app screen.

[1753] 3. Audio Data Processing

[1754] The device sends the recorded voice data to the voice recognition engine in real time.

[1755] Software used: Google Cloud Speech-to-Text API

[1756] Example: The device sends recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is temporarily stored on the device.

[1757] 4. Saving text data

[1758] The server stores the text data received from the speech recognition engine in a database.

[1759] Hardware used: Cloud server (e.g. AWS EC2)

[1760] Software used: MySQL database

[1761] Example: The server receives text data sent from a device and saves it in a MySQL database on AWS EC2. When saving is complete, the server sends a notification to the device that saving is complete.

[1762] 5. Input to the generative AI model

[1763] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[1764] Software used: OpenAI GPT-3

[1765] Example: A user enters a prompt such as "Please write a summary of this deal" into an input field in an app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[1766] 6. Displaying the results

[1767] The device receives the response from the generative AI model and displays it to the user.

[1768] Example: The device receives a summary and analysis of a business deal from the generative AI model and displays it within the app. The user can check the summary and analysis on the app screen.

[1769] Examples of prompt statements

[1770] "Please write a summary for this opportunity"

[1771] "Analyze the content of the business negotiation and tell us the key points to propose next time."

[1772] The above is an embodiment of the sales assistant system of the present invention. This system makes it possible to accurately record the contents of business negotiations and perform summarization and analysis in real time.

[1773] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1774] Step 1:

[1775] Launching the app

[1776] Users launch the app on their smartphones or tablets during sales calls.

[1777] Specific behavior: The user taps the "Sales Assistant" app on the home screen of their smartphone to launch it. When the app launches, a "Start Recording" button appears on the home screen.

[1778] Input: User taps

[1779] Output: App home screen display

[1780] Step 2:

[1781] Start voice recognition

[1782] When the user presses the "Start Recording" button in the app, the device begins recording the business negotiations.

[1783] Specific operation: When the user taps the "Start Recording" button in the app, the device's microphone will be activated and the business conversation will begin to be recorded. During recording, the recording time will be displayed on the app screen.

[1784] Input: User taps

[1785] Output: Start recording, display recording time

[1786] Step 3:

[1787] Audio data processing

[1788] The device sends the recorded voice data to the voice recognition engine in real time.

[1789] Software used: Google Cloud Speech-to-Text API

[1790] How it works: The device sends the recorded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data, which is then temporarily stored on the device.

[1791] Input: Recorded audio data

[1792] Output: Text data

[1793] Step 4:

[1794] Saving text data

[1795] The server stores the text data received from the speech recognition engine in a database.

[1796] Hardware used: Cloud server (e.g. AWS EC2)

[1797] Software used: MySQL database

[1798] Specific operation: The server receives the text data sent from the device and saves it in a MySQL database on AWS EC2. Once saving is complete, the server sends a notification to the device.

[1799] Input: Text data

[1800] Output: Saving to database, notification of completion of saving

[1801] Step 5:

[1802] Input to generative AI models

[1803] When a user wants to summarize or analyze the contents of a business negotiation, they input a prompt statement into the generative AI model.

[1804] Software used: OpenAI GPT-3

[1805] Specific operation: The user enters the prompt "Please write a summary of this deal" into the input field in the app and presses the send button. The device sends the prompt and the text data to the generative AI model.

[1806] Input: prompt, text data

[1807] Output: Send to generative AI model

[1808] Step 6:

[1809] Displaying the results

[1810] The device receives the response from the generative AI model and displays it to the user.

[1811] Specific operation: The device receives the summary and analysis results of the business negotiation received from the generative AI model and displays them within the app. The user can check the summary and analysis results on the app screen.

[1812] Input: Response from a generative AI model

[1813] Output: Display of summary and analysis results

[1814] (Application example 1)

[1815] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1816] Conventional sales assistant systems were capable of recording and transcribing sales negotiations, but lacked the functionality to analyze the content of sales negotiations and automatically generate next actions and proposals. As a result, sales staff had to analyze the content themselves after the negotiation and consider the next steps, which was time-consuming and labor-intensive. In addition, summarizing and organizing the content of sales negotiations had to be done manually, which was inefficient. A system that could solve these problems and make sales activities more efficient was needed.

[1817] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1818] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, and a means for analyzing the content of business negotiations using a generative AI model and automatically generating next actions and proposals. This makes it possible to consistently automate the process from recording the content of business negotiations to analyzing it and proposing the next action.

[1819] "Speech recognition means" is a technology that acquires voice data and converts it into text data.

[1820] The "means for converting text" is a process for converting the voice data obtained by the voice recognition means into a text format.

[1821] "Generative AI methods" are technologies that summarize and organize text data, and then use generative AI models to analyze and make suggestions.

[1822] "Minutes-taking" is the process of recording the contents of a business meeting or negotiation so that it can be referenced later.

[1823] "Standard Answers to Questions" is a feature that automatically provides pre-set answers to frequently asked questions.

[1824] "Homework review" is the process of tracking and reviewing issues and next steps that arise from a business meeting or negotiation.

[1825] "Download materials" is a function that allows you to obtain necessary materials from the Internet or an internal network.

[1826] "Registration" is the process of storing and managing acquired data and information in a system.

[1827] A "generative AI model" is an artificial intelligence technology that generates new information and suggestions based on text data.

[1828] "Analyzing the content of business negotiations" is the process of analyzing the content discussed during business negotiations and extracting important points and next actions.

[1829] "Means for automatically generating next actions and proposals" refers to technology that automatically generates next actions and proposals based on the content of business negotiations.

[1830] The following system configuration will be described as an embodiment of the present invention.

[1831] System Configuration

[1832] This system includes speech recognition means, text conversion means, generative AI means, minutes creation, standard answers to questions, homework confirmation, document download and registration, analysis of business negotiation content using generative AI models, and means for automatically generating next actions and proposals. These means are realized as applications installed on mobile information terminals such as smartphones and tablets.

[1833] Hardware and software used

[1834] Hardware: Smartphone (iOS or Android), tablet

[1835] Software: Python, speech recognition library, transformers library

[1836] Processing flow

[1837] 1. Voice recording: The user starts the smartphone app and records the details of the business negotiation. The details of the business negotiation are captured as voice data using voice recognition.

[1838] 2. Speech recognition: The server uses a speech recognition library to convert the recorded voice data into text data.

[1839] 3. Texting: The audio data is converted into text format through textual processing.

[1840] 4. Data Summarization and Organizing: Summarize and organize text data using generative AI methods.

[1841] 5. Analyze the content of sales negotiations: Using generative AI models, analyze the content of sales negotiations and automatically generate next actions and proposals.

[1842] 6. Automation functions: Automate functions such as creating minutes, standard answers to questions, checking homework, downloading materials, and registering.

[1843] Specific examples

[1844] For example, if a customer says, "I'd like to discuss the introduction of a new product" during a sales negotiation, the user can launch an app on their smartphone and record the content of the negotiation. The recorded voice data is converted into text data using a speech recognition tool. The text data is then summarized and organized using a generative AI tool. The content of the sales negotiation is analyzed using a generative AI model, and a proposal such as "As a next step regarding the introduction of the new product, I suggest scheduling a demonstration" is automatically generated.

[1845] Prompt Sentence Examples

[1846] Negotiation details: We would like to discuss the introduction of a new product.

[1847] Next actions and suggestions:

[1848] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[1849] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1850] Step 1:

[1851] The user launches the smartphone app and records the contents of the business meeting. The input is the audio data of the business meeting, and the output is the recorded audio file. Specifically, the user presses the record button on the app and then presses the stop button when the business meeting is over.

[1852] Step 2:

[1853] The server converts the recorded audio file into text data using a speech recognition means. The input is the audio file and the output is text data. Specifically, the server analyzes the audio data using a speech recognition library and generates corresponding text.

[1854] Step 3:

[1855] The server uses generative AI tools to summarize and organize text data. The input is text data, and the output is summarized text data. Specifically, the server uses a generative AI model to extract important parts of the text and create a summary.

[1856] Step 4:

[1857] The server uses a generative AI model to analyze the content of the business negotiation and automatically generate the next action and proposal. The input is summarized text data, and the output is text data containing the next action and proposal. Specifically, the server inputs a prompt sentence into the generative AI model and obtains the generated text.

[1858] Step 5:

[1859] The server automates functions such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering. The input is text data of the business negotiations, and the output is automatically generated minutes, proposals, and confirmation items. Specifically, the server executes the processes corresponding to each function and generates and saves the necessary information.

[1860] In this way, users can automate the entire process from recording sales negotiations to analyzing them and proposing the next action, thereby making sales activities more efficient.

[1861] Example 2

[1862] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1863] In conventional sales assistant systems, the process of recording voice data, converting it to text, summarizing, and organizing it was often done manually, resulting in inefficiency. Furthermore, it was difficult to use highly accurate voice recognition technology to convert voice data into text, resulting in low quality text data. Furthermore, automation of tasks such as creating meeting minutes and standard responses to questions based on the generated text data was insufficient, placing a heavy burden on users.

[1864] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1865] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for transmitting recorded speech data from a terminal to the server, a means for the server to call a speech recognition service and convert the speech data into text data, a means for transmitting the converted text data to the terminal and displaying it, and a means for automating the creation of minutes, standard answers to questions, confirmation of homework items, downloading materials, registration, etc. This makes it possible to consistently and efficiently perform processes from recording speech data to converting it into text, summarizing, organizing, and even automating various tasks.

[1866] "Speech recognition means" refers to technology that analyzes voice data and converts it into corresponding text data.

[1867] The "means for converting text" refers to a technology for recording voice data obtained by a voice recognition means as text data.

[1868] "Generative AI methods" refers to artificial intelligence techniques for summarizing and organizing written data.

[1869] "Means for transmitting recorded voice data from a terminal to a server" refers to a technology for uploading voice data recorded on a terminal to a server via a network.

[1870] "Means for the server to call a voice recognition service and convert voice data into text data" refers to a technology in which the server uses an external voice recognition service to convert voice data into text data.

[1871] "Means for transmitting the converted text data to the terminal and displaying it" refers to a technology for transmitting the text data converted by the server to the terminal and displaying the text data on the terminal.

[1872] "Means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering" refers to technology that uses generated text data to automatically perform tasks such as creating minutes, answering questions, checking homework, downloading materials, and registering materials.

[1873] MODE FOR CARRYING OUT THE INVENTION

[1874] This invention is a system that efficiently converts voice data into text data and automates various tasks based on the text data. This system includes a voice recognition means, a text generation means, a generative AI means, a means for transmitting recorded voice data from a terminal to a server, a means for the server to call a voice recognition service and convert the voice data into text data, a means for transmitting the converted text data to a terminal and displaying it, and a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering.

[1875] Hardware and software used

[1876] Hardware: smartphones, PCs, servers

[1877] Software: Speech recognition APIs (e.g., general speech recognition services), generative AI models

[1878] Specific operation of the system

[1879] 1. A user records audio data using a recording app on a smartphone or computer. For example, to record the contents of a meeting, the user presses the record button on their smartphone to record the audio of the meeting.

[1880] 2. The device sends the recorded audio data to the server. Specifically, the recording app uploads the audio data to the server as an HTTP request.

[1881] 3. The server temporarily stores the received voice data and prepares it for the next process.

[1882] 4. The server calls the speech recognition service and converts the voice data into text data. Specifically, the server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text.

[1883] 5. The voice recognition service analyzes the received voice data and converts it into text data.

[1884] 6. The server receives the converted text data and sends it to the user's device. Specifically, the server returns the text data to the user's device as an HTTP response.

[1885] 7. The device displays the received text data to the user. For example, a smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[1886] Specific examples

[1887] For example, when a user records the contents of a meeting and converts it into text, the specific operation is as follows.

[1888] 1. The user launches the recording app on their smartphone and presses the record button to record the audio of the meeting.

[1889] 2. The device sends the recorded audio data to the server. For example, a recording app uploads a file called "MeetingRecording.wav" to the server as an HTTP request.

[1890] 3. The server receives the "Meeting Recording.wav" and temporarily stores it.

[1891] 4. The server calls the speech recognition service and sends "Meeting Recording.wav" to the API.

[1892] 5. The voice recognition service analyzes the voice data and generates text data such as "The contents of the meeting have been transcribed into text."

[1893] 6. The server receives the generated text data and sends it to the user's smartphone.

[1894] 7. The device displays the received text data on the app screen, allowing the user to check the contents of the meeting as text.

[1895] Prompt Sentence Examples

[1896] Examples of prompts to be input to a generative AI model include:

[1897] "Please convert the recorded audio data of the meeting into text data. The audio data can be obtained from the following URL. Please use a general speech recognition service for the conversion."

[1898] In this way, a system can be constructed that can efficiently convert recorded voice data into text data.

[1899] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1900] Step 1:

[1901] Users record audio data using a recording app on their smartphone or computer.

[1902] Input: Audio from meetings, business meetings, etc.

[1903] Output: Recorded audio data file (e.g., MeetingRecording.wav)

[1904] Specific operation: The user launches the recording app, presses the record button to record the audio of the meeting, and when the recording is finished, presses the save button to save the audio data.

[1905] Step 2:

[1906] The terminal transmits the audio data to the server after recording is completed.

[1907] Input: Recorded audio data file (e.g., MeetingRecording.wav)

[1908] Output: Audio data uploaded to the server

[1909] Specific operation: The recording app uploads the audio data to the server as an HTTP request. The user presses the send button to send the audio data to the server.

[1910] Step 3:

[1911] The server temporarily stores the received voice data in preparation for the next process.

[1912] Input: Uploaded audio data

[1913] Output: Audio data stored on the server

[1914] Specific operation: The server temporarily stores the received voice data in storage in preparation for calling the voice recognition service.

[1915] Step 4:

[1916] The server invokes a speech recognition service to convert the speech data into text data.

[1917] Input: Audio data stored on the server

[1918] Output: Audio data sent to the speech recognition service

[1919] Specific operation: The server sends the voice data to the speech recognition service's API and requests that the voice data be converted into text. For example, it creates an API request, attaches the voice data, and sends it.

[1920] Step 5:

[1921] The voice recognition service analyzes the received voice data and converts it into text data.

[1922] Input: Audio data sent to the speech recognition service

[1923] Output: Converted text data

[1924] What it does: A speech recognition service analyzes audio data and generates corresponding text data, for example, by converting speech to text using a speech recognition algorithm.

[1925] Step 6:

[1926] The server receives the converted text data and sends it to the user's terminal.

[1927] Input: Converted text data

[1928] Output: Text data sent to the user's device

[1929] Specific operation: The server receives the text data returned from the speech recognition service and returns it to the user's device as an HTTP response.

[1930] Step 7:

[1931] The terminal displays the received text data to the user.

[1932] Input: Text data sent from the server

[1933] Output: Text data displayed on the terminal

[1934] Specific operation: The smartphone app displays the text data on the screen, allowing the user to check the contents of the meeting as text.

[1935] (Application example 2)

[1936] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1937] While systems using conventional speech recognition technology were able to convert recorded audio data into text, they lacked the ability to generate and display subtitles for video content in real time. This made it difficult to immediately understand the audio content when watching video content, limiting the viewing experience, especially when watching content in a different language.

[1938] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for generating subtitles for video content in real time, and a display means for displaying subtitles. This makes it possible to recognize the voice of video content in real time and instantly display it as subtitles.

[1939] "Speech recognition means" is a technology that analyzes voice data and converts it into text data.

[1940] The "means for converting text" is a technology for converting voice data obtained by the voice recognition means into text format.

[1941] "Generative AI methods" are artificial intelligence techniques for summarizing and organizing text data.

[1942] "Means for generating subtitles for video content in real time" refers to technology that analyzes the audio data of video in real time and displays it as subtitles.

[1943] The "display means for displaying subtitles" is a technique for displaying generated subtitles superimposed on video frames.

[1944] The following system configuration will be described as an embodiment of the present invention.

[1945] System Configuration

[1946] This system includes a speech recognition means, a text generation means, a generative AI means, a means for generating subtitles in real time for video content, and a display means for displaying the subtitles.

[1947] Voice recognition means

[1948] The server uses a technology to analyze voice data and convert it into text data as a means of speech recognition, using a common speech recognition API (such as Google's speech recognition API or IBM's Watson).

[1949] Means of writing

[1950] The speech data obtained by the speech recognition means is converted into text format, and this converted text data is used for subsequent processing.

[1951] Generative AI means

[1952] Generative AI methods use artificial intelligence techniques to summarize and organize text data, including natural language processing (NLP) models. For example, generative AI models such as GPT-3 can be used.

[1953] A means of generating subtitles for video content in real time

[1954] The server uses technology to analyze the audio data of the video in real time and display it as subtitles. It generates subtitles in real time based on the text data obtained by the speech recognition means.

[1955] Display means for displaying subtitles

[1956] We use a technique to overlay the generated subtitles onto the video frames. Specifically, we use an image processing library such as OpenCV to add subtitles to the video frames.

[1957] Specific examples

[1958] For example, consider an application that generates English subtitles in real time while a user is watching a Japanese movie. This application is installed on a smartphone or head-mounted display, and while the user is watching the video, it converts the recorded audio data into text data in real time and displays it as subtitles.

[1959] Prompt Sentence Examples

[1960] "Create an app that generates English subtitles in real time while watching a Japanese movie. Use Google's Speech Recognition API for speech recognition and Google Translate API for translation. The subtitles should be displayed in the video frame."

[1961] In this way, the mode for carrying out the invention can be concretely shown.

[1962] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1963] Step 1:

[1964] The server acquires the audio data of the video content. Specifically, it extracts the audio track from the video file and saves it as audio data. The input is the video file, and the output is the audio data.

[1965] Step 2:

[1966] The server converts the voice data into text data using a voice recognition means. Specifically, it calls a voice recognition API, analyzes the voice data, and generates text data. The input is voice data, and the output is text data.

[1967] Step 3:

[1968] The server uses generative AI tools to summarize and organize text data. Specifically, it uses a natural language processing model to summarize the text data and extract the necessary information. The input is text data, and the output is summarized text data.

[1969] Step 4:

[1970] The server generates subtitles in real time based on the summarized text data. Specifically, it converts the generated text data into a subtitle format and adds a timestamp. The input is the summarized text data, and the output is the subtitle data.

[1971] Step 5:

[1972] The server overlays subtitle data onto video frames and displays them in real time using an image processing library. The input is subtitle data and video frames, and the output is video frames with added subtitles.

[1973] Step 6:

[1974] The user watches video frames with added subtitles. Specifically, they use a smartphone or a head-mounted display to watch videos with subtitles displayed in real time. The input is the video frames with added subtitles, and the output is the viewing experience.

[1975] Example 3

[1976] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1977] In conventional sales assistant systems, the transcription, summarization, and organization of voice data was often done manually, which was time-consuming and labor-intensive. Furthermore, tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering were not automated, resulting in inefficiency and a high likelihood of human error. Furthermore, there was a lack of a way to extract important information and execute automated tasks based on that information, which led to a decline in the accuracy and efficiency of the work.

[1978] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1979] In this invention, the server includes a speech recognition means, a means for converting speech data obtained by the speech recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summaries. This enables the automatic textualization, summarization, and organization of speech data, and automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering, improving the efficiency and accuracy of tasks.

[1980] "Speech recognition means" is a technology for analyzing voice data and converting it into text data.

[1981] The "text conversion means" is a technology that executes a process of converting the voice data obtained by the voice recognition means into text data.

[1982] "Generative AI methods" are technologies that use natural language processing technology to analyze text data, extract important information, and generate summaries based on that information.

[1983] "Minutes writing" is the process of recording the contents of a meeting or conference and creating a document that summarizes it.

[1984] "Standard answers to questions" is the process of providing predefined answers to questions posed by users.

[1985] "Homework review" is the process of reviewing action items and tasks decided upon in meetings and tracking their progress.

[1986] "Downloading materials" is the process of automatically obtaining required materials from the Internet or internal systems.

[1987] "Registration" is the process of storing and managing generated data and documents in an appropriate location.

[1988] "Means for analyzing text data and extracting important information" refers to technology that analyzes text data and extracts important keywords and phrases from it.

[1989] "Means for performing automated tasks based on generated summaries" refers to technology that automatically performs tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering information based on generated summaries.

[1990] MODE FOR CARRYING OUT THE INVENTION

[1991] This invention is a system including a speech recognition means, a text generation means, a generative AI means, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering, a means for analyzing text data and extracting important information, and a means for executing automated tasks based on the generated summary.

[1992] 1. Generating the system program

[1993] The server uses a speech recognition means to acquire voice data and generates a program that converts it into text data. Next, it uses a generative AI means to analyze the text data, extract important information, and generate a program that generates a summary based on that information. Furthermore, it generates a program that automates tasks such as creating minutes, providing standard answers to questions, checking homework, downloading materials, and registering based on the generated summary.

[1994] 2. Explanation of program processing

[1995] The server uses voice recognition software to convert the voice data provided by the user into text data. Specifically, voice recognition software such as Google Speech-to-Text can be used. The converted text data is sent to the server.

[1996] Next, the server uses natural language processing models such as BERT and Generative AI-3 as generative AI tools to analyze the text data. As a result of the analysis, important information is extracted and a summary is generated. This summary is then stored on the server.

[1997] Furthermore, the server performs the following automated tasks based on the generated summary:

[1998] Automatic generation of meeting minutes

[1999] Providing standard answers to questions

[2000] Checking homework

[2001] Download materials

[2002] Data registration

[2003] These tasks are automated through integration with meeting minutes-taking tools and CRM systems.

[2004] 3. Examples of concrete examples and prompts

[2005] Examples:

[2006] When a user creates minutes of a meeting, the audio data of the meeting is converted into text and the text data is sent to the server. The server then analyzes the text data using BERT to extract important topics and decisions. The server then generates a summary based on the extracted information and automatically generates the minutes.

[2007] Example prompt sentence:

[2008] "Please summarize the text data from the following meeting and prepare minutes:

[2009] 1. Meeting start and end times

[2010] 2. Participant's name

[2011] 3. Agenda and details

[2012] 4. Decisions and action items

[2013] By inputting this prompt sentence into the generative AI model, the server automatically generates a summary sentence and creates the minutes. The flow of the specific processing in the third embodiment will be described with reference to FIG.

[2014] Program processing flow

[2015] Step 1: Input audio data

[2016] Users record audio data of meetings and business negotiations, and the recorded audio data is saved on the device.

[2017] Specific behavior:

[2018] The user starts the recording application and records the audio of a meeting or business negotiation.

[2019] Once the recording is complete, the audio data is saved on the device.

[2020] Input: Audio data from meetings and business negotiations

[2021] Output: Audio data stored on the device

[2022] Step 2: Converting audio data to text

[2023] The device uses voice recognition software to convert the recorded voice data into text data, which is then sent to the server.

[2024] Specific behavior:

[2025] The device launches voice recognition software (e.g., Google Speech-to-Text).

[2026] The audio data is input into the software and converted into text data.

[2027] The converted text data is sent to the server.

[2028] Input: Audio data stored on the device

[2029] Output: Text data sent to the server

[2030] Step 3: Analyzing the text data

[2031] The server uses generative AI tools (e.g., BERT or Generative AI-3) to analyze the received text data. As a result of the analysis, important information is extracted.

[2032] Specific behavior:

[2033] The server activates the generative AI means.

[2034] Text data is input into a generative AI tool and analyzed.

[2035] Extract important keywords and phrases.

[2036] Input: Text data sent to the server

[2037] Output: Extracted important information

[2038] Step 4: Generate a summary

[2039] The server generates a summary based on the extracted important information, and the generated summary is stored on the server.

[2040] Specific behavior:

[2041] The server inputs the extracted information into a summary generation model.

[2042] A summary sentence is generated using a summary sentence generation model.

[2043] The generated summary is stored on the server.

[2044] Input: Extracted important information

[2045] Output: Generated summary

[2046] Step 5: Execute the automation

[2047] The server performs the following automated tasks based on the generated summary:

[2048] 1. Automatic generation of meeting minutes

[2049] 2. Providing standard answers to questions

[2050] 3. Check homework

[2051] 4. Download materials

[2052] 5. Data Registration

[2053] Specific behavior:

[2054] The server automatically generates minutes based on the summary and sends them to the user.

[2055] The server generates and provides standard answers to questions from users based on the summary text.

[2056] The server checks the homework items based on the summary and notifies the user.

[2057] The server automatically downloads the necessary materials and provides them to the user.

[2058] The server registers the generated data in the appropriate location (e.g., a CRM system).

[2059] Input: Generated summary

[2060] Output: Execution results of automated tasks

[2061] (Application example 3)

[2062] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2063] In conventional sales assistant systems and factory work efficiency systems, the texting, summarization, and organization of voice data was often done manually, resulting in problems with reduced work efficiency. Furthermore, they lacked functions such as summarizing and organizing work instructions, automatically recording work progress, providing standard responses to worker questions, and automatically downloading and registering necessary parts and materials, and as a result, work efficiency was not sufficiently improved. This increased the burden on workers and led to issues with reduced overall work efficiency.

[2064] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[2065] In this invention, the server includes a voice recognition means, a means for converting voice data obtained by the voice recognition means into text, a generative AI means for summarizing and organizing the textual data, a means for automating tasks such as creating minutes, providing standard answers to questions, checking homework, downloading and registering materials, a means for summarizing and organizing work instructions, a means for automatically recording work progress, a means for providing standard answers to questions from workers, and a means for automatically downloading and registering necessary parts and materials. This enables efficient textualization, summarization, and organization of voice data, and...

Claims

1. A means for converting the voice data obtained by the voice recognition means into text data; A means for extracting important information by analyzing the text data using a generative AI and generating a summary based on the extracted important information; A means for executing an automated task based on the generated summary sentence; A system including:

2. The automated tasks include creating minutes, providing standard answers to questions, checking homework items, downloading materials, and registering data. The system of claim 1 .

3. The voice recognition means starts an application when the user visits a predetermined location and records the contents of the conversation. The system of claim 1 .

Images