System

Abstract

To provide a system that can quantitatively grasp the mood and energy level of a conference and increase the effect of progress of the conference and collection of opinions from participants.SOLUTION: A system includes: means that acquires text data representing voice data of participants in a conference; means that extracts the emotion scores of words or phrases included in the text data, by using a prompt sentence instructing extracting the emotion scores of words or phrases included in the text data and a generative AI model; means that adds up or averages the extracted emotion scores to estimate the entire emotion score; and means that visualizes a change in the estimated emotion score.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In traditional meetings, the information obtained from participants' comments is not fully utilized, resulting in a lack of feedback on the progress of the meeting and decision-making.In addition, there is no way to grasp and visualize the mood and energy level of the meeting, which makes it difficult to effectively progress the meeting or gather participants' opinions. [Means for solving the problem]

[0005] This invention transcribes audio data and inputs it into generative AI to extract emotion scores for the words and phrases included, then sums or averages them to infer the overall emotion. Furthermore, it visualizes the mood and energy level of the meeting, allowing for more effective progress in the meeting and the collection of participants' opinions. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, speech data from participants in a conference is acquired and transcribed using speech recognition technology. This transcription can be performed using, for example, Google's (registered trademark) speech recognition API.

[0029] "Example 2"

[0030] The transcribed data is then fed into a generative AI, which is designed using a deep learning model, such as BERT or GPT-3.

[0031] "Example 3"

[0032] Generative AI extracts sentiment scores for words and phrases, sums or averages them, and infers the overall sentiment. This sentiment score can be extracted using a sentiment analysis tool such as VADER.

[0033] "Example 4"

[0034] Finally, the mood and energy level of the meeting can be visualized, for example, by displaying it as a graph on a dashboard. This can help improve the effectiveness of the meeting and the gathering of participants' opinions.

[0035] The processing flow of each embodiment will be described below.

[0036] "Example 1"

[0037] Step 1: The speech of the meeting participants is acquired as audio data using a microphone or the like.

[0038] Step 2: The acquired voice data is transcribed using speech recognition technology. This transcription can be performed using, for example, Google's speech recognition API.

[0039] "Example 2"

[0040] Step 1: Input the transcribed data from Example 1 into the generative AI.

[0041] Step 2: Generative AI is designed using deep learning models, such as BERT or GPT-3.

[0042] "Example 3"

[0043] Step 1: Extract sentiment scores for the words and phrases contained in the data input into the generative AI in Example 2.

[0044] Step 2: The extracted emotion scores are summed or averaged to estimate the overall emotion. This emotion score can be extracted using an emotion analysis tool such as VADER. "Example 4"

[0045] Step 1: Visualize the mood and energy level of the meeting using the overall emotions inferred in Example 3.

[0046] Step 2: This visualization can be done, for example, by displaying it as a graph on a dashboard, which will help improve the effectiveness of the meeting and the gathering of participants' opinions.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] In conventional meetings, it was difficult to accurately record what participants said and share it in real time. Furthermore, there was a lack of efficient ways to proceed with the meeting and gather participants' opinions. This led to a decrease in meeting efficiency and the risk of important information being overlooked.

[0050] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server, means for the server to transcribe the voice data using voice recognition technology, means for storing the transcribed data in a database, means for transmitting the stored data to a terminal and displaying it, and means for efficiently progressing the meeting and collecting opinions of participants. This makes it possible to accurately record and share the contents of statements made in the meeting in real time. Furthermore, since the meeting can be progressed and opinions can be efficiently collected, the efficiency of the meeting is improved and important information can be prevented from being overlooked.

[0051] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[0052] The "means of collection" refers to a method of acquiring voice data using a microphone or voice input device.

[0053] The "means of transmission" refers to a method for transferring collected voice data to a server via a network.

[0054] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0055] "Transcribing means" refers to a method of converting audio data into text data using voice recognition technology.

[0056] "Database" means a digital storage system for storing transcribed text data.

[0057] "Means for storage" refers to the method for storing the transcribed text data in a database.

[0058] "Display means" refers to a method for displaying text data on the terminal screen.

[0059] "Proceeding a meeting" is the process of advancing the discussion in line with the meeting agenda.

[0060] "Convergence" is the process of summarizing the opinions of meeting participants and reaching a common conclusion.

[0061] "Means for improving effectiveness" are methods and tools for efficiently conducting meetings and gathering opinions.

[0062] The present invention provides a system for transcribing speeches made by participants in a meeting in real time, thereby improving the efficiency of the progress of the meeting and the collection of opinions. Specific embodiments of this system will be described below.

[0063] Hardware and software used

[0064] Hardware

[0065] Microphone: Used to collect the speech of meeting participants.

[0066] Terminal: A device such as a personal computer, tablet, or smartphone that collects and transmits voice data to a server.

[0067] Server: A computer system that receives audio data and transcribes it using speech recognition technology.

[0068] software

[0069] Audio collection application: An application that is installed on a device to collect audio data and send it to a server.

[0070] Speech recognition API: An API for converting voice data into text data. Specifically, it uses common voice recognition technology.

[0071] Database: A digital storage system for storing transcribed text data.

[0072] System Operation

[0073] server

[0074] The server receives the voice data sent from the device. The received voice data is sent to a voice recognition API and converted into text data. The converted text data is saved in a database. The saved text data is then sent to the device and displayed in real time.

[0075] Terminal

[0076] The device collects audio data via a microphone when participants speak, and sends the collected audio data to a server via a voice collection application. The text data returned from the server is displayed on the device screen, allowing users to view it in real time.

[0077] User

[0078] Users speak during the meeting. Their speech is collected as audio data through the device's microphone. Users can then see their speech being transcribed on the device's screen.

[0079] Specific examples

[0080] For example, if a user says "Let's discuss the next item on the agenda" during a meeting, the device's microphone will collect this voice and send it to the server via a voice collection application. The server will then use a speech recognition API to convert this voice data into text data, generating the text "Let's discuss the next item on the agenda." This text data will then be stored in a database and sent to the device. Finally, the text "Let's discuss the next item on the agenda" will be displayed on the device screen, allowing the user to view it in real time.

[0081] Prompt Sentence Examples

[0082] "Design a system that transcribes meeting participants' speech in real time. Include a process for collecting audio data, converting it to text using speech recognition technology, and displaying it on a device screen."

[0083] This system is very useful for improving meeting efficiency and automating the creation of meeting minutes.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1: Collecting audio data

[0086] The device collects audio data through a microphone when a participant speaks. Specifically, the device's audio collection application digitally captures the audio input from the microphone and saves it as an audio file. The input is the participant's speech, and the output is a digital audio file.

[0087] Step 2: Sending audio data

[0088] The device sends the collected voice data to a server. Specifically, the voice collection application uploads the voice file to the server via the network. The input is a digital voice file, and the output is the voice data sent to the server.

[0089] Step 3: Receiving audio data

[0090] The server receives the voice data sent from the terminal. Specifically, the server's receiving module receives the voice file sent over the network and stores it in the server. The input is the voice data sent from the terminal, and the output is the voice data stored in the server.

[0091] Step 4: Transcribe the audio data

[0092] The server transcribes the received voice data using speech recognition technology. Specifically, the server calls a speech recognition API and converts the voice data into text data. The input is the voice data stored on the server, and the output is the text data returned by the speech recognition API.

[0093] Step 5: Save the text data

[0094] The server saves the transcribed text data in a database. Specifically, the server's database module stores the text data in the database. The input is the text data returned from the speech recognition API, and the output is the text data saved in the database.

[0095] Step 6: Send text data

[0096] The server transmits the stored text data to the terminal. Specifically, the server's transmission module retrieves the text data from the database and transmits it to the terminal via the network. The input is the text data stored in the database, and the output is the text data transmitted to the terminal.

[0097] Step 7: Displaying Text Data

[0098] The terminal receives the text data sent from the server and displays it on the screen. Specifically, the display application of the terminal receives the text data via the network and displays it on the user interface. The input is the text data sent from the server, and the output is the text data displayed on the terminal screen.

[0099] Through this series of processes, the remarks of meeting participants are transcribed in real time and displayed on the device screen, allowing for efficient progress in the meeting and the collection of opinions.

[0100] (Application example 1)

[0101] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Furthermore, when workers give voice instructions to machines at work sites such as factories, there is no system that can accurately recognize those instructions and perform the appropriate operation. This creates the problem of insufficient progress in meeting efficiency and automation of on-site work.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0104] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included. This makes it possible to visualize the mood and energy level of a meeting, and to improve the effectiveness of meeting progress and gathering participants' opinions. Furthermore, by including a means for analyzing voice data and controlling machine operation based on specific keywords, workers in factories and other workplaces can issue voice instructions to machines, which then perform appropriate operations based on those instructions.

[0105] "Audio data" refers to digital data of audio collected at meetings, work sites, etc.

[0106] "Transcription" is the process of converting audio data into text data.

[0107] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new data and information from input data.

[0108] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0109] "Meeting mood" refers to the emotional atmosphere of all participants during the meeting.

[0110] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or at a workplace.

[0111] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[0112] "Controlling the operation of a machine" means instructing a machine to perform a specific operation based on voice data or text data.

[0113] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0114] A "deep learning model" is a type of artificial intelligence that uses multi-layered neural networks to analyze and learn from data.

[0115] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, and means for analyzing the voice data and controlling the operation of a machine based on specific keywords.

[0116] The server acquires the audio data and transcribes it using speech recognition technology. For example, Google's speech recognition API is used as the speech recognition technology. The transcribed text data is then input into a generative AI system, which extracts emotion scores for the words and phrases contained within. The emotion scores are then summed or averaged to estimate the overall emotion. This makes it possible to visualize the mood and energy level of the meeting, thereby improving the effectiveness of meeting progress and gathering participant opinions.

[0117] Furthermore, the server analyzes the voice data and controls the machine's operation based on specific keywords. For example, if a worker in a factory says, "Please proceed to the next process," the robot will proceed to the next process. In this case, speech recognition technology is used to convert the voice into text, and the robot's operation is controlled based on that text.

[0118] The hardware used includes a microphone and a factory robot, while the software uses Python, the speech_recognition library, and Google's speech recognition API.

[0119] For example, in a factory, when a worker says "Please proceed to the next step," the robot will proceed to the next step. The following is an example of a prompt:

[0120] When you say "Please proceed to the next step," create a program that will make the robot proceed to the next step. Use Google's speech recognition API to convert speech into text and control the robot's actions based on that text.

[0121] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0122] Step 1:

[0123] The server captures voice data through a microphone: the input is the user's voice, and the output is digital voice data, which is used in subsequent processing steps.

[0124] Step 2:

[0125] The server transcribes the acquired voice data using speech recognition technology. Specifically, it converts the voice data into text data using Google's speech recognition API. The input is the voice data acquired in step 1, and the output is text data.

[0126] Step 3:

[0127] The server inputs the transcribed text data into a generative AI and extracts sentiment scores for the words and phrases contained within. The input is the text data obtained in step 2, and the output is the sentiment score for each word or phrase. The generative AI calculates the sentiment scores using a deep learning model.

[0128] Step 4:

[0129] The server sums or averages the extracted emotion scores to infer the overall emotion. The input is the emotion scores obtained in step 3, and the output is an overall emotion score. This overall emotion score is an indicator of the mood and energy level of the meeting.

[0130] Step 5:

[0131] The server visualizes the overall emotion score. The input is the overall emotion score obtained in step 4, and the output is a visually displayed emotion score. This helps to improve the progress of the meeting and the effectiveness of gathering participants' opinions.

[0132] Step 6:

[0133] The server analyzes the voice data and controls the machine's operation based on specific keywords. The input is the text data obtained in step 2, and the output is instructions for the machine's operation. For example, if the keyword "Please proceed to the next step" is included, the robot will proceed to the next step.

[0134] Step 7:

[0135] The server executes the machine's operations. The input is the operation instructions obtained in step 6, and the output is the actual machine operation. Specifically, the server performs operations such as the robot proceeding to the next process or installing parts.

[0136] Example 2

[0137] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] Conventional conference systems are required not only to transcribe audio data but also to visualize the mood and energy level of the meeting and effectively gather opinions from participants. However, there are few systems that provide these functions in an integrated manner, and in particular, they lack advanced analysis and output using generative AI models. For this reason, a new system is needed to improve the progress of meetings and the effectiveness of gathering opinions from participants.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0140] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the effectiveness of meeting progress and participant opinion gathering, means for preprocessing the transcription data, means for generating prompt sentences for the generative AI model, means for acquiring generation results from the generative AI model, means for post-processing the generation results, and means for outputting the generation results, thereby enabling the meeting to be progressed effectively and participant opinions to be gathered.

[0141] "Audio data" refers to data in which audio is recorded in digital format.

[0142] "Transcription" is the process of converting audio data into text data.

[0143] A "generative AI model" is an artificial intelligence model that generates text data using deep learning technology.

[0144] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0145] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[0146] "Energy level" refers to the level of activity and enthusiasm of participants in a meeting or conversation.

[0147] "Preprocessing" is the initial process of preparing data in a format suitable for analysis and processing.

[0148] A "prompt sentence" is an input sentence that instructs a generative AI model on a specific task.

[0149] "Post-processing" refers to the process of preparing the generated data in a final output format.

[0150] "Output" refers to providing processed data to a user or other system.

[0151] MODE FOR CARRYING OUT THE INVENTION

[0152] This invention relates to a system that transcribes voice data and analyzes and outputs the results using a generative AI model. A specific embodiment of this system is described below.

[0153] System configuration

[0154] The server system consists of the following hardware and software:

[0155] Hardware:

[0156] High-performance processor

[0157] Large capacity memory

[0158] High-speed storage devices

[0159] software:

[0160] Voice recognition technology (e.g., Google Speech-to-Text API)

[0161] Generative AI models (e.g., BERT, GPT-3)

[0162] Scripts for data pre- and post-processing

[0163] Program processing

[0164] The server receives the voice data sent from the user or device and performs the following processes:

[0165] 1. Transcription of audio data:

[0166] The server converts the audio data into text using speech recognition technology, for example, by using the Google Speech-to-Text API to transcribe the audio file.

[0167] 2. Data preprocessing:

[0168] The server preprocesses the transcription data and removes unnecessary noise and special characters. For example, it converts the transcription data "Hello, this is a test" into "Hello, this is a test."

[0169] 3. Generative AI model selection:

[0170] The server selects the generative AI model to use, for example, a deep learning model such as BERT or GPT-3.

[0171] 4. Prompt generation:

[0172] The server generates prompts to be input to the generative AI model. Prompts instruct the model on specific tasks. For example, it generates a prompt such as "Please summarize the following sentence: 'Hello, this is a test.'"

[0173] 5. Inputting data into generative AI models:

[0174] The server inputs the generated prompt and preprocessed data into a generative AI model. For example, the server inputs the prompt "Please summarize the following sentence: 'Hello, this is a test'" into the BERT model.

[0175] 6. Obtaining the generated results:

[0176] The server receives the output from the generative AI model, which generates an appropriate output based on the input prompt. For example, the server receives the output "Hello, this is the test result" from the BERT model.

[0177] 7. Post-processing of results:

[0178] The server post-processes the generated output and formats it as needed, for example converting the output to JSON format and adding necessary metadata.

[0179] 8. Result output:

[0180] The server outputs the final results to the user or to a terminal, where the user can review the generated results and perform further processing as needed. For example, the server may display the generated results on the user's dashboard and allow the user to download the results.

[0181] Specific examples

[0182] Example prompt sentence:

[0183] "Summarize the following sentence: 'Hello, this is a test.'"

[0184] "Please translate the following sentence: 'Hello, this is a test.'"

[0185] This allows users to use generative AI models to perform a variety of tasks, such as summarizing and translating transcripts.

[0186] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] The server receives audio data sent from the user or device. The audio data is provided in an audio file format (e.g., WAV, MP3). The server inputs this audio data into transcription software (e.g., Google Speech-to-Text API) and outputs text data. Specifically, the server sends the audio file to the API and obtains the returned text data.

[0189] Step 2:

[0190] The server preprocesses the received text data. This preprocessing removes unnecessary noise and special characters and formats the data. The input is transcribed text data, and the output is formatted text data. Specifically, it uses regular expressions to remove special characters and converts the data into the required format.

[0191] Step 3:

[0192] The server selects the generative AI model to use. For example, it initializes a deep learning model such as BERT or GPT-3. The input is formatted text data, and the output is the model's initialized state. Specifically, it loads the model library and sets the necessary parameters.

[0193] Step 4:

[0194] The server generates prompt sentences to be input to the generative AI model. Prompt sentences instruct the model on specific tasks. The input is formatted text data, and the output is the generated prompt sentence. The specific operation is to create the prompt sentence using a template.

[0195] Step 5:

[0196] The server inputs the generated prompt sentence and preprocessed data into the generative AI model. The input is the prompt sentence and formatted text data, and the output is the analysis result from the generative AI model. Specifically, the server inputs the prompt sentence and text data into the model and obtains the analysis result.

[0197] Step 6:

[0198] The server receives the output from the generative AI model. The model generates appropriate output based on the input prompt. The input is the prompt and formatted text data, and the output is the generated text data. The specific operation is to receive the results returned from the model.

[0199] Step 7:

[0200] The server then post-processes the generated output, formatting it as needed and adding additional information. The input is the generated text data, and the output is the final formatted data, specifically converting it to JSON and adding any necessary metadata.

[0201] Step 8:

[0202] The server outputs the final results to the user or to a device. The user can review the generated results and perform further processing if necessary. The input is the final formatted data, and the output is the results provided to the user. Specific operations include displaying the results on the user's dashboard and providing a download link.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] In today's business environment, improving meeting efficiency and gathering participant opinions are important issues. However, conventional systems have difficulty visualizing the mood and energy level of a meeting, and they have not yet realized the automatic generation of content based on the meeting content. For this reason, there is a need for a system that can improve the effectiveness of meeting progress and gathering participant opinions, as well as automatically generate content based on the meeting content.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores for included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for automatically generating content based on the voice data, and a means for displaying the generated content. This not only makes it possible to improve the efficiency of meetings and effectively gather participants' opinions, but also enables the automatic generation of content based on the meeting content.

[0208] "Audio data" refers to data in which audio is recorded in digital format.

[0209] "Transcription" is the process of converting audio data into text data.

[0210] "Generative AI" is artificial intelligence that uses deep learning models to generate new text and content.

[0211] An "emotion score" is a numerical representation of the intensity and type of emotion associated with a word or phrase in a text.

[0212] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[0213] "Energy level" is an indicator of the level of activity and enthusiasm of participants in a meeting or conversation.

[0214] "Visualization" is the process of visually displaying data or information.

[0215] "Opinion gathering" is the process of summarizing the opinions of multiple participants to arrive at a single conclusion or direction.

[0216] "Automatic content generation" is the process of automatically generating text and media based on input data.

[0217] "Display" is the process of visually presenting generated content to a user.

[0218] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, means for automatically generating content based on the voice data, and means for displaying the generated content.

[0219] System configuration

[0220] Hardware

[0221] Smartphone: Used for voice input and display.

[0222] Server: Used to process generative AI models.

[0223] software

[0224] Speech recognition technology: Used to transcribe voice data.

[0225] Deep learning models (BERT and GPT-3): Used as generative AI.

[0226] Sentiment analysis tools: Used to extract sentiment scores from text data.

[0227] Data visualization tools: Used to visualize the mood and energy level of a meeting.

[0228] Data processing and calculation

[0229] 1. Speech recognition: The smartphone's microphone is used to collect the user's voice, which is then transcribed using speech recognition technology.

[0230] 2. Extract sentiment scores: The transcribed text data is fed into a sentiment analysis tool to extract sentiment scores for words and phrases.

[0231] 3. Sentiment inference: The extracted sentiment scores are summed or averaged to infer the overall sentiment.

[0232] 4. Content generation using generative AI: Transcribed text data is fed into a generative AI model to automatically generate content such as blog posts and news articles.

[0233] 5. Displaying content: The generated content is displayed on the smartphone screen.

[0234] Specific examples

[0235] When a user speaks into their smartphone, "I want to write a blog post about today's weather," the system transcribes the speech and generates text data that reads, "I want to write a blog post about today's weather." This text data is then fed into a sentiment analysis tool to extract a sentiment score. The text data is then fed into a generative AI model, which automatically generates a blog post like the one below.

[0236] "Today's forecast is for sunny skies with occasional clouds. We will have a clear blue sky in the morning, with clouds gathering in the afternoon, but there is no need to worry about rain. The temperature is expected to be a comfortable day with a maximum of 25 degrees and a minimum of 15 degrees."

[0237] Prompt Sentence Examples

[0238] If the user says "I want to write a blog post about today's weather," the prompt would be:

[0239] "I want to write a blog post about today's weather."

[0240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0241] Step 1:

[0242] The user speaks into the smartphone.

[0243] Input: User's voice data

[0244] Output: Audio data

[0245] Specific action: The user speaks into the smartphone microphone, "I want to write a blog post about today's weather."

[0246] Step 2:

[0247] The smartphone uses voice recognition technology to transcribe the audio data.

[0248] Input: Audio data

[0249] Output: Transcribed text data

[0250] How it works: The smartphone's voice recognition software analyzes the voice data and generates text data such as, "I want to write a blog post about today's weather."

[0251] Step 3:

[0252] The server inputs the transcribed text data into a sentiment analysis tool to extract a sentiment score.

[0253] Input: Transcribed text data

[0254] Output: Sentiment score

[0255] What it does: The server sends the text data to a sentiment analysis tool, which extracts a numerical sentiment score for each word or phrase.

[0256] Step 4:

[0257] The server sums or averages the emotion scores to infer the overall emotion.

[0258] Input: Sentiment score

[0259] Output: Overall sentiment score

[0260] Specific operation: The server sums or averages the extracted emotion scores to calculate the overall emotional state as a number.

[0261] Step 5:

[0262] The server inputs the transcribed text data into a generative AI model to automatically generate content.

[0263] Input: Transcribed text data

[0264] Output: Generated content

[0265] How it works: The server sends text data to a generative AI model, which then generates content such as blog posts and news articles.

[0266] Step 6:

[0267] The smartphone displays the generated content.

[0268] Input: Generated content

[0269] Output: The displayed content

[0270] Specific operation: The generated blog post is displayed on the smartphone screen and the user can view it.

[0271] Example 3

[0272] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] Conventional meeting systems make it difficult to quantitatively grasp the mood and energy level of a meeting, and lack specific indicators for improving meeting progress and gathering participants' opinions. Furthermore, sentiment analysis requires a lot of manual work, making efficient operation difficult.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0275] In this invention, the server includes: means for acquiring text data representing the voice data of each participant in the conference; means for extracting emotion scores for words or phrases included in the text data using a prompt statement and a generative AI model; means for estimating the overall emotion score by summing or averaging the extracted emotion scores; and means for visualizing changes in the estimated emotion scores. This allows for a quantitative understanding of the mood and energy level of the conference, thereby improving the progress of the conference and the effectiveness of gathering participants' opinions.

[0276] "Audio data" refers to data in which audio is recorded in digital format.

[0277] "Transcription" is the process of converting audio data into text data.

[0278] A "generative AI model" is an artificial intelligence that has the ability to generate new data based on given data.

[0279] An "emotion score" is a numerical value that indicates the intensity of emotion calculated for a word or phrase in text data.

[0280] A "sentiment analysis tool" is software or algorithms for extracting sentiment scores from text data.

[0281] A "dashboard" is an interface for visually displaying data.

[0282] "Meeting mood" refers to the overall atmosphere and emotional state felt during the meeting.

[0283] "Energy level" is an indicator of the liveliness and positivity of participants during a meeting.

[0284] A "user" is a person or organization that operates the system and inputs data.

[0285] A "server" is a computer system that processes and stores data.

[0286] "Text data" is digital data that includes character information.

[0287] "Summing or averaging" is the process of adding together or calculating the average of several numbers.

[0288] This invention is a system for quantitatively understanding the mood and energy level of a meeting, improving the effectiveness of meeting progress and gathering participants' opinions. The system transcribes audio data, extracts emotion scores using generative artificial intelligence, sums or averages the scores to estimate the overall emotion, and visualizes them on a dashboard.

[0289] Hardware and software used

[0290] Hardware: Servers (e.g., cloud computing services)

[0291] Software: Sentiment analysis tools (e.g., VADER), dashboard display tools (e.g., Grafana)

[0292] Program processing explanation

[0293] Users input text data, such as meeting minutes or chat logs, into the system. Input is done through web forms or dedicated applications. The server receives the text data sent by the user and temporarily stores it. The server then uses VADER, a sentiment analysis tool, to extract sentiment scores for each word or phrase in the received text data. VADER calculates sentiment scores as positive, negative, or neutral.

[0294] The server then sums or averages the sentiment scores for each extracted word or phrase to calculate an overall sentiment score, which quantitatively evaluates the sentiment of the entire text. Finally, the server displays the changes in the calculated overall sentiment score on a dashboard. The dashboard, built using a tool such as Grafana, visually shows the mood and energy level of the meeting.

[0295] Specific examples

[0296] An example of a prompt that a user might enter into a generative AI model is:

[0297] After inputting text data representing the voice data of each participant in the meeting, the following question is asked: "This text data is a meeting transcript. Please extract the sentiment scores of words or phrases contained in the meeting transcript."

[0298] Example of meeting minutes:

[0299] Participant A: "This project is going well."

[0300] Participant B: "There are some challenges, but we have found solutions."

[0301] Participant C: “Overall, there has been a lot of positive feedback.”

[0302] When this prompt sentence is input into the generative AI model, the emotion score of the words or phrases contained in each utterance is extracted. The server calculates the total emotion score for each utterance by summing or averaging the emotion scores of the words or phrases contained in that utterance. The server then estimates the change in the total emotion score calculated for each utterance as a change in the overall mood and displays it as a graph on the dashboard. For example, if there are many positive emotions, the graph rises, and if there are many negative emotions, the graph falls. This allows the progress of the meeting and the effectiveness of gathering participants' opinions to be visually confirmed. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[0303] Step 1:

[0304] The user inputs text data.

[0305] Users enter text data such as meeting minutes and chat logs into the system through web forms or dedicated applications, and the entered text data is sent to the server.

[0306] Input: Text data such as meeting minutes and chat logs

[0307] Output: Text data sent to the server

[0308] Step 2:

[0309] The server receives the text data.

[0310] The server receives the text data sent by the user and temporarily stores it in a database, for example, a MySQL (registered trademark) database.

[0311] Input: Text data sent by the user

[0312] Output: Text data stored in the database

[0313] Step 3:

[0314] The server extracts the sentiment score using a generative AI model or sentiment analysis tool.

[0315] The server passes the stored text data to a generative AI model or sentiment analysis tool (e.g., VADER) to calculate a sentiment score for each word or phrase. VADER calculates a sentiment score as positive, negative, or neutral.

[0316] Input: Text data stored in a database

[0317] Output: Sentiment score for each word or phrase

[0318] Step 4:

[0319] The server sums or averages the sentiment scores for each utterance.

[0320] For each utterance, the server calculates an overall sentiment score by summing or averaging the sentiment scores of each word or phrase extracted from that utterance. For example, if the positive score is 3 and the negative score is 1, the overall sentiment score is 2.

[0321] Input: Sentiment scores for each word or phrase

[0322] Output: Overall sentiment score

[0323] Step 5:

[0324] The server displays the changes in the estimated emotion scores on a dashboard.

[0325] The server displays the calculated changes in the overall emotion score on a dashboard (e.g., Grafana). Users can visually check the mood and energy level of the meeting through the dashboard. For example, if there is a lot of positive emotion, the graph will rise, and if there is a lot of negative emotion, the graph will fall.

[0326] Input: Overall sentiment score

[0327] Output: A graph of the sentiment scores displayed in a dashboard The server may then perform processing to detect a peak in the change in the estimated emotion score, and when a peak is detected, may notify that it is time to summarize the conference. Alternatively, the server may estimate and present the progress of the meeting using a generative AI model and a prompt sentence that instructs the server to estimate the progress of the meeting based on changes in the estimated emotion score.

[0328] (Application example 3)

[0329] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] There is a lack of means to monitor the emotional state of workers in factories in real time to improve work efficiency and safety. Conventional methods make it difficult to properly grasp workers' moods and energy levels, making it difficult to optimize the work environment. Therefore, there is a need for a system that visualizes workers' emotional state and allows managers to respond quickly.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0332] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included in the data, which allows visualization of the mood and energy levels of workers and enables optimization of the work environment.

[0333] "Audio data" refers to data in which audio is recorded in digital format.

[0334] "Transcription" is the process of converting audio data into text form.

[0335] "Generative AI" is an artificial intelligence technology that generates new data based on input data.

[0336] An "emotion score" is a numerical representation of the intensity of emotion associated with a particular word or phrase.

[0337] "Mood" refers to the emotional state of a worker at a particular time.

[0338] "Energy level" is an indicator of a worker's vitality and fatigue level.

[0339] "Visualization" is the process of displaying data in a visual format such as a graph or chart.

[0340] "Work environment optimization" refers to adjusting the work environment to improve worker efficiency and safety.

[0341] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI system, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy levels of workers, and means for optimizing the work environment.

[0342] System configuration

[0343] This system uses the following hardware and software:

[0344] Hardware: Smart glasses (e.g., Google Glass®)

[0345] Software: Python, VADER Sentiment Analysis, Matplotlib

[0346] Program processing explanation

[0347] The server first acquires the audio data and transcribes it using speech recognition technology. The transcribed text data is then fed into a generative AI system, which extracts sentiment scores for the words and phrases contained within. This sentiment score is derived using VADER Sentiment Analysis. The extracted sentiment scores are then summed or averaged to estimate the overall sentiment.

[0348] The server then visualizes the workers' moods and energy levels based on the inferred emotional data, using Matplotlib to generate graphs and charts, allowing managers to understand the workers' emotional state in real time and optimize the work environment.

[0349] Specific examples

[0350] For example, if a worker says, "I'm very tired today," the voice data is sent through the smart glasses to a server. The server uses speech recognition technology to convert this speech into text and feeds it into a generative AI system. The generative AI then uses VADER Sentiment Analysis to extract an emotion score, which then visualizes the worker's mood and energy level.

[0351] Example prompt sentence:

[0352] Calculate the emotion score when a worker says, "I'm very tired today," and display it in a graph.

[0353] By inputting this prompt into a generative AI model, an emotion score based on the worker's utterance is calculated and visualized, allowing managers to quickly understand the worker's emotional state and take appropriate action.

[0354] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0355] Step 1:

[0356] The server receives voice data from the smart glasses. The input is the worker's speech, and the output is digital voice data. The server stores this voice data.

[0357] Step 2:

[0358] The server converts the voice data into text data using speech recognition technology. The input is the voice data, and the output is the transcribed text data. Specifically, the server uses speech recognition software (e.g., Google Speech-to-Text API) to analyze the voice and generate the corresponding text.

[0359] Step 3:

[0360] The server inputs text data into the generative AI. The input is the transcribed text data, and the output is an emotion score. Specifically, the emotion score of the text data is calculated using VADER Sentiment Analysis.

[0361] Step 4:

[0362] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. Specifically, each emotion score is aggregated as a numerical value, and the sum or average is calculated.

[0363] Step 5:

[0364] The server visualizes the mood and energy levels of workers based on the estimated overall emotion score. The input is the overall emotion score, and the output is a graph or chart. Specifically, Matplotlib is used to visually display the emotion score.

[0365] Step 6:

[0366] The server provides visualized data to managers to optimize the work environment. The input is graphs and charts, and the output is notifications and reports to managers. Specifically, graphs are displayed on a dashboard, allowing managers to understand the emotional state of workers in real time.

[0367] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0368] "Example 1"

[0369] In the first embodiment, a voice recognition technology is used as a means for transcribing voice data. Specifically, the voice data is converted into text data using a voice recognition API such as Google's.

[0370] "Example 2"

[0371] In Example 2, a deep learning model is used as a means for inputting data into generative AI. Specifically, text data is analyzed using models such as BERT and GPT-3.

[0372] "Example 3"

[0373] In the third embodiment, an emotion engine is used as a means for extracting emotion scores of included words and phrases. Specifically, emotion scores are extracted from text data using an emotion analysis tool such as VADER.

[0374] "Example 4"

[0375] In the fourth embodiment, an emotion engine is used as a means for inferring the overall emotion by summing or averaging the extracted emotion scores. Specifically, the overall emotion is inferred by summing or averaging the extracted emotion scores.

[0376] "Example 5"

[0377] In the fifth embodiment, a dashboard is used as a means for visualizing the mood and energy level of the meeting. Specifically, the mood and energy level of the meeting are displayed as a graph on the dashboard using the estimated overall emotions.

[0378] "Example 6"

[0379] In the sixth embodiment, an emotion engine is used as a means for improving the efficiency of proceeding with a meeting and gathering opinions of participants. Specifically, emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, thereby improving the efficiency of proceeding with the meeting and gathering opinions.

[0380] The processing flow of each embodiment will be described below.

[0381] "Example 1"

[0382] Step 1: Step 1: Collect audio data during the meeting.

[0383] Step 2: Step 2: Convert the collected voice data into text data using Google's voice recognition API or similar.

[0384] "Example 2"

[0385] Step 1: Step 1: Convert the audio data into text data.

[0386] Step 2: Step 2: The converted text data is fed into a deep learning model such as BERT or GPT-3 and analyzed.

[0387] "Example 3"

[0388] Step 1: Step 1: Analyze the text data.

[0389] Step 2: Step 2: Extract sentiment scores from the analyzed text data using a sentiment analysis tool such as VADER.

[0390] "Example 4"

[0391] Step 1: Step 1: Extract sentiment scores from text data.

[0392] Step 2: Step 2: Sum or average the extracted emotion scores to infer the overall emotion.

[0393] "Example 5"

[0394] Step 1:Step 1: Guess the overall emotion.

[0395] Step 2: Step 2: Using the estimated overall sentiment, the mood and energy level of the meeting are displayed as a graph on a dashboard.

[0396] "Example 6"

[0397] Step 1:Step 1:Visualize the mood and energy level of the meeting.

[0398] Step 2: Step 2: The emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, improving the efficiency of meeting progress and opinion gathering.

[0399] Example 1

[0400] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Conventional meeting systems have difficulty accurately recording participants' comments and grasping the mood and energy level of the meeting in real time. They also lack the means to effectively manage the meeting and gather participants' opinions. This leads to problems such as reduced meeting efficiency and impaired decision-making quality.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0403] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data using voice recognition technology, means for inputting the converted text data into generative artificial intelligence, means for extracting emotion scores for the words and phrases included, means for estimating the overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of the meeting, and means for improving the progress of the meeting and the collection of participants' opinions. This makes it possible to accurately record comments made in the meeting and grasp the mood and energy level of the meeting in real time. Furthermore, it is possible to effectively progress the meeting and collect opinions, thereby improving the efficiency of the meeting and the quality of decision-making.

[0404] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[0405] "Voice recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[0406] "Character data" refers to data in text format that has been converted using voice recognition technology.

[0407] "Generative AI" is an AI technology that uses deep learning models to generate new information and predictions from input data.

[0408] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0409] "Mood" refers to the overall atmosphere and emotional state of the meeting.

[0410] "Energy level" is an indicator of the liveliness of a meeting and the energy of the participants.

[0411] "Visualization" is a technique that makes data easier to understand by visually displaying it.

[0412] "Convergence" is the process of summarizing the opinions of meeting participants and arriving at a common conclusion or direction.

[0413] "Effectiveness" is the optimization of processes and means to achieve a specific purpose.

[0414] MODE FOR CARRYING OUT THE INVENTION

[0415] The present invention relates to a system for acquiring speech data from conference participants and transcribing the data using speech recognition technology. Specific embodiments of this system will be described below.

[0416] Acquiring audio data

[0417] When a user speaks during a meeting, the device picks up the voice through a microphone. This voice data is sent to the server in real time. The device is equipped with a high-quality microphone and also has a noise-canceling function.

[0418] Sending audio data

[0419] The voice data acquired by the device is sent to a server via the Internet. The server receives the voice data and proceeds to the next processing step. HTTP, WebSocket, or other communication protocols are used.

[0420] Use of voice recognition technology

[0421] The server converts the received voice data into text data using speech recognition technology. Specifically, it uses Google's speech recognition API, which can convert voice to text with high accuracy.

[0422] Use of generative artificial intelligence

[0423] The converted text data is then fed into a generative AI system that uses deep learning models to extract sentiment scores for the words and phrases contained within the text data.

[0424] Extracting and visualizing emotion scores

[0425] The server sums or averages the emotion scores to infer an overall emotion. This information is used to visualize the mood and energy level of the meeting. Visualization can be done using graphs or charts.

[0426] Meeting progress and opinion gathering

[0427] The server provides a means for effectively managing the progress of meetings and gathering opinions from participants, for example, through real-time feedback and voting functions, which can improve the efficiency of meetings.

[0428] Specific examples

[0429] For example, if a user says "Let's move on to the next topic" during a meeting, the device's microphone captures this audio. The device then sends this audio data to the server in real time. The server then sends the received audio data to Google's speech recognition API, receiving the text data "Let's move on to the next topic." The server then inputs this text data into a generative artificial intelligence system to extract an emotion score. Ultimately, the system visualizes the mood and energy level of the meeting, helping to guide the meeting.

[0430] Prompt Sentence Examples

[0431] Examples of prompts to input to a generative AI model might include:

[0432] "Design a system that transcribes speech from meeting participants in real time. Include the process of capturing audio data, converting it to text using Google's speech recognition API, and storing it in a database."

[0433] In this way, by clarifying the specific embodiment of the system, it becomes possible to accurately understand and practice the technical scope of the invention.

[0434] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0435] Step 1:

[0436] When a user speaks during a conference, the device captures the voice through a microphone. The input is the user's speech, and the output is digital voice data. The device transmits this voice data to the server in real time. Specifically, the device's microphone captures the voice and generates voice data.

[0437] Step 2:

[0438] The voice data acquired by the device is sent to a server via the Internet. The input is the voice data sent from the device, and the output is the voice data received by the server. The server receives this voice data and proceeds to the next processing step. Specifically, data is sent and received using communication protocols such as HTTP or WebSocket.

[0439] Step 3:

[0440] The server converts the received voice data into text data using voice recognition technology. The input is the voice data received by the server, and the output is the text data returned from the voice recognition API. Specifically, the server sends the voice data to Google's voice recognition API and receives the text data returned from the API.

[0441] Step 4:

[0442] The server inputs the converted text data into the generative AI. The input is the text data returned from the speech recognition API, and the output is the emotion score analyzed by the generative AI. Specifically, the server inputs the text data into the generative AI and extracts the emotion score.

[0443] Step 5:

[0444] The server sums or averages the emotion scores to estimate the overall emotion. The input is the emotion score obtained from the generative AI, and the output is numerical data indicating the overall emotion. Specifically, the server aggregates the emotion scores and calculates the overall emotion.

[0445] Step 6:

[0446] The server visualizes the mood and energy level of the meeting. The input is numerical data that indicates the overall emotion, and the output is a visual representation such as a graph or chart. Specifically, the server converts the data into a graph or chart and displays it to the user.

[0447] Step 7:

[0448] The server provides a means to effectively manage the progress of the meeting and gather participants' opinions. The input is the progress of the meeting and the opinions of the participants, and the output is real-time feedback and voting results. Specifically, the server collects feedback in real time and gathers opinions through the voting function.

[0449] In this way, by clarifying the specific operations performed at each processing step and their inputs and outputs, it becomes easier to understand the overall processing flow of the system.

[0450] (Application example 1)

[0451] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0452] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Also, when work instructions are given by voice in a factory, there is no way to accurately record those instructions and manage them as a work log, making it difficult to accurately grasp the progress of work and the content of instructions. For this reason, there is a need for a system that can improve the efficiency of meetings and accurately record and manage work instructions in a factory.

[0453] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores of included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for acquiring work instructions by voice and transcribing and recording them in real time, and a means for managing a work log. This enables more efficient meetings and accurate recording and management of work instructions in the factory.

[0454] "Audio data" refers to data in which audio information such as conference and work instructions is recorded in digital format.

[0455] "Transcription" is the process of converting audio data into text data.

[0456] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new information and results from input data.

[0457] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in audio or text data.

[0458] "Mood" refers to the overall atmosphere and emotional state of a meeting or work environment.

[0459] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or work environment.

[0460] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[0461] "Work instructions" are voice or text information that instructs the content and procedures of work to be done in a factory.

[0462] "Real-time" refers to the instantaneous processing of data and provision of information.

[0463] A "work log" is data that records work instructions and progress within a factory.

[0464] "Management" is the process of organizing data and information and storing and using them appropriately.

[0465] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of a meeting and the collection of opinions of participants, means for obtaining work instructions via voice and transcribing and recording them in real time, and means for managing a work log.

[0466] System program processing explanation

[0467] Acquiring and transcribing audio data

[0468] The server uses a microphone to capture audio data, which is recorded using the Python speech_recognition library and converted to text using the Google Cloud Speech-to-Text API.

[0469] Extracting and visualizing emotion scores

[0470] Generative AI uses natural language processing techniques to extract sentiment scores for words and phrases contained in text data. The extracted sentiment scores are summed or averaged to infer overall sentiment. The inferred sentiment is visualized as the mood or energy level of the meeting.

[0471] Real-time transcription of work instructions and work log management

[0472] The server receives voice instructions from within the factory and transcribes them in real time. This allows for accurate recording of work instructions. The recorded work instructions are managed as a work log. This makes it possible to accurately grasp the progress of work and the content of instructions.

[0473] Examples of specific examples and prompts

[0474] As a concrete example, consider work instructions in a factory. For example, when a worker issues a voice command such as "Please proceed to the next process," the command is transcribed in real time and recorded in a work log.

[0475] Example prompts to input to a generative AI model:

[0476] Transcribe voice instructions such as "Proceed to the next step" in real time and record them in your work log.

[0477] Using this prompt, the generative AI model can accurately transcribe the voice instructions and take appropriate action to record them in the work log.

[0478] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0479] Step 1: Acquire audio data

[0480] The server captures voice data using a microphone. When a user speaks into the microphone, the voice is recorded in digital format. The input is the user's voice, and the output is digital voice data. This voice data is captured using the Python speech_recognition library.

[0481] Step 2: Transcribe the audio data

[0482] The server sends the acquired voice data to the Google Cloud Speech-to-Text API for transcription. The input is digital voice data, and the output is text data. The Google Cloud Speech-to-Text API analyzes the voice data and generates corresponding text data.

[0483] Step 3: Extract sentiment scores

[0484] The server uses generative AI to extract sentiment scores for words and phrases contained in text data. The input is text data, and the output is a sentiment score. Natural language processing technology is used to quantify the emotional nuance of each word or phrase.

[0485] Step 4: Sum or average sentiment scores

[0486] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. This allows us to numerically express the mood and energy level of the meeting.

[0487] Step 5: Visualize emotions

[0488] The server visualizes the inferred overall emotion score. The input is the overall emotion score, and the output is a visual graph or chart, which provides a visual understanding of the mood and energy level of the meeting.

[0489] Step 6: Real-time transcription of work instructions

[0490] The server receives voice instructions from the factory and transcribes them in real time. The input is the user's voice instructions, and the output is text data. Voice recognition technology is used to instantly convert voice instructions into text data.

[0491] Step 7: Manage your work log

[0492] The server manages the transcribed work instructions as a work log. The input is text data, and the output is an organized work log. This allows the progress of work and the content of instructions to be accurately recorded and managed.

[0493] Example 2

[0494] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0495] Conventional methods for recording meetings and interviews not only transcribe audio data, but also make it difficult to efficiently analyze the content and extract important information. Furthermore, there is a lack of a way to visualize the mood and energy level of the meeting and effectively summarize the opinions of participants. This leads to problems such as a decrease in the efficiency of meeting progress and decision-making.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0497] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the collection of participant opinions, means for inputting the transcribed data into the generative AI model and analyzing the text data using a deep learning model, means for generating prompts for the generative AI model and inputting them together with the text data, and means for providing the generated information to the user. This enables efficient analysis of voice data, extraction of important information, visualization of the mood and energy level of the meeting, and effective collection of participant opinions.

[0498] "Audio data" refers to data in which audio is recorded in digital format.

[0499] "Transcription" is the process of converting audio data into text data.

[0500] A "generative AI model" is an artificial intelligence model that uses deep learning technology to analyze text data and generate new information.

[0501] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0502] "Mood" refers to the overall atmosphere or emotional state of a meeting or interview.

[0503] "Energy level" refers to the level of activity and enthusiasm of participants in a meeting or interview.

[0504] A "prompt" is an instruction entered into a generative AI model to make it perform a specific task.

[0505] A "deep learning model" is a machine learning model that uses a multi-layered neural network to analyze data and learn patterns.

[0506] "Text data" is digital data made up of letters and symbols.

[0507] "User" means a person or organization that uses the system to transcribe and analyze audio data.

[0508] A "server" is a computer that is the central part of a system and is a device that processes and stores data.

[0509] MODE FOR CARRYING OUT THE INVENTION

[0510] This invention is a system that transcribes audio data and analyzes it using a generative AI model. A specific embodiment of this system is described below.

[0511] System configuration

[0512] This system consists of a user, a server, and a device. The user provides voice data and accesses the server through the device. The server transcribes the voice data, analyzes it using a generative AI model, and provides the results.

[0513] Hardware and software used

[0514] Hardware: Servers, devices (PCs, smartphones, etc.)

[0515] Software: Speech recognition technology (e.g., Google Speech-to-Text API), generative AI models (e.g., BERT, GPT-3 (registered trademark))

[0516] Data processing and calculation

[0517] 1. Transcription of audio data:

[0518] Users can record audio data from meetings or interviews on their device and transcribe it using speech recognition technology. For example, by using the Google Speech-to-Text API, the audio data can be converted into text data with high accuracy.

[0519] 2. Upload the transcription data:

[0520] Users upload the transcribed text data to the server via their device, using a web interface or a dedicated upload tool.

[0521] 3. Analysis using generative AI models:

[0522] The server receives the uploaded text data and feeds it into a generative AI model, specifically a deep learning model such as BERT or GPT-3. The server generates appropriate prompts for these models and feeds them along with the text data.

[0523] 4. Prompt generation:

[0524] The server generates a prompt for the generative AI model, such as:

[0525] Prompt: "Please summarize the following meeting minutes. Transcript: [transcribed text data]"

[0526] 5. Provision of generated information:

[0527] The generative AI model analyzes the prompt and text data to generate the necessary information. For example, it extracts a summary of the meeting or key points. The server then provides the generated information to the user. The user can view the generated summary and analysis results through a web interface or a dedicated dashboard.

[0528] Specific examples

[0529] For example, a user records audio data from a meeting and transcribes it using the Google Speech-to-Text API. The user then uploads the text data to a server. The server then inputs the uploaded text data into the BERT model to extract key points. The server then uses a generative AI model (GPT-3) to generate a summary based on the extracted points. The user can then view the generated summary through a web interface and download it if necessary.

[0530] In this way, the present invention efficiently analyzes voice data and extracts important information, visualizes the mood and energy level of a meeting, and effectively aggregates participants' opinions.

[0531] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0532] Step 1:

[0533] The user records the audio data.

[0534] Specifically, the user records the audio of a meeting or interview using a recording device (such as a smartphone or a dedicated recorder). The input is audio data, and the output is a recorded audio file.

[0535] Step 2:

[0536] The user transcribes the audio data.

[0537] Specifically, the user converts the recorded audio file into text data using speech recognition technology (e.g., Google Speech-to-Text API). The input is the audio file, and the output is the transcribed text data.

[0538] Step 3:

[0539] The user uploads the transcription data to the server.

[0540] Specifically, users upload transcribed text data to the server using a web interface or a dedicated upload tool. The input is the transcribed text data, and the output is the text data stored on the server.

[0541] Step 4:

[0542] The server feeds the transcription data into a generative AI model.

[0543] Specifically, the server receives the uploaded text data, generates an appropriate prompt for the generative AI model, and inputs it along with the text data. The input is the text data stored on the server and the generated prompt, and the output is the data input to the generative AI model.

[0544] Step 5:

[0545] A generative AI model analyzes the text data and generates the necessary information.

[0546] Specifically, a generative AI model (e.g., BERT, GPT-3) analyzes the prompt and text data to extract a summary or key points. The input is the text data and prompt entered into the generative AI model, and the output is the generated summary or analysis results.

[0547] Step 6:

[0548] The server provides the generated information to the user.

[0549] Specifically, the server provides the user with the summary and analysis results obtained from the generative AI model. The user can check the generated information through a web interface or a dedicated dashboard and download it as needed. The input is the generated summary and analysis results, and the output is the information provided to the user.

[0550] (Application example 2)

[0551] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0552] Conventional conference systems and content generation systems lack the functionality to not only transcribe audio data but also to analyze text data using generative AI and automatically generate optimal content for users. Furthermore, there are insufficient means for users to edit, save, and share the generated content. This creates the problem of ineffective progress in meetings and the gathering of opinions from participants.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0554] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the collection of participant opinions, means for transcribing voice input by the user in real time, means for inputting the transcribed data into a generative AI and analyzing the text data, means for automatically generating optimal content for the user based on the analysis results, and means for allowing the user to edit, save, and share the generated content. This not only improves the progress of the meeting and the collection of participant opinions, but also enables the user to automatically generate optimal content based on voice input, and edit, save, and share it.

[0555] "Audio data" refers to data in which audio is recorded in digital format.

[0556] "Transcription" is the process of converting audio data into text data.

[0557] "Generative AI" is artificial intelligence that uses deep learning models to generate or analyze text data.

[0558] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0559] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[0560] "Energy level" is an indicator of the level of activity and enthusiasm of participants in a meeting or conversation.

[0561] "Visualization" is the process of visually displaying data or information.

[0562] "Progress" refers to the way a meeting or conversation is conducted and its progress.

[0563] "Convergence" is the process of summarizing the opinions of multiple participants.

[0564] "Real-time" refers to processing or reaction occurring immediately, without delay.

[0565] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[0566] "Content" is a collection of information or data, including articles, blogs, social media posts, etc.

[0567] "Editing" is the process of modifying or changing generated content.

[0568] "Preservation" is the process of recording generated content in a digital format so that it can be accessed at a later time.

[0569] "Sharing" is the process of sharing generated content with other users.

[0570] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for included words and phrases, means for inferring overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of meeting progress and gathering opinions of participants, means for transcribing voice input by a user in real time, means for inputting the transcribed data into a generative AI and analyzing the text data, means for automatically generating optimal content for the user based on the analysis results, and means for users to edit, save, and share the generated content.

[0571] Hardware and software used

[0572] Hardware: Smartphone (microphone, internet connection)

[0573] Software: Python, speech_recognition library, transformers library

[0574] Data processing and calculation

[0575] 1. Speech Recognition: The user inputs speech through the smartphone microphone. The speech data is transcribed in real time using the speech_recognition library.

[0576] 2. Text analysis: The transcribed text data is fed into a generative AI (such as BERT or GPT-3) to analyze the text data. The transformers library is used for analysis.

[0577] 3. Extracting emotion scores: Generative AI extracts emotion scores for words and phrases contained in text data, then sums or averages them to infer the overall emotion.

[0578] 4. Content generation: Based on the analysis results, the most suitable content for the user (articles, blogs, social media posts, etc.) is automatically generated.

[0579] 5. Edit, save and share: Generated content can be edited, saved and shared by users.

[0580] Specific examples

[0581] When a user says, "I want to write a blog about today's weather," the system performs the following process:

[0582] 1. Speech recognition: The user's speech is transcribed and text data such as "I want to write a blog about today's weather" is generated.

[0583] 2. Text analysis: Generative AI analyzes this text data and gathers relevant information.

[0584] 3. Content generation: Based on the analysis results, blog articles like the following are automatically generated.

[0585] "Today's forecast is for sunny skies with occasional cloudy spells. This morning the sky was clear and the temperature was comfortable, but the clouds will increase in the afternoon and it may feel a bit chilly. It's a good idea to bring a light jacket with you when you go out."

[0586] Prompt Sentence Examples

[0587] "I want to blog about today's weather."

[0588] I want to write a review of a new product.

[0589] "I want to plan a trip"

[0590] In this way, generative AI can automatically generate content based on the user's voice input, which can then be edited, saved, and shared.

[0591] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0592] Step 1:

[0593] The user speaks into the smartphone's microphone, and the speech data is captured and transcribed in real time using the speech_recognition library.

[0594] Input: User's voice data

[0595] Data processing: Converting voice data into text data

[0596] Output: Transcribed text data

[0597] Step 2:

[0598] The server then feeds the transcribed text data into a generative AI (such as BERT or GPT-3), which analyzes the text data and extracts sentiment scores for the words and phrases it contains.

[0599] Input: Transcribed text data

[0600] Data Computing: Analyzing text data and extracting sentiment scores

[0601] Output: Text data with sentiment scores

[0602] Step 3:

[0603] The server sums or averages the extracted emotion scores to infer the overall emotion.

[0604] Input: Text data with sentiment scores

[0605] Data operations: summing or averaging sentiment scores

[0606] Output: Overall sentiment score

[0607] Step 4:

[0608] The server visualizes the mood and energy level of the meeting based on the overall emotional score.

[0609] Input: Overall sentiment score

[0610] Data processing: Visualization of emotion scores

[0611] Output: Visualized data on mood and energy levels

[0612] Step 5:

[0613] The server uses generative AI to automatically generate optimal content based on the voice input provided by the user.

[0614] Input: What the user says

[0615] Data Computation: Content Generation by Generative AI

[0616] Output: Auto-generated content

[0617] Step 6:

[0618] Users can edit, save, and share the generated content. Through a smartphone interface, users can modify the generated content, save it digitally, and share it with other users.

[0619] Input: Auto-generated content

[0620] Data processing: Editing, saving, and sharing content

[0621] Output: Edited, saved, and shared content

[0622] In this way, specific data processing and calculations are performed at each step, allowing users to generate optimal content based on their voice input, which can then be edited, saved, and shared.

[0623] Example 3

[0624] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0625] Conventional meeting systems make it difficult to grasp the mood and energy level of a meeting in real time, which means that meeting progress and the gathering of participants' opinions are not efficiently managed. Furthermore, extracting and visualizing emotion scores is often done manually, which is time-consuming and labor-intensive.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0627] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative artificial intelligence, means for extracting emotion scores for included words and phrases, means for estimating overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, means for inputting text data, means for extracting emotion scores using an emotion analysis tool, means for aggregating the extracted emotion scores, and means for displaying the scores as a graph on a dashboard. This allows the mood and energy level of the meeting to be grasped in real time, enabling efficient meeting management.

[0628] "Audio data" refers to data in which audio is recorded in digital format.

[0629] "Transcription" is the process of converting audio data into text data.

[0630] "Generative AI" is an AI technology that generates new data and information based on input data.

[0631] An "emotion score" is a numerical representation of positive, negative, neutral, or other emotions generated by analyzing words and phrases in text.

[0632] A "sentiment analysis tool" is software or algorithms for extracting sentiment scores from text data.

[0633] A "dashboard" is an interface for visually displaying data.

[0634] A "graph" is a graphic representation of data visually.

[0635] "Meeting mood" refers to the overall emotional atmosphere and tone of the participants in a meeting.

[0636] "Energy level" is an indicator of the liveliness and positivity of participants in a meeting.

[0637] "Opinion gathering" is the process of summarizing the opinions of participants in a meeting.

[0638] "Efficiency" refers to making processes and tasks more effective and faster.

[0639] MODE FOR CARRYING OUT THE INVENTION

[0640] This invention is a system for efficiently managing meetings by grasping the mood and energy level of the meeting in real time. This system transcribes audio data, extracts emotion scores using generative artificial intelligence, and visualizes them, allowing for effective progress in the meeting and the collection of opinions of participants.

[0641] Hardware and software used

[0642] Hardware:

[0643] server

[0644] Devices (computers and smartphones used by users)

[0645] software:

[0646] Speech recognition technology (to transcribe voice data)

[0647] Generative AI (to analyze text data)

[0648] Sentiment analysis tools (e.g., VADER)

[0649] Dashboard (interface for visualizing data)

[0650] Data processing and calculation

[0651] A user inputs text data such as minutes of a meeting, a chat log, etc. into a terminal. For example, the user copies the minutes of a meeting and pastes them into an input field of the system.

[0652] The server receives the input text data and analyzes it using a sentiment analysis tool such as VADER. Specifically, the server passes the text data to VADER, which extracts a sentiment score for each word or phrase. For example, the word "good" has a positive score, and the word "bad" has a negative score.

[0653] The server sums or averages the extracted emotion scores to infer the overall emotion. Specifically, the server aggregates the emotion scores of each word or phrase as a numerical value and calculates an overall positive and negative score. For example, if the positive score is 70% and the negative score is 30%, the server infers that the overall mood is positive.

[0654] The server then displays the estimated overall sentiment as a graph on the dashboard. Specifically, the server uses the aggregated sentiment scores to plot the percentage of positive and negative sentiment on the dashboard as a pie chart or bar graph. For example, it visually shows that the positive mood is 70% and the negative mood is 30%.

[0655] Users can use the information on the dashboard to streamline meeting progress and gather participant opinions. Specifically, users can look at the dashboard and take measures such as changing the agenda if the mood remains negative. For example, if there are a lot of positive opinions, they can continue the discussion as is, but if there are a lot of negative opinions, they can identify problems and consider solutions.

[0656] Examples of specific examples and prompts

[0657] Examples:

[0658] The user inputs the minutes of the meeting into the system, for example, "Today's meeting discussed the development of a new product. Many participants had positive opinions, but some participants expressed concerns."

[0659] The server analyzes this text using VADER and extracts a positive score of 0.7 and a negative score of 0.3.

[0660] The server aggregates these scores and estimates the overall mood as 70% positive and 30% negative.

[0661] The server displays the results as a pie chart on a dashboard.

[0662] The user looks at the dashboard, sees that there are many positive opinions, and decides to proceed with the discussion.

[0663] Example prompt sentence:

[0664] "Extract a sentiment score from the following text data, infer the overall sentiment, and display it on a dashboard. Text data: 'Today's meeting discussed the development of a new product. Many participants had positive opinions, but some expressed concerns.'"

[0665] In this way, the user can visualize the mood and energy level of the meeting by utilizing generative AI, and can manage the meeting effectively. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0666] Step 1:

[0667] A user inputs text data such as minutes of a meeting, a chat log, etc. into a terminal. Specifically, the user copies the minutes of the meeting and pastes them into an input field of the system.

[0668] Input: Text data such as meeting minutes and chat logs

[0669] Output: The input text data

[0670] Step 2:

[0671] The server receives the input text data and analyzes it using a sentiment analysis tool (e.g., VADER). Specifically, the server passes the text data to VADER, which extracts a sentiment score for each word or phrase. For example, the word "good" has a positive score, and the word "bad" has a negative score.

[0672] Input: Entered text data

[0673] Output: Sentiment score for each word or phrase

[0674] Step 3:

[0675] The server sums or averages the extracted emotion scores to infer the overall emotion. Specifically, the server aggregates the emotion scores of each word or phrase as a numerical value and calculates an overall positive and negative score. For example, if the positive score is 70% and the negative score is 30%, the server infers that the overall mood is positive.

[0676] Input: Sentiment scores for each word or phrase

[0677] Output: Overall sentiment score (positive and negative scores)

[0678] Step 4:

[0679] The server then displays the estimated overall sentiment as a graph on the dashboard. Specifically, the server uses the aggregated sentiment scores to plot the percentage of positive and negative sentiment on the dashboard as a pie chart or bar graph. For example, it visually shows that the positive mood is 70% and the negative mood is 30%.

[0680] Input: Overall sentiment score (positive and negative scores)

[0681] Output: Graphical display on the dashboard

[0682] Step 5:

[0683] Users can use the information on the dashboard to streamline meeting progress and gather participant opinions. Specifically, users can look at the dashboard and take measures such as changing the agenda if the mood remains negative. For example, if there are a lot of positive opinions, they can continue the discussion as is, but if there are a lot of negative opinions, they can identify problems and consider solutions.

[0684] Input: Graph display on dashboard

[0685] Output: Efficient meeting progress and opinion gathering

[0686] (Application example 3)

[0687] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0688] In the work environment within a factory, there is a need to understand the communication and working mood of workers in real time in order to improve work efficiency and safety. However, conventional systems have difficulty accurately understanding the emotions and mood of workers, which has led to issues such as insufficient efficiency in work progress and opinion gathering.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0690] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores for included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the work environment, a means for improving the efficiency of work progress and the collection of worker opinions, a means for monitoring worker conversations in real time, and a means for displaying emotion scores on a dashboard. This makes it possible to grasp the emotions and moods of workers in real time and improve work efficiency and safety.

[0691] "Audio data" refers to data in which audio is recorded in digital format.

[0692] "Transcription" is the process of converting audio data into text data.

[0693] "Generative AI" is an artificial intelligence technology that generates new data from given data.

[0694] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0695] "Mood" is the overall atmosphere or emotional state in a particular environment or situation.

[0696] "Energy level" refers to the degree of activity or vitality in a particular environment or situation.

[0697] "Visualization" is the process of visually displaying data or information.

[0698] "Real-time" means that data and information are processed and displayed immediately.

[0699] A "dashboard" is an interface for displaying data and information in a centralized manner.

[0700] "Monitoring" is the process of continuously observing and recording specific data or conditions.

[0701] "Opinion gathering" is the process of bringing together multiple opinions and ideas to arrive at a single conclusion or direction.

[0702] A system for carrying out the present invention monitors and visualizes in real time the communication and working mood of workers in a work environment within a factory. A specific embodiment of this system will be described below.

[0703] System Configuration

[0704] The system consists of the following main components:

[0705] 1. Transcribing voice data: Using voice recognition technology, workers' conversations are converted into text data in real time.

[0706] 2. Method of inputting data into generative AI: Input the transcribed text data into generative AI.

[0707] 3. Extracting sentiment scores: Using sentiment analysis tools such as VADER, sentiment scores are extracted from text data.

[0708] 4. Means of inferring overall sentiment: The extracted sentiment scores are summed or averaged to infer overall sentiment.

[0709] 5. A way to visualize mood and energy levels: The estimated overall emotions are displayed as graphs on a dashboard.

[0710] 6. A means to improve the efficiency of work progress and opinion gathering: Based on the information in the dashboard, work progress and the gathering of opinions from workers can be made more efficient.

[0711] Hardware and software used

[0712] Hardware: Microphones and displays mounted on factory robots.

[0713] Software: Python, vaderSentiment library, matplotlib library.

[0714] Data processing and calculation

[0715] The server transcribes the voice data acquired from the factory robots in real time and inputs it into the generative AI. The generative AI then uses a sentiment analysis tool such as VADER to extract sentiment scores from the text data. The extracted sentiment scores are summed or averaged to infer overall sentiment. The inferred sentiment is displayed as a graph on a dashboard, helping to improve the efficiency of work progress and opinion gathering.

[0716] Specific examples

[0717] For example, consider a system that monitors the conversations of workers in a factory in real time and displays an emotion score on a dashboard. This system makes it possible to grasp the emotions and moods of workers in real time and improve work efficiency and safety.

[0718] Example prompts to input to the generative AI model

[0719] Design a system that monitors the conversations of factory workers in real time, extracts emotion scores, and displays them on a dashboard. Use VADER to extract emotion scores and average the overall emotion to display them. Create graphs using Python and matplotlib.

[0720] By inputting this prompt into the generative AI model, more detailed design and implementation methods can be suggested.

[0721] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0722] Step 1:

[0723] The server acquires voice data from microphones installed on factory robots. The input is real-time voice data, and the output is digital voice data, which is used for subsequent transcription processing.

[0724] Step 2:

[0725] The server uses speech recognition technology to transcribe the audio data. The input is digital audio data, and the output is text data. This text data is then fed into generative AI.

[0726] Step 3:

[0727] The server feeds text data into the generative AI. The input is transcribed text data, and the output is data ready for extracting emotion scores.

[0728] Step 4:

[0729] The server extracts sentiment scores from text data using sentiment analysis tools such as VADER. The input is the text data fed into the generative AI, and the output is the sentiment score for each word or phrase.

[0730] Step 5:

[0731] The server sums or averages the extracted emotion scores to infer an overall emotion. The input is the emotion score for each word or phrase, and the output is an overall emotion score, which indicates the mood or energy level of the work environment.

[0732] Step 6:

[0733] The server displays the estimated overall sentiment score as a graph on a dashboard. The input is the overall sentiment score, and the output is a visually displayed graph. This graph can be used to streamline work progress and opinion gathering.

[0734] Step 7:

[0735] Users can streamline work progress and opinion gathering based on the graphs on the dashboard. The input is the graph on the dashboard, and the output is the streamlined work progress and opinion gathering.

[0736] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0737] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0738] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0739] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0740] [Second embodiment]

[0741] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0742] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0743] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0744] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0745] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0746] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0747] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0748] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0749] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0750] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0751] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0752] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0753] "Example 1"

[0754] In one embodiment of the present invention, speech data from participants in a conference is acquired and transcribed using speech recognition technology. This transcription can be performed using, for example, Google's speech recognition API.

[0755] "Example 2"

[0756] The transcribed data is then fed into a generative AI, which is designed using a deep learning model, such as BERT or GPT-3.

[0757] "Example 3"

[0758] Generative AI extracts sentiment scores for words and phrases, sums or averages them, and infers the overall sentiment. This sentiment score can be extracted using a sentiment analysis tool such as VADER.

[0759] "Example 4"

[0760] Finally, the mood and energy level of the meeting can be visualized, for example, by displaying it as a graph on a dashboard. This can help improve the effectiveness of the meeting and the gathering of participants' opinions.

[0761] The processing flow of each embodiment will be described below.

[0762] "Example 1"

[0763] Step 1: The speech of the meeting participants is acquired as audio data using a microphone or the like.

[0764] Step 2: The acquired voice data is transcribed using speech recognition technology. This transcription can be performed using, for example, Google's speech recognition API.

[0765] "Example 2"

[0766] Step 1: Input the transcribed data from Example 1 into the generative AI.

[0767] Step 2: Generative AI is designed using deep learning models, such as BERT or GPT-3.

[0768] "Example 3"

[0769] Step 1: Extract sentiment scores for the words and phrases contained in the data input into the generative AI in Example 2.

[0770] Step 2: The extracted emotion scores are summed or averaged to estimate the overall emotion. This emotion score can be extracted using an emotion analysis tool such as VADER. "Example 4"

[0771] Step 1: Visualize the mood and energy level of the meeting using the overall emotions inferred in Example 3.

[0772] Step 2: This visualization can be done, for example, by displaying it as a graph on a dashboard, which will help improve the effectiveness of the meeting and the gathering of participants' opinions.

[0773] Example 1

[0774] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0775] In conventional meetings, it was difficult to accurately record what participants said and share it in real time. Furthermore, there was a lack of efficient ways to proceed with the meeting and gather participants' opinions. This led to a decrease in meeting efficiency and the risk of important information being overlooked.

[0776] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server, means for the server to transcribe the voice data using voice recognition technology, means for storing the transcribed data in a database, means for transmitting the stored data to a terminal and displaying it, and means for efficiently progressing the meeting and collecting opinions of participants. This makes it possible to accurately record and share the contents of statements made in the meeting in real time. Furthermore, since the meeting can be progressed and opinions can be efficiently collected, the efficiency of the meeting is improved and important information can be prevented from being overlooked.

[0777] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[0778] The "means of collection" refers to a method of acquiring voice data using a microphone or voice input device.

[0779] The "means of transmission" refers to a method for transferring collected voice data to a server via a network.

[0780] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0781] "Transcribing means" refers to a method of converting audio data into text data using voice recognition technology.

[0782] "Database" means a digital storage system for storing transcribed text data.

[0783] "Means for storage" refers to the method for storing the transcribed text data in a database.

[0784] "Display means" refers to a method for displaying text data on the terminal screen.

[0785] "Proceeding a meeting" is the process of advancing the discussion in line with the meeting agenda.

[0786] "Convergence" is the process of summarizing the opinions of meeting participants and reaching a common conclusion.

[0787] "Means for improving effectiveness" are methods and tools for efficiently conducting meetings and gathering opinions.

[0788] The present invention provides a system for transcribing speeches made by participants in a meeting in real time, thereby improving the efficiency of the progress of the meeting and the collection of opinions. Specific embodiments of this system will be described below.

[0789] Hardware and software used

[0790] Hardware

[0791] Microphone: Used to collect the speech of meeting participants.

[0792] Terminal: A device such as a personal computer, tablet, or smartphone that collects and transmits voice data to a server.

[0793] Server: A computer system that receives audio data and transcribes it using speech recognition technology.

[0794] software

[0795] Audio collection application: An application that is installed on a device to collect audio data and send it to a server.

[0796] Speech recognition API: An API for converting voice data into text data. Specifically, it uses common voice recognition technology.

[0797] Database: A digital storage system for storing transcribed text data.

[0798] System Operation

[0799] server

[0800] The server receives the voice data sent from the device. The received voice data is sent to a voice recognition API and converted into text data. The converted text data is saved in a database. The saved text data is then sent to the device and displayed in real time.

[0801] Terminal

[0802] The device collects audio data via a microphone when participants speak, and sends the collected audio data to a server via a voice collection application. The text data returned from the server is displayed on the device screen, allowing users to view it in real time.

[0803] User

[0804] Users speak during the meeting. Their speech is collected as audio data through the device's microphone. Users can then see their speech being transcribed on the device's screen.

[0805] Specific examples

[0806] For example, if a user says "Let's discuss the next item on the agenda" during a meeting, the device's microphone will collect this voice and send it to the server via a voice collection application. The server will then use a speech recognition API to convert this voice data into text data, generating the text "Let's discuss the next item on the agenda." This text data will then be stored in a database and sent to the device. Finally, the text "Let's discuss the next item on the agenda" will be displayed on the device screen, allowing the user to view it in real time.

[0807] Prompt Sentence Examples

[0808] "Design a system that transcribes meeting participants' speech in real time. Include a process for collecting audio data, converting it to text using speech recognition technology, and displaying it on a device screen."

[0809] This system is very useful for improving meeting efficiency and automating the creation of meeting minutes.

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] Step 1: Collecting audio data

[0812] The device collects audio data through a microphone when a participant speaks. Specifically, the device's audio collection application digitally captures the audio input from the microphone and saves it as an audio file. The input is the participant's speech, and the output is a digital audio file.

[0813] Step 2: Sending audio data

[0814] The device sends the collected voice data to a server. Specifically, the voice collection application uploads the voice file to the server via the network. The input is a digital voice file, and the output is the voice data sent to the server.

[0815] Step 3: Receiving audio data

[0816] The server receives the voice data sent from the terminal. Specifically, the server's receiving module receives the voice file sent over the network and stores it in the server. The input is the voice data sent from the terminal, and the output is the voice data stored in the server.

[0817] Step 4: Transcribe the audio data

[0818] The server transcribes the received voice data using speech recognition technology. Specifically, the server calls a speech recognition API and converts the voice data into text data. The input is the voice data stored on the server, and the output is the text data returned by the speech recognition API.

[0819] Step 5: Save the text data

[0820] The server saves the transcribed text data in a database. Specifically, the server's database module stores the text data in the database. The input is the text data returned from the speech recognition API, and the output is the text data saved in the database.

[0821] Step 6: Send text data

[0822] The server transmits the stored text data to the terminal. Specifically, the server's transmission module retrieves the text data from the database and transmits it to the terminal via the network. The input is the text data stored in the database, and the output is the text data transmitted to the terminal.

[0823] Step 7: Displaying Text Data

[0824] The terminal receives the text data sent from the server and displays it on the screen. Specifically, the display application of the terminal receives the text data via the network and displays it on the user interface. The input is the text data sent from the server, and the output is the text data displayed on the terminal screen.

[0825] Through this series of processes, the remarks of meeting participants are transcribed in real time and displayed on the device screen, allowing for efficient progress in the meeting and the collection of opinions.

[0826] (Application example 1)

[0827] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Furthermore, when workers give voice instructions to machines at work sites such as factories, there is no system that can accurately recognize those instructions and perform the appropriate operation. This creates the problem of insufficient progress in meeting efficiency and automation of on-site work.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0830] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included. This makes it possible to visualize the mood and energy level of a meeting, and to improve the effectiveness of meeting progress and gathering participants' opinions. Furthermore, by including a means for analyzing voice data and controlling machine operation based on specific keywords, workers in factories and other workplaces can issue voice instructions to machines, which then perform appropriate operations based on those instructions.

[0831] "Audio data" refers to digital data of audio collected at meetings, work sites, etc.

[0832] "Transcription" is the process of converting audio data into text data.

[0833] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new data and information from input data.

[0834] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0835] "Meeting mood" refers to the emotional atmosphere of all participants during the meeting.

[0836] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or at a workplace.

[0837] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[0838] "Controlling the operation of a machine" means instructing a machine to perform a specific operation based on voice data or text data.

[0839] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[0840] A "deep learning model" is a type of artificial intelligence that uses multi-layered neural networks to analyze and learn from data.

[0841] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, and means for analyzing the voice data and controlling the operation of a machine based on specific keywords.

[0842] The server acquires the audio data and transcribes it using speech recognition technology. For example, Google's speech recognition API is used as the speech recognition technology. The transcribed text data is then input into a generative AI system, which extracts emotion scores for the words and phrases contained within. The emotion scores are then summed or averaged to estimate the overall emotion. This makes it possible to visualize the mood and energy level of the meeting, thereby improving the effectiveness of meeting progress and gathering participant opinions.

[0843] Furthermore, the server analyzes the voice data and controls the machine's operation based on specific keywords. For example, if a worker in a factory says, "Please proceed to the next process," the robot will proceed to the next process. In this case, speech recognition technology is used to convert the voice into text, and the robot's operation is controlled based on that text.

[0844] The hardware used includes a microphone and a factory robot, while the software uses Python, the speech_recognition library, and Google's speech recognition API.

[0845] For example, in a factory, when a worker says "Please proceed to the next step," the robot will proceed to the next step. The following is an example of a prompt:

[0846] When you say "Please proceed to the next step," create a program that will make the robot proceed to the next step. Use Google's speech recognition API to convert speech into text and control the robot's actions based on that text.

[0847] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0848] Step 1:

[0849] The server captures voice data through a microphone: the input is the user's voice, and the output is digital voice data, which is used in subsequent processing steps.

[0850] Step 2:

[0851] The server transcribes the acquired voice data using speech recognition technology. Specifically, it converts the voice data into text data using Google's speech recognition API. The input is the voice data acquired in step 1, and the output is text data.

[0852] Step 3:

[0853] The server inputs the transcribed text data into a generative AI and extracts sentiment scores for the words and phrases contained within. The input is the text data obtained in step 2, and the output is the sentiment score for each word or phrase. The generative AI calculates the sentiment scores using a deep learning model.

[0854] Step 4:

[0855] The server sums or averages the extracted emotion scores to infer the overall emotion. The input is the emotion scores obtained in step 3, and the output is an overall emotion score. This overall emotion score is an indicator of the mood and energy level of the meeting.

[0856] Step 5:

[0857] The server visualizes the overall emotion score. The input is the overall emotion score obtained in step 4, and the output is a visually displayed emotion score. This helps to improve the progress of the meeting and the effectiveness of gathering participants' opinions.

[0858] Step 6:

[0859] The server analyzes the voice data and controls the machine's operation based on specific keywords. The input is the text data obtained in step 2, and the output is instructions for the machine's operation. For example, if the keyword "Please proceed to the next step" is included, the robot will proceed to the next step.

[0860] Step 7:

[0861] The server executes the machine's operations. The input is the operation instructions obtained in step 6, and the output is the actual machine operation. Specifically, the server performs operations such as the robot proceeding to the next process or installing parts.

[0862] Example 2

[0863] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] Conventional conference systems are required not only to transcribe audio data but also to visualize the mood and energy level of the meeting and effectively gather opinions from participants. However, there are few systems that provide these functions in an integrated manner, and in particular, they lack advanced analysis and output using generative AI models. For this reason, a new system is needed to improve the progress of meetings and the effectiveness of gathering opinions from participants.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0866] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the effectiveness of meeting progress and participant opinion gathering, means for preprocessing the transcription data, means for generating prompt sentences for the generative AI model, means for acquiring generation results from the generative AI model, means for post-processing the generation results, and means for outputting the generation results, thereby enabling the meeting to be progressed effectively and participant opinions to be gathered.

[0867] "Audio data" refers to data in which audio is recorded in digital format.

[0868] "Transcription" is the process of converting audio data into text data.

[0869] A "generative AI model" is an artificial intelligence model that generates text data using deep learning technology.

[0870] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[0871] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[0872] "Energy level" refers to the level of activity and enthusiasm of participants in a meeting or conversation.

[0873] "Preprocessing" is the initial process of preparing data in a format suitable for analysis and processing.

[0874] A "prompt sentence" is an input sentence that instructs a generative AI model on a specific task.

[0875] "Post-processing" refers to the process of preparing the generated data in a final output format.

[0876] "Output" refers to providing processed data to a user or other system.

[0877] MODE FOR CARRYING OUT THE INVENTION

[0878] This invention relates to a system that transcribes voice data and analyzes and outputs the results using a generative AI model. A specific embodiment of this system is described below.

[0879] System configuration

[0880] The server system consists of the following hardware and software:

[0881] Hardware:

[0882] High-performance processor

[0883] Large capacity memory

[0884] High-speed storage devices

[0885] software:

[0886] Voice recognition technology (e.g., Google Speech-to-Text API)

[0887] Generative AI models (e.g., BERT, GPT-3)

[0888] Scripts for data pre- and post-processing

[0889] Program processing

[0890] The server receives the voice data sent from the user or device and performs the following processes:

[0891] 1. Transcription of audio data:

[0892] The server converts the audio data into text using speech recognition technology, for example, by using the Google Speech-to-Text API to transcribe the audio file.

[0893] 2. Data preprocessing:

[0894] The server preprocesses the transcription data and removes unnecessary noise and special characters. For example, it converts the transcription data "Hello, this is a test" into "Hello, this is a test."

[0895] 3. Generative AI model selection:

[0896] The server selects the generative AI model to use, for example, a deep learning model such as BERT or GPT-3.

[0897] 4. Prompt generation:

[0898] The server generates prompts to be input to the generative AI model. Prompts instruct the model on specific tasks. For example, it generates a prompt such as "Please summarize the following sentence: 'Hello, this is a test.'"

[0899] 5. Inputting data into generative AI models:

[0900] The server inputs the generated prompt and preprocessed data into a generative AI model. For example, the server inputs the prompt "Please summarize the following sentence: 'Hello, this is a test'" into the BERT model.

[0901] 6. Obtaining the generated results:

[0902] The server receives the output from the generative AI model, which generates an appropriate output based on the input prompt. For example, the server receives the output "Hello, this is the test result" from the BERT model.

[0903] 7. Post-processing of results:

[0904] The server post-processes the generated output and formats it as needed, for example converting the output to JSON format and adding necessary metadata.

[0905] 8. Result output:

[0906] The server outputs the final results to the user or to a terminal, where the user can review the generated results and perform further processing as needed. For example, the server may display the generated results on the user's dashboard and allow the user to download the results.

[0907] Specific examples

[0908] Example prompt sentence:

[0909] "Summarize the following sentence: 'Hello, this is a test.'"

[0910] "Please translate the following sentence: 'Hello, this is a test.'"

[0911] This allows users to use generative AI models to perform a variety of tasks, such as summarizing and translating transcripts.

[0912] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0913] Step 1:

[0914] The server receives audio data sent from the user or device. The audio data is provided in an audio file format (e.g., WAV, MP3). The server inputs this audio data into transcription software (e.g., Google Speech-to-Text API) and outputs text data. Specifically, the server sends the audio file to the API and obtains the returned text data.

[0915] Step 2:

[0916] The server preprocesses the received text data. This preprocessing removes unnecessary noise and special characters and formats the data. The input is transcribed text data, and the output is formatted text data. Specifically, it uses regular expressions to remove special characters and converts the data into the required format.

[0917] Step 3:

[0918] The server selects the generative AI model to use. For example, it initializes a deep learning model such as BERT or GPT-3. The input is formatted text data, and the output is the model's initialized state. Specifically, it loads the model library and sets the necessary parameters.

[0919] Step 4:

[0920] The server generates prompt sentences to be input to the generative AI model. Prompt sentences instruct the model on specific tasks. The input is formatted text data, and the output is the generated prompt sentence. The specific operation is to create the prompt sentence using a template.

[0921] Step 5:

[0922] The server inputs the generated prompt sentence and preprocessed data into the generative AI model. The input is the prompt sentence and formatted text data, and the output is the analysis result from the generative AI model. Specifically, the server inputs the prompt sentence and text data into the model and obtains the analysis result.

[0923] Step 6:

[0924] The server receives the output from the generative AI model. The model generates appropriate output based on the input prompt. The input is the prompt and formatted text data, and the output is the generated text data. The specific operation is to receive the results returned from the model.

[0925] Step 7:

[0926] The server then post-processes the generated output, formatting it as needed and adding additional information. The input is the generated text data, and the output is the final formatted data, specifically converting it to JSON and adding any necessary metadata.

[0927] Step 8:

[0928] The server outputs the final results to the user or to a device. The user can review the generated results and perform further processing if necessary. The input is the final formatted data, and the output is the results provided to the user. Specific operations include displaying the results on the user's dashboard and providing a download link.

[0929] (Application example 2)

[0930] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0931] In today's business environment, improving meeting efficiency and gathering participant opinions are important issues. However, conventional systems have difficulty visualizing the mood and energy level of a meeting, and they have not yet realized the automatic generation of content based on the meeting content. For this reason, there is a need for a system that can improve the effectiveness of meeting progress and gathering participant opinions, as well as automatically generate content based on the meeting content.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0933] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores for included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for automatically generating content based on the voice data, and a means for displaying the generated content. This not only makes it possible to improve the efficiency of meetings and effectively gather participants' opinions, but also enables the automatic generation of content based on the meeting content.

[0934] "Audio data" refers to data in which audio is recorded in digital format.

[0935] "Transcription" is the process of converting audio data into text data.

[0936] "Generative AI" is artificial intelligence that uses deep learning models to generate new text and content.

[0937] An "emotion score" is a numerical representation of the intensity and type of emotion associated with a word or phrase in a text.

[0938] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[0939] "Energy level" is an indicator of the level of activity and enthusiasm of participants in a meeting or conversation.

[0940] "Visualization" is the process of visually displaying data or information.

[0941] "Opinion gathering" is the process of summarizing the opinions of multiple participants to arrive at a single conclusion or direction.

[0942] "Automatic content generation" is the process of automatically generating text and media based on input data.

[0943] "Display" is the process of visually presenting generated content to a user.

[0944] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, means for automatically generating content based on the voice data, and means for displaying the generated content.

[0945] System configuration

[0946] Hardware

[0947] Smartphone: Used for voice input and display.

[0948] Server: Used to process generative AI models.

[0949] software

[0950] Speech recognition technology: Used to transcribe voice data.

[0951] Deep learning models (BERT and GPT-3): Used as generative AI.

[0952] Sentiment analysis tools: Used to extract sentiment scores from text data.

[0953] Data visualization tools: Used to visualize the mood and energy level of a meeting.

[0954] Data processing and calculation

[0955] 1. Speech recognition: The smartphone's microphone is used to collect the user's voice, which is then transcribed using speech recognition technology.

[0956] 2. Extract sentiment scores: The transcribed text data is fed into a sentiment analysis tool to extract sentiment scores for words and phrases.

[0957] 3. Sentiment inference: The extracted sentiment scores are summed or averaged to infer the overall sentiment.

[0958] 4. Content generation using generative AI: Transcribed text data is fed into a generative AI model to automatically generate content such as blog posts and news articles.

[0959] 5. Displaying content: The generated content is displayed on the smartphone screen.

[0960] Specific examples

[0961] When a user speaks into their smartphone, "I want to write a blog post about today's weather," the system transcribes the speech and generates text data that reads, "I want to write a blog post about today's weather." This text data is then fed into a sentiment analysis tool to extract a sentiment score. The text data is then fed into a generative AI model, which automatically generates a blog post like the one below.

[0962] "Today's forecast is for sunny skies with occasional clouds. We will have a clear blue sky in the morning, with clouds gathering in the afternoon, but there is no need to worry about rain. The temperature is expected to be a comfortable day with a maximum of 25 degrees and a minimum of 15 degrees."

[0963] Prompt Sentence Examples

[0964] If the user says "I want to write a blog post about today's weather," the prompt would be:

[0965] "I want to write a blog post about today's weather."

[0966] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0967] Step 1:

[0968] The user speaks into the smartphone.

[0969] Input: User's voice data

[0970] Output: Audio data

[0971] Specific action: The user speaks into the smartphone microphone, "I want to write a blog post about today's weather."

[0972] Step 2:

[0973] The smartphone uses voice recognition technology to transcribe the audio data.

[0974] Input: Audio data

[0975] Output: Transcribed text data

[0976] How it works: The smartphone's voice recognition software analyzes the voice data and generates text data such as, "I want to write a blog post about today's weather."

[0977] Step 3:

[0978] The server inputs the transcribed text data into a sentiment analysis tool to extract a sentiment score.

[0979] Input: Transcribed text data

[0980] Output: Sentiment score

[0981] What it does: The server sends the text data to a sentiment analysis tool, which extracts a numerical sentiment score for each word or phrase.

[0982] Step 4:

[0983] The server sums or averages the emotion scores to infer the overall emotion.

[0984] Input: Sentiment score

[0985] Output: Overall sentiment score

[0986] Specific operation: The server sums or averages the extracted emotion scores to calculate the overall emotional state as a number.

[0987] Step 5:

[0988] The server inputs the transcribed text data into a generative AI model to automatically generate content.

[0989] Input: Transcribed text data

[0990] Output: Generated content

[0991] How it works: The server sends text data to a generative AI model, which then generates content such as blog posts and news articles.

[0992] Step 6:

[0993] The smartphone displays the generated content.

[0994] Input: Generated content

[0995] Output: The displayed content

[0996] Specific operation: The generated blog post is displayed on the smartphone screen and the user can view it.

[0997] Example 3

[0998] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0999] Conventional meeting systems make it difficult to quantitatively grasp the mood and energy level of a meeting, and lack specific indicators for improving meeting progress and gathering participants' opinions. Furthermore, sentiment analysis requires a lot of manual work, making efficient operation difficult.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1001] In this invention, the server includes: means for acquiring text data representing the voice data of each participant in the conference; means for extracting emotion scores for words or phrases included in the text data using a prompt statement and a generative AI model; means for estimating the overall emotion score by summing or averaging the extracted emotion scores; and means for visualizing changes in the estimated emotion scores. This allows for a quantitative understanding of the mood and energy level of the conference, thereby improving the progress of the conference and the effectiveness of gathering participants' opinions.

[1002] "Audio data" refers to data in which audio is recorded in digital format.

[1003] "Transcription" is the process of converting audio data into text data.

[1004] A "generative AI model" is an artificial intelligence that has the ability to generate new data based on given data.

[1005] An "emotion score" is a numerical value that indicates the intensity of emotion calculated for a word or phrase in text data.

[1006] A "sentiment analysis tool" is software or algorithms for extracting sentiment scores from text data.

[1007] A "dashboard" is an interface for visually displaying data.

[1008] "Meeting mood" refers to the overall atmosphere and emotional state felt during the meeting.

[1009] "Energy level" is an indicator of the liveliness and positivity of participants during a meeting.

[1010] A "user" is a person or organization that operates the system and inputs data.

[1011] A "server" is a computer system that processes and stores data.

[1012] "Text data" is digital data that includes character information.

[1013] "Summing or averaging" is the process of adding together or calculating the average of several numbers.

[1014] This invention is a system for quantitatively understanding the mood and energy level of a meeting, improving the effectiveness of meeting progress and gathering participants' opinions. The system transcribes audio data, extracts emotion scores using generative artificial intelligence, sums or averages the scores to estimate the overall emotion, and visualizes them on a dashboard.

[1015] Hardware and software used

[1016] Hardware: Servers (e.g., cloud computing services)

[1017] Software: Sentiment analysis tools (e.g., VADER), dashboard display tools (e.g., Grafana)

[1018] Program processing explanation

[1019] Users input text data, such as meeting minutes or chat logs, into the system. Input is done through web forms or dedicated applications. The server receives the text data sent by the user and temporarily stores it. The server then uses VADER, a sentiment analysis tool, to extract sentiment scores for each word or phrase in the received text data. VADER calculates sentiment scores as positive, negative, or neutral.

[1020] The server then sums or averages the sentiment scores for each extracted word or phrase to calculate an overall sentiment score, which quantitatively evaluates the sentiment of the entire text. Finally, the server displays the changes in the calculated overall sentiment score on a dashboard. The dashboard, built using a tool such as Grafana, visually shows the mood and energy level of the meeting.

[1021] Specific examples

[1022] An example of a prompt that a user might enter into a generative AI model is:

[1023] After inputting text data representing the voice data of each participant in the meeting, the following question is asked: "This text data is a meeting transcript. Please extract the sentiment scores of words or phrases contained in the meeting transcript."

[1024] Example of meeting minutes:

[1025] Participant A: "This project is going well."

[1026] Participant B: "There are some challenges, but we have found solutions."

[1027] Participant C: “Overall, there has been a lot of positive feedback.”

[1028] When this prompt sentence is input into the generative AI model, the emotion score of the words or phrases contained in each utterance is extracted. The server calculates the total emotion score for each utterance by summing or averaging the emotion scores of the words or phrases contained in that utterance. The server then estimates the change in the total emotion score calculated for each utterance as a change in the overall mood and displays it as a graph on the dashboard. For example, if there are many positive emotions, the graph rises, and if there are many negative emotions, the graph falls. This allows the progress of the meeting and the effectiveness of gathering participants' opinions to be visually confirmed. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1029] Step 1:

[1030] The user inputs text data.

[1031] Users enter text data such as meeting minutes and chat logs into the system through web forms or dedicated applications, and the entered text data is sent to the server.

[1032] Input: Text data such as meeting minutes and chat logs

[1033] Output: Text data sent to the server

[1034] Step 2:

[1035] The server receives the text data.

[1036] The server receives the text data sent by the user and temporarily stores it in a database, for example, a MySQL (registered trademark) database.

[1037] Input: Text data sent by the user

[1038] Output: Text data stored in the database

[1039] Step 3:

[1040] The server extracts the sentiment score using a generative AI model or sentiment analysis tool.

[1041] The server passes the stored text data to a generative AI model or sentiment analysis tool (e.g., VADER) to calculate a sentiment score for each word or phrase. VADER calculates a sentiment score as positive, negative, or neutral.

[1042] Input: Text data stored in a database

[1043] Output: Sentiment score for each word or phrase

[1044] Step 4:

[1045] The server sums or averages the sentiment scores for each utterance.

[1046] For each utterance, the server calculates an overall sentiment score by summing or averaging the sentiment scores of each word or phrase extracted from that utterance. For example, if the positive score is 3 and the negative score is 1, the overall sentiment score is 2.

[1047] Input: Sentiment scores for each word or phrase

[1048] Output: Overall sentiment score

[1049] Step 5:

[1050] The server displays the changes in the estimated emotion scores on a dashboard.

[1051] The server displays the calculated changes in the overall emotion score on a dashboard (e.g., Grafana). Users can visually check the mood and energy level of the meeting through the dashboard. For example, if there is a lot of positive emotion, the graph will rise, and if there is a lot of negative emotion, the graph will fall.

[1052] Input: Overall sentiment score

[1053] Output: A graph of the sentiment scores displayed in a dashboard The server may then perform processing to detect a peak in the change in the estimated emotion score, and when a peak is detected, may notify that it is time to summarize the conference. Alternatively, the server may estimate and present the progress of the meeting using a generative AI model and a prompt sentence that instructs the server to estimate the progress of the meeting based on changes in the estimated emotion score.

[1054] (Application example 3)

[1055] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1056] There is a lack of means to monitor the emotional state of workers in factories in real time to improve work efficiency and safety. Conventional methods make it difficult to properly grasp workers' moods and energy levels, making it difficult to optimize the work environment. Therefore, there is a need for a system that visualizes workers' emotional state and allows managers to respond quickly.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1058] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included in the data, which allows visualization of the mood and energy levels of workers and enables optimization of the work environment.

[1059] "Audio data" refers to data in which audio is recorded in digital format.

[1060] "Transcription" is the process of converting audio data into text form.

[1061] "Generative AI" is an artificial intelligence technology that generates new data based on input data.

[1062] An "emotion score" is a numerical representation of the intensity of emotion associated with a particular word or phrase.

[1063] "Mood" refers to the emotional state of a worker at a particular time.

[1064] "Energy level" is an indicator of a worker's vitality and fatigue level.

[1065] "Visualization" is the process of displaying data in a visual format such as a graph or chart.

[1066] "Work environment optimization" refers to adjusting the work environment to improve worker efficiency and safety.

[1067] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI system, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy levels of workers, and means for optimizing the work environment.

[1068] System configuration

[1069] This system uses the following hardware and software:

[1070] Hardware: Smart glasses (e.g., Google Glass)

[1071] Software: Python, VADER Sentiment Analysis, Matplotlib

[1072] Program processing explanation

[1073] The server first acquires the audio data and transcribes it using speech recognition technology. The transcribed text data is then fed into a generative AI system, which extracts sentiment scores for the words and phrases contained within. This sentiment score is derived using VADER Sentiment Analysis. The extracted sentiment scores are then summed or averaged to estimate the overall sentiment.

[1074] The server then visualizes the workers' moods and energy levels based on the inferred emotional data, using Matplotlib to generate graphs and charts, allowing managers to understand the workers' emotional state in real time and optimize the work environment.

[1075] Specific examples

[1076] For example, if a worker says, "I'm very tired today," the voice data is sent through the smart glasses to a server. The server uses speech recognition technology to convert this speech into text and feeds it into a generative AI system. The generative AI then uses VADER Sentiment Analysis to extract an emotion score, which then visualizes the worker's mood and energy level.

[1077] Example prompt sentence:

[1078] Calculate the emotion score when a worker says, "I'm very tired today," and display it in a graph.

[1079] By inputting this prompt into a generative AI model, an emotion score based on the worker's utterance is calculated and visualized, allowing managers to quickly understand the worker's emotional state and take appropriate action.

[1080] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1081] Step 1:

[1082] The server receives voice data from the smart glasses. The input is the worker's speech, and the output is digital voice data. The server stores this voice data.

[1083] Step 2:

[1084] The server converts the voice data into text data using speech recognition technology. The input is the voice data, and the output is the transcribed text data. Specifically, the server uses speech recognition software (e.g., Google Speech-to-Text API) to analyze the voice and generate the corresponding text.

[1085] Step 3:

[1086] The server inputs text data into the generative AI. The input is the transcribed text data, and the output is an emotion score. Specifically, the emotion score of the text data is calculated using VADER Sentiment Analysis.

[1087] Step 4:

[1088] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. Specifically, each emotion score is aggregated as a numerical value, and the sum or average is calculated.

[1089] Step 5:

[1090] The server visualizes the mood and energy levels of workers based on the estimated overall emotion score. The input is the overall emotion score, and the output is a graph or chart. Specifically, Matplotlib is used to visually display the emotion score.

[1091] Step 6:

[1092] The server provides visualized data to managers to optimize the work environment. The input is graphs and charts, and the output is notifications and reports to managers. Specifically, graphs are displayed on a dashboard, allowing managers to understand the emotional state of workers in real time.

[1093] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1094] "Example 1"

[1095] In the first embodiment, a voice recognition technology is used as a means for transcribing voice data. Specifically, the voice data is converted into text data using a voice recognition API such as Google's.

[1096] "Example 2"

[1097] In Example 2, a deep learning model is used as a means for inputting data into generative AI. Specifically, text data is analyzed using models such as BERT and GPT-3.

[1098] "Example 3"

[1099] In the third embodiment, an emotion engine is used as a means for extracting emotion scores of included words and phrases. Specifically, emotion scores are extracted from text data using an emotion analysis tool such as VADER.

[1100] "Example 4"

[1101] In the fourth embodiment, an emotion engine is used as a means for inferring the overall emotion by summing or averaging the extracted emotion scores. Specifically, the overall emotion is inferred by summing or averaging the extracted emotion scores.

[1102] "Example 5"

[1103] In the fifth embodiment, a dashboard is used as a means for visualizing the mood and energy level of the meeting. Specifically, the mood and energy level of the meeting are displayed as a graph on the dashboard using the estimated overall emotions.

[1104] "Example 6"

[1105] In the sixth embodiment, an emotion engine is used as a means for improving the efficiency of proceeding with a meeting and gathering opinions of participants. Specifically, emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, thereby improving the efficiency of proceeding with the meeting and gathering opinions.

[1106] The processing flow of each embodiment will be described below.

[1107] "Example 1"

[1108] Step 1: Step 1: Collect audio data during the meeting.

[1109] Step 2: Step 2: Convert the collected voice data into text data using Google's voice recognition API or similar.

[1110] "Example 2"

[1111] Step 1: Step 1: Convert the audio data into text data.

[1112] Step 2: Step 2: The converted text data is fed into a deep learning model such as BERT or GPT-3 and analyzed.

[1113] "Example 3"

[1114] Step 1: Step 1: Analyze the text data.

[1115] Step 2: Step 2: Extract sentiment scores from the analyzed text data using a sentiment analysis tool such as VADER.

[1116] "Example 4"

[1117] Step 1: Step 1: Extract sentiment scores from text data.

[1118] Step 2: Step 2: Sum or average the extracted emotion scores to infer the overall emotion.

[1119] "Example 5"

[1120] Step 1:Step 1: Guess the overall emotion.

[1121] Step 2: Step 2: Using the estimated overall sentiment, the mood and energy level of the meeting are displayed as a graph on a dashboard.

[1122] "Example 6"

[1123] Step 1:Step 1:Visualize the mood and energy level of the meeting.

[1124] Step 2: Step 2: The emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, improving the efficiency of meeting progress and opinion gathering.

[1125] Example 1

[1126] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1127] Conventional meeting systems have difficulty accurately recording participants' comments and grasping the mood and energy level of the meeting in real time. They also lack the means to effectively manage the meeting and gather participants' opinions. This leads to problems such as reduced meeting efficiency and impaired decision-making quality.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1129] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data using voice recognition technology, means for inputting the converted text data into generative artificial intelligence, means for extracting emotion scores for the words and phrases included, means for estimating the overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of the meeting, and means for improving the progress of the meeting and the collection of participants' opinions. This makes it possible to accurately record comments made in the meeting and grasp the mood and energy level of the meeting in real time. Furthermore, it is possible to effectively progress the meeting and collect opinions, thereby improving the efficiency of the meeting and the quality of decision-making.

[1130] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[1131] "Voice recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[1132] "Character data" refers to data in text format that has been converted using voice recognition technology.

[1133] "Generative AI" is an AI technology that uses deep learning models to generate new information and predictions from input data.

[1134] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1135] "Mood" refers to the overall atmosphere and emotional state of the meeting.

[1136] "Energy level" is an indicator of the liveliness of a meeting and the energy of the participants.

[1137] "Visualization" is a technique that makes data easier to understand by visually displaying it.

[1138] "Convergence" is the process of summarizing the opinions of meeting participants and arriving at a common conclusion or direction.

[1139] "Effectiveness" is the optimization of processes and means to achieve a specific purpose.

[1140] MODE FOR CARRYING OUT THE INVENTION

[1141] The present invention relates to a system for acquiring speech data from conference participants and transcribing the data using speech recognition technology. Specific embodiments of this system will be described below.

[1142] Acquiring audio data

[1143] When a user speaks during a meeting, the device picks up the voice through a microphone. This voice data is sent to the server in real time. The device is equipped with a high-quality microphone and also has a noise-canceling function.

[1144] Sending audio data

[1145] The voice data acquired by the device is sent to a server via the Internet. The server receives the voice data and proceeds to the next processing step. HTTP, WebSocket, or other communication protocols are used.

[1146] Use of voice recognition technology

[1147] The server converts the received voice data into text data using speech recognition technology. Specifically, it uses Google's speech recognition API, which can convert voice to text with high accuracy.

[1148] Use of generative artificial intelligence

[1149] The converted text data is then fed into a generative AI system that uses deep learning models to extract sentiment scores for the words and phrases contained within the text data.

[1150] Extracting and visualizing emotion scores

[1151] The server sums or averages the emotion scores to infer an overall emotion. This information is used to visualize the mood and energy level of the meeting. Visualization can be done using graphs or charts.

[1152] Meeting progress and opinion gathering

[1153] The server provides a means for effectively managing the progress of meetings and gathering opinions from participants, for example, through real-time feedback and voting functions, which can improve the efficiency of meetings.

[1154] Specific examples

[1155] For example, if a user says "Let's move on to the next topic" during a meeting, the device's microphone captures this audio. The device then sends this audio data to the server in real time. The server then sends the received audio data to Google's speech recognition API, receiving the text data "Let's move on to the next topic." The server then inputs this text data into a generative artificial intelligence system to extract an emotion score. Ultimately, the system visualizes the mood and energy level of the meeting, helping to guide the meeting.

[1156] Prompt Sentence Examples

[1157] Examples of prompts to input to a generative AI model might include:

[1158] "Design a system that transcribes speech from meeting participants in real time. Include the process of capturing audio data, converting it to text using Google's speech recognition API, and storing it in a database."

[1159] In this way, by clarifying the specific embodiment of the system, it becomes possible to accurately understand and practice the technical scope of the invention.

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] When a user speaks during a conference, the device captures the voice through a microphone. The input is the user's speech, and the output is digital voice data. The device transmits this voice data to the server in real time. Specifically, the device's microphone captures the voice and generates voice data.

[1163] Step 2:

[1164] The voice data acquired by the device is sent to a server via the Internet. The input is the voice data sent from the device, and the output is the voice data received by the server. The server receives this voice data and proceeds to the next processing step. Specifically, data is sent and received using communication protocols such as HTTP or WebSocket.

[1165] Step 3:

[1166] The server converts the received voice data into text data using voice recognition technology. The input is the voice data received by the server, and the output is the text data returned from the voice recognition API. Specifically, the server sends the voice data to Google's voice recognition API and receives the text data returned from the API.

[1167] Step 4:

[1168] The server inputs the converted text data into the generative AI. The input is the text data returned from the speech recognition API, and the output is the emotion score analyzed by the generative AI. Specifically, the server inputs the text data into the generative AI and extracts the emotion score.

[1169] Step 5:

[1170] The server sums or averages the emotion scores to estimate the overall emotion. The input is the emotion score obtained from the generative AI, and the output is numerical data indicating the overall emotion. Specifically, the server aggregates the emotion scores and calculates the overall emotion.

[1171] Step 6:

[1172] The server visualizes the mood and energy level of the meeting. The input is numerical data that indicates the overall emotion, and the output is a visual representation such as a graph or chart. Specifically, the server converts the data into a graph or chart and displays it to the user.

[1173] Step 7:

[1174] The server provides a means to effectively manage the progress of the meeting and gather participants' opinions. The input is the progress of the meeting and the opinions of the participants, and the output is real-time feedback and voting results. Specifically, the server collects feedback in real time and gathers opinions through the voting function.

[1175] In this way, by clarifying the specific operations performed at each processing step and their inputs and outputs, it becomes easier to understand the overall processing flow of the system.

[1176] (Application example 1)

[1177] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1178] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Also, when work instructions are given by voice in a factory, there is no way to accurately record those instructions and manage them as a work log, making it difficult to accurately grasp the progress of work and the content of instructions. For this reason, there is a need for a system that can improve the efficiency of meetings and accurately record and manage work instructions in a factory.

[1179] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores of included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for acquiring work instructions by voice and transcribing and recording them in real time, and a means for managing a work log. This enables more efficient meetings and accurate recording and management of work instructions in the factory.

[1180] "Audio data" refers to data in which audio information such as conference and work instructions is recorded in digital format.

[1181] "Transcription" is the process of converting audio data into text data.

[1182] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new information and results from input data.

[1183] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in audio or text data.

[1184] "Mood" refers to the overall atmosphere and emotional state of a meeting or work environment.

[1185] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or work environment.

[1186] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[1187] "Work instructions" are voice or text information that instructs the content and procedures of work to be done in a factory.

[1188] "Real-time" refers to the instantaneous processing of data and provision of information.

[1189] A "work log" is data that records work instructions and progress within a factory.

[1190] "Management" is the process of organizing data and information and storing and using them appropriately.

[1191] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of a meeting and the collection of opinions of participants, means for obtaining work instructions via voice and transcribing and recording them in real time, and means for managing a work log.

[1192] System program processing explanation

[1193] Acquiring and transcribing audio data

[1194] The server uses a microphone to capture audio data, which is recorded using the Python speech_recognition library and converted to text using the Google Cloud Speech-to-Text API.

[1195] Extracting and visualizing emotion scores

[1196] Generative AI uses natural language processing techniques to extract sentiment scores for words and phrases contained in text data. The extracted sentiment scores are summed or averaged to infer overall sentiment. The inferred sentiment is visualized as the mood or energy level of the meeting.

[1197] Real-time transcription of work instructions and work log management

[1198] The server receives voice instructions from within the factory and transcribes them in real time. This allows for accurate recording of work instructions. The recorded work instructions are managed as a work log. This makes it possible to accurately grasp the progress of work and the content of instructions.

[1199] Examples of specific examples and prompts

[1200] As a concrete example, consider work instructions in a factory. For example, when a worker issues a voice command such as "Please proceed to the next process," the command is transcribed in real time and recorded in a work log.

[1201] Example prompts to input to a generative AI model:

[1202] Transcribe voice instructions such as "Proceed to the next step" in real time and record them in your work log.

[1203] Using this prompt, the generative AI model can accurately transcribe the voice instructions and take appropriate action to record them in the work log.

[1204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1205] Step 1: Acquire audio data

[1206] The server captures voice data using a microphone. When a user speaks into the microphone, the voice is recorded in digital format. The input is the user's voice, and the output is digital voice data. This voice data is captured using the Python speech_recognition library.

[1207] Step 2: Transcribe the audio data

[1208] The server sends the acquired voice data to the Google Cloud Speech-to-Text API for transcription. The input is digital voice data, and the output is text data. The Google Cloud Speech-to-Text API analyzes the voice data and generates corresponding text data.

[1209] Step 3: Extract sentiment scores

[1210] The server uses generative AI to extract sentiment scores for words and phrases contained in text data. The input is text data, and the output is a sentiment score. Natural language processing technology is used to quantify the emotional nuance of each word or phrase.

[1211] Step 4: Sum or average sentiment scores

[1212] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. This allows us to numerically express the mood and energy level of the meeting.

[1213] Step 5: Visualize emotions

[1214] The server visualizes the inferred overall emotion score. The input is the overall emotion score, and the output is a visual graph or chart, which provides a visual understanding of the mood and energy level of the meeting.

[1215] Step 6: Real-time transcription of work instructions

[1216] The server receives voice instructions from the factory and transcribes them in real time. The input is the user's voice instructions, and the output is text data. Voice recognition technology is used to instantly convert voice instructions into text data.

[1217] Step 7: Manage your work log

[1218] The server manages the transcribed work instructions as a work log. The input is text data, and the output is an organized work log. This allows the progress of work and the content of instructions to be accurately recorded and managed.

[1219] Example 2

[1220] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1221] Conventional methods for recording meetings and interviews not only transcribe audio data, but also make it difficult to efficiently analyze the content and extract important information. Furthermore, there is a lack of a way to visualize the mood and energy level of the meeting and effectively summarize the opinions of participants. This leads to problems such as a decrease in the efficiency of meeting progress and decision-making.

[1222] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1223] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the collection of participant opinions, means for inputting the transcribed data into the generative AI model and analyzing the text data using a deep learning model, means for generating prompts for the generative AI model and inputting them together with the text data, and means for providing the generated information to the user. This enables efficient analysis of voice data, extraction of important information, visualization of the mood and energy level of the meeting, and effective collection of participant opinions.

[1224] "Audio data" refers to data in which audio is recorded in digital format.

[1225] "Transcription" is the process of converting audio data into text data.

[1226] A "generative AI model" is an artificial intelligence model that uses deep learning technology to analyze text data and generate new information.

[1227] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1228] "Mood" refers to the overall atmosphere or emotional state of a meeting or interview.

[1229] "Energy level" refers to the level of activity and enthusiasm of participants in a meeting or interview.

[1230] A "prompt" is an instruction entered into a generative AI model to make it perform a specific task.

[1231] A "deep learning model" is a machine learning model that uses a multi-layered neural network to analyze data and learn patterns.

[1232] "Text data" is digital data made up of letters and symbols.

[1233] "User" means a person or organization that uses the system to transcribe and analyze audio data.

[1234] A "server" is a computer that is the central part of a system and is a device that processes and stores data.

[1235] MODE FOR CARRYING OUT THE INVENTION

[1236] This invention is a system that transcribes audio data and analyzes it using a generative AI model. A specific embodiment of this system is described below.

[1237] System configuration

[1238] This system consists of a user, a server, and a device. The user provides voice data and accesses the server through the device. The server transcribes the voice data, analyzes it using a generative AI model, and provides the results.

[1239] Hardware and software used

[1240] Hardware: Servers, devices (PCs, smartphones, etc.)

[1241] Software: speech recognition technology (e.g., Google Speech-to-Text API), generative AI models (e.g., BERT, GPT-3)

[1242] Data processing and calculation

[1243] 1. Transcription of audio data:

[1244] Users can record audio data from meetings or interviews on their device and transcribe it using speech recognition technology. For example, by using the Google Speech-to-Text API, the audio data can be converted into text data with high accuracy.

[1245] 2. Upload the transcription data:

[1246] Users upload the transcribed text data to the server via their device, using a web interface or a dedicated upload tool.

[1247] 3. Analysis using generative AI models:

[1248] The server receives the uploaded text data and feeds it into a generative AI model, specifically a deep learning model such as BERT or GPT-3. The server generates appropriate prompts for these models and feeds them along with the text data.

[1249] 4. Prompt generation:

[1250] The server generates a prompt for the generative AI model, such as:

[1251] Prompt: "Please summarize the following meeting minutes. Transcript: [transcribed text data]"

[1252] 5. Provision of generated information:

[1253] The generative AI model analyzes the prompt and text data to generate the necessary information. For example, it extracts a summary of the meeting or key points. The server then provides the generated information to the user. The user can view the generated summary and analysis results through a web interface or a dedicated dashboard.

[1254] Specific examples

[1255] For example, a user records audio data from a meeting and transcribes it using the Google Speech-to-Text API. The user then uploads the text data to a server. The server then inputs the uploaded text data into the BERT model to extract key points. The server then uses a generative AI model (GPT-3) to generate a summary based on the extracted points. The user can then view the generated summary through a web interface and download it if necessary.

[1256] In this way, the present invention efficiently analyzes voice data and extracts important information, visualizes the mood and energy level of a meeting, and effectively aggregates participants' opinions.

[1257] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1258] Step 1:

[1259] The user records the audio data.

[1260] Specifically, the user records the audio of a meeting or interview using a recording device (such as a smartphone or a dedicated recorder). The input is audio data, and the output is a recorded audio file.

[1261] Step 2:

[1262] The user transcribes the audio data.

[1263] Specifically, the user converts the recorded audio file into text data using speech recognition technology (e.g., Google Speech-to-Text API). The input is the audio file, and the output is the transcribed text data.

[1264] Step 3:

[1265] The user uploads the transcription data to the server.

[1266] Specifically, users upload transcribed text data to the server using a web interface or a dedicated upload tool. The input is the transcribed text data, and the output is the text data stored on the server.

[1267] Step 4:

[1268] The server feeds the transcription data into a generative AI model.

[1269] Specifically, the server receives the uploaded text data, generates an appropriate prompt for the generative AI model, and inputs it along with the text data. The input is the text data stored on the server and the generated prompt, and the output is the data input to the generative AI model.

[1270] Step 5:

[1271] A generative AI model analyzes the text data and generates the necessary information.

[1272] Specifically, a generative AI model (e.g., BERT, GPT-3) analyzes the prompt and text data to extract a summary or key points. The input is the text data and prompt entered into the generative AI model, and the output is the generated summary or analysis results.

[1273] Step 6:

[1274] The server provides the generated information to the user.

[1275] Specifically, the server provides the user with the summary and analysis results obtained from the generative AI model. The user can check the generated information through a web interface or a dedicated dashboard and download it as needed. The input is the generated summary and analysis results, and the output is the information provided to the user.

[1276] (Application example 2)

[1277] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1278] Conventional conference systems and content generation systems lack the functionality to not only transcribe audio data but also to analyze text data using generative AI and automatically generate optimal content for users. Furthermore, there are insufficient means for users to edit, save, and share the generated content. This creates the problem of ineffective progress in meetings and the gathering of opinions from participants.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1280] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the collection of participant opinions, means for transcribing voice input by the user in real time, means for inputting the transcribed data into a generative AI and analyzing the text data, means for automatically generating optimal content for the user based on the analysis results, and means for allowing the user to edit, save, and share the generated content. This not only improves the progress of the meeting and the collection of participant opinions, but also enables the user to automatically generate optimal content based on voice input, and edit, save, and share it.

[1281] "Audio data" refers to data in which audio is recorded in digital format.

[1282] "Transcription" is the process of converting audio data into text data.

[1283] "Generative AI" is artificial intelligence that uses deep learning models to generate or analyze text data.

[1284] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1285] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[1286] "Energy level" is an indicator of the level of activity and enthusiasm of participants in a meeting or conversation.

[1287] "Visualization" is the process of visually displaying data or information.

[1288] "Progress" refers to the way a meeting or conversation is conducted and its progress.

[1289] "Convergence" is the process of summarizing the opinions of multiple participants.

[1290] "Real-time" refers to processing or reaction occurring immediately, without delay.

[1291] "Analysis" is the process of examining data in detail to understand its meaning and structure.

[1292] "Content" is a collection of information or data, including articles, blogs, social media posts, etc.

[1293] "Editing" is the process of modifying or changing generated content.

[1294] "Preservation" is the process of recording generated content in a digital format so that it can be accessed at a later time.

[1295] "Sharing" is the process of sharing generated content with other users.

[1296] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for included words and phrases, means for inferring overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of meeting progress and gathering opinions of participants, means for transcribing voice input by a user in real time, means for inputting the transcribed data into a generative AI and analyzing the text data, means for automatically generating optimal content for the user based on the analysis results, and means for users to edit, save, and share the generated content.

[1297] Hardware and software used

[1298] Hardware: Smartphone (microphone, internet connection)

[1299] Software: Python, speech_recognition library, transformers library

[1300] Data processing and calculation

[1301] 1. Speech Recognition: The user inputs speech through the smartphone microphone. The speech data is transcribed in real time using the speech_recognition library.

[1302] 2. Text analysis: The transcribed text data is fed into a generative AI (such as BERT or GPT-3) to analyze the text data. The transformers library is used for analysis.

[1303] 3. Extracting emotion scores: Generative AI extracts emotion scores for words and phrases contained in text data, then sums or averages them to infer the overall emotion.

[1304] 4. Content generation: Based on the analysis results, the most suitable content for the user (articles, blogs, social media posts, etc.) is automatically generated.

[1305] 5. Edit, save and share: Generated content can be edited, saved and shared by users.

[1306] Specific examples

[1307] When a user says, "I want to write a blog about today's weather," the system performs the following process:

[1308] 1. Speech recognition: The user's speech is transcribed and text data such as "I want to write a blog about today's weather" is generated.

[1309] 2. Text analysis: Generative AI analyzes this text data and gathers relevant information.

[1310] 3. Content generation: Based on the analysis results, blog articles like the following are automatically generated.

[1311] "Today's forecast is for sunny skies with occasional cloudy spells. This morning the sky was clear and the temperature was comfortable, but the clouds will increase in the afternoon and it may feel a bit chilly. It's a good idea to bring a light jacket with you when you go out."

[1312] Prompt Sentence Examples

[1313] "I want to blog about today's weather."

[1314] I want to write a review of a new product.

[1315] "I want to plan a trip"

[1316] In this way, generative AI can automatically generate content based on the user's voice input, which can then be edited, saved, and shared.

[1317] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1318] Step 1:

[1319] The user speaks into the smartphone's microphone, and the speech data is captured and transcribed in real time using the speech_recognition library.

[1320] Input: User's voice data

[1321] Data processing: Converting voice data into text data

[1322] Output: Transcribed text data

[1323] Step 2:

[1324] The server then feeds the transcribed text data into a generative AI (such as BERT or GPT-3), which analyzes the text data and extracts sentiment scores for the words and phrases it contains.

[1325] Input: Transcribed text data

[1326] Data Computing: Analyzing text data and extracting sentiment scores

[1327] Output: Text data with sentiment scores

[1328] Step 3:

[1329] The server sums or averages the extracted emotion scores to infer the overall emotion.

[1330] Input: Text data with sentiment scores

[1331] Data operations: summing or averaging sentiment scores

[1332] Output: Overall sentiment score

[1333] Step 4:

[1334] The server visualizes the mood and energy level of the meeting based on the overall emotional score.

[1335] Input: Overall sentiment score

[1336] Data processing: Visualization of emotion scores

[1337] Output: Visualized data on mood and energy levels

[1338] Step 5:

[1339] The server uses generative AI to automatically generate optimal content based on the voice input provided by the user.

[1340] Input: What the user says

[1341] Data Computation: Content Generation by Generative AI

[1342] Output: Auto-generated content

[1343] Step 6:

[1344] Users can edit, save, and share the generated content. Through a smartphone interface, users can modify the generated content, save it digitally, and share it with other users.

[1345] Input: Auto-generated content

[1346] Data processing: Editing, saving, and sharing content

[1347] Output: Edited, saved, and shared content

[1348] In this way, specific data processing and calculations are performed at each step, allowing users to generate optimal content based on their voice input, which can then be edited, saved, and shared.

[1349] Example 3

[1350] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1351] Conventional meeting systems make it difficult to grasp the mood and energy level of a meeting in real time, which means that meeting progress and the gathering of participants' opinions are not efficiently managed. Furthermore, extracting and visualizing emotion scores is often done manually, which is time-consuming and labor-intensive.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1353] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative artificial intelligence, means for extracting emotion scores for included words and phrases, means for estimating overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, means for inputting text data, means for extracting emotion scores using an emotion analysis tool, means for aggregating the extracted emotion scores, and means for displaying the scores as a graph on a dashboard. This allows the mood and energy level of the meeting to be grasped in real time, enabling efficient meeting management.

[1354] "Audio data" refers to data in which audio is recorded in digital format.

[1355] "Transcription" is the process of converting audio data into text data.

[1356] "Generative AI" is an AI technology that generates new data and information based on input data.

[1357] An "emotion score" is a numerical representation of positive, negative, neutral, or other emotions generated by analyzing words and phrases in text.

[1358] A "sentiment analysis tool" is software or algorithms for extracting sentiment scores from text data.

[1359] A "dashboard" is an interface for visually displaying data.

[1360] A "graph" is a graphic representation of data visually.

[1361] "Meeting mood" refers to the overall emotional atmosphere and tone of the participants in a meeting.

[1362] "Energy level" is an indicator of the liveliness and positivity of participants in a meeting.

[1363] "Opinion gathering" is the process of summarizing the opinions of participants in a meeting.

[1364] "Efficiency" refers to making processes and tasks more effective and faster.

[1365] MODE FOR CARRYING OUT THE INVENTION

[1366] This invention is a system for efficiently managing meetings by grasping the mood and energy level of the meeting in real time. This system transcribes audio data, extracts emotion scores using generative artificial intelligence, and visualizes them, allowing for effective progress in the meeting and the collection of opinions of participants.

[1367] Hardware and software used

[1368] Hardware:

[1369] server

[1370] Devices (computers and smartphones used by users)

[1371] software:

[1372] Speech recognition technology (to transcribe voice data)

[1373] Generative AI (to analyze text data)

[1374] Sentiment analysis tools (e.g., VADER)

[1375] Dashboard (interface for visualizing data)

[1376] Data processing and calculation

[1377] A user inputs text data such as minutes of a meeting, a chat log, etc. into a terminal. For example, the user copies the minutes of a meeting and pastes them into an input field of the system.

[1378] The server receives the input text data and analyzes it using a sentiment analysis tool such as VADER. Specifically, the server passes the text data to VADER, which extracts a sentiment score for each word or phrase. For example, the word "good" has a positive score, and the word "bad" has a negative score.

[1379] The server sums or averages the extracted emotion scores to infer the overall emotion. Specifically, the server aggregates the emotion scores of each word or phrase as a numerical value and calculates an overall positive and negative score. For example, if the positive score is 70% and the negative score is 30%, the server infers that the overall mood is positive.

[1380] The server then displays the estimated overall sentiment as a graph on the dashboard. Specifically, the server uses the aggregated sentiment scores to plot the percentage of positive and negative sentiment on the dashboard as a pie chart or bar graph. For example, it visually shows that the positive mood is 70% and the negative mood is 30%.

[1381] Users can use the information on the dashboard to streamline meeting progress and gather participant opinions. Specifically, users can look at the dashboard and take measures such as changing the agenda if the mood remains negative. For example, if there are a lot of positive opinions, they can continue the discussion as is, but if there are a lot of negative opinions, they can identify problems and consider solutions.

[1382] Examples of specific examples and prompts

[1383] Examples:

[1384] The user inputs the minutes of the meeting into the system, for example, "Today's meeting discussed the development of a new product. Many participants had positive opinions, but some participants expressed concerns."

[1385] The server analyzes this text using VADER and extracts a positive score of 0.7 and a negative score of 0.3.

[1386] The server aggregates these scores and estimates the overall mood as 70% positive and 30% negative.

[1387] The server displays the results as a pie chart on a dashboard.

[1388] The user looks at the dashboard, sees that there are many positive opinions, and decides to proceed with the discussion.

[1389] Example prompt sentence:

[1390] "Extract a sentiment score from the following text data, infer the overall sentiment, and display it on a dashboard. Text data: 'Today's meeting discussed the development of a new product. Many participants had positive opinions, but some expressed concerns.'"

[1391] In this way, the user can visualize the mood and energy level of the meeting by utilizing generative AI, and can manage the meeting effectively. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1392] Step 1:

[1393] A user inputs text data such as minutes of a meeting, a chat log, etc. into a terminal. Specifically, the user copies the minutes of the meeting and pastes them into an input field of the system.

[1394] Input: Text data such as meeting minutes and chat logs

[1395] Output: The input text data

[1396] Step 2:

[1397] The server receives the input text data and analyzes it using a sentiment analysis tool (e.g., VADER). Specifically, the server passes the text data to VADER, which extracts a sentiment score for each word or phrase. For example, the word "good" has a positive score, and the word "bad" has a negative score.

[1398] Input: Entered text data

[1399] Output: Sentiment score for each word or phrase

[1400] Step 3:

[1401] The server sums or averages the extracted emotion scores to infer the overall emotion. Specifically, the server aggregates the emotion scores of each word or phrase as a numerical value and calculates an overall positive and negative score. For example, if the positive score is 70% and the negative score is 30%, the server infers that the overall mood is positive.

[1402] Input: Sentiment scores for each word or phrase

[1403] Output: Overall sentiment score (positive and negative scores)

[1404] Step 4:

[1405] The server then displays the estimated overall sentiment as a graph on the dashboard. Specifically, the server uses the aggregated sentiment scores to plot the percentage of positive and negative sentiment on the dashboard as a pie chart or bar graph. For example, it visually shows that the positive mood is 70% and the negative mood is 30%.

[1406] Input: Overall sentiment score (positive and negative scores)

[1407] Output: Graphical display on the dashboard

[1408] Step 5:

[1409] Users can use the information on the dashboard to streamline meeting progress and gather participant opinions. Specifically, users can look at the dashboard and take measures such as changing the agenda if the mood remains negative. For example, if there are a lot of positive opinions, they can continue the discussion as is, but if there are a lot of negative opinions, they can identify problems and consider solutions.

[1410] Input: Graph display on dashboard

[1411] Output: Efficient meeting progress and opinion gathering

[1412] (Application example 3)

[1413] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1414] In the work environment within a factory, there is a need to understand the communication and working mood of workers in real time in order to improve work efficiency and safety. However, conventional systems have difficulty accurately understanding the emotions and mood of workers, which has led to issues such as insufficient efficiency in work progress and opinion gathering.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1416] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores for included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the work environment, a means for improving the efficiency of work progress and the collection of worker opinions, a means for monitoring worker conversations in real time, and a means for displaying emotion scores on a dashboard. This makes it possible to grasp the emotions and moods of workers in real time and improve work efficiency and safety.

[1417] "Audio data" refers to data in which audio is recorded in digital format.

[1418] "Transcription" is the process of converting audio data into text data.

[1419] "Generative AI" is an artificial intelligence technology that generates new data from given data.

[1420] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1421] "Mood" is the overall atmosphere or emotional state in a particular environment or situation.

[1422] "Energy level" refers to the degree of activity or vitality in a particular environment or situation.

[1423] "Visualization" is the process of visually displaying data or information.

[1424] "Real-time" means that data and information are processed and displayed immediately.

[1425] A "dashboard" is an interface for displaying data and information in a centralized manner.

[1426] "Monitoring" is the process of continuously observing and recording specific data or conditions.

[1427] "Opinion gathering" is the process of bringing together multiple opinions and ideas to arrive at a single conclusion or direction.

[1428] A system for carrying out the present invention monitors and visualizes in real time the communication and working mood of workers in a work environment within a factory. A specific embodiment of this system will be described below.

[1429] System Configuration

[1430] The system consists of the following main components:

[1431] 1. Transcribing voice data: Using voice recognition technology, workers' conversations are converted into text data in real time.

[1432] 2. Method of inputting data into generative AI: Input the transcribed text data into generative AI.

[1433] 3. Extracting sentiment scores: Using sentiment analysis tools such as VADER, sentiment scores are extracted from text data.

[1434] 4. Means of inferring overall sentiment: The extracted sentiment scores are summed or averaged to infer overall sentiment.

[1435] 5. A way to visualize mood and energy levels: The estimated overall emotions are displayed as graphs on a dashboard.

[1436] 6. A means to improve the efficiency of work progress and opinion gathering: Based on the information in the dashboard, work progress and the gathering of opinions from workers can be made more efficient.

[1437] Hardware and software used

[1438] Hardware: Microphones and displays mounted on factory robots.

[1439] Software: Python, vaderSentiment library, matplotlib library.

[1440] Data processing and calculation

[1441] The server transcribes the voice data acquired from the factory robots in real time and inputs it into the generative AI. The generative AI then uses a sentiment analysis tool such as VADER to extract sentiment scores from the text data. The extracted sentiment scores are summed or averaged to infer overall sentiment. The inferred sentiment is displayed as a graph on a dashboard, helping to improve the efficiency of work progress and opinion gathering.

[1442] Specific examples

[1443] For example, consider a system that monitors the conversations of workers in a factory in real time and displays an emotion score on a dashboard. This system makes it possible to grasp the emotions and moods of workers in real time and improve work efficiency and safety.

[1444] Example prompts to input to the generative AI model

[1445] Design a system that monitors the conversations of factory workers in real time, extracts emotion scores, and displays them on a dashboard. Use VADER to extract emotion scores and average the overall emotion to display them. Create graphs using Python and matplotlib.

[1446] By inputting this prompt into the generative AI model, more detailed design and implementation methods can be suggested.

[1447] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1448] Step 1:

[1449] The server acquires voice data from microphones installed on factory robots. The input is real-time voice data, and the output is digital voice data, which is used for subsequent transcription processing.

[1450] Step 2:

[1451] The server uses speech recognition technology to transcribe the audio data. The input is digital audio data, and the output is text data. This text data is then fed into generative AI.

[1452] Step 3:

[1453] The server feeds text data into the generative AI. The input is transcribed text data, and the output is data ready for extracting emotion scores.

[1454] Step 4:

[1455] The server extracts sentiment scores from text data using sentiment analysis tools such as VADER. The input is the text data fed into the generative AI, and the output is the sentiment score for each word or phrase.

[1456] Step 5:

[1457] The server sums or averages the extracted emotion scores to infer an overall emotion. The input is the emotion score for each word or phrase, and the output is an overall emotion score, which indicates the mood or energy level of the work environment.

[1458] Step 6:

[1459] The server displays the estimated overall sentiment score as a graph on a dashboard. The input is the overall sentiment score, and the output is a visually displayed graph. This graph can be used to streamline work progress and opinion gathering.

[1460] Step 7:

[1461] Users can streamline work progress and opinion gathering based on the graphs on the dashboard. The input is the graph on the dashboard, and the output is the streamlined work progress and opinion gathering.

[1462] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1464] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1465] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1466] [Third embodiment]

[1467] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1468] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32.

[1470] The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1471] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1472] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1474] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1475] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1478] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1479] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1480] "Example 1"

[1481] In one embodiment of the present invention, speech data from participants in a conference is acquired and transcribed using speech recognition technology. This transcription can be performed using, for example, Google's speech recognition API.

[1482] "Example 2"

[1483] The transcribed data is then fed into a generative AI, which is designed using a deep learning model, such as BERT or GPT-3.

[1484] "Example 3"

[1485] Generative AI extracts sentiment scores for words and phrases, sums or averages them, and infers the overall sentiment. This sentiment score can be extracted using a sentiment analysis tool such as VADER.

[1486] "Example 4"

[1487] Finally, the mood and energy level of the meeting can be visualized, for example, by displaying it as a graph on a dashboard. This can help improve the effectiveness of the meeting and the gathering of participants' opinions.

[1488] The processing flow of each embodiment will be described below.

[1489] "Example 1"

[1490] Step 1: The speech of the meeting participants is acquired as audio data using a microphone or the like.

[1491] Step 2: The acquired voice data is transcribed using speech recognition technology. This transcription can be performed using, for example, Google's speech recognition API.

[1492] "Example 2"

[1493] Step 1: Input the transcribed data from Example 1 into the generative AI.

[1494] Step 2: Generative AI is designed using deep learning models, such as BERT or GPT-3.

[1495] "Example 3"

[1496] Step 1: Extract sentiment scores for the words and phrases contained in the data input into the generative AI in Example 2.

[1497] Step 2: The extracted emotion scores are summed or averaged to estimate the overall emotion. This emotion score can be extracted using an emotion analysis tool such as VADER. "Example 4"

[1498] Step 1: Visualize the mood and energy level of the meeting using the overall emotions inferred in Example 3.

[1499] Step 2: This visualization can be done, for example, by displaying it as a graph on a dashboard, which will help improve the effectiveness of the meeting and the gathering of participants' opinions.

[1500] Example 1

[1501] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1502] In conventional meetings, it was difficult to accurately record what participants said and share it in real time. Furthermore, there was a lack of efficient ways to proceed with the meeting and gather participants' opinions. This led to a decrease in meeting efficiency and the risk of important information being overlooked.

[1503] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server, means for the server to transcribe the voice data using voice recognition technology, means for storing the transcribed data in a database, means for transmitting the stored data to a terminal and displaying it, and means for efficiently progressing the meeting and collecting opinions of participants. This makes it possible to accurately record and share the contents of statements made in the meeting in real time. Furthermore, since the meeting can be progressed and opinions can be efficiently collected, the efficiency of the meeting is improved and important information can be prevented from being overlooked.

[1504] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[1505] The "means of collection" refers to a method of acquiring voice data using a microphone or voice input device.

[1506] The "means of transmission" refers to a method for transferring collected voice data to a server via a network.

[1507] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1508] "Transcribing means" refers to a method of converting audio data into text data using voice recognition technology.

[1509] "Database" means a digital storage system for storing transcribed text data.

[1510] "Means for storage" refers to the method for storing the transcribed text data in a database.

[1511] "Display means" refers to a method for displaying text data on the terminal screen.

[1512] "Proceeding a meeting" is the process of advancing the discussion in line with the meeting agenda.

[1513] "Convergence" is the process of summarizing the opinions of meeting participants and reaching a common conclusion.

[1514] "Means for improving effectiveness" are methods and tools for efficiently conducting meetings and gathering opinions.

[1515] The present invention provides a system for transcribing speeches made by participants in a meeting in real time, thereby improving the efficiency of the progress of the meeting and the collection of opinions. Specific embodiments of this system will be described below.

[1516] Hardware and software used

[1517] Hardware

[1518] Microphone: Used to collect the speech of meeting participants.

[1519] Terminal: A device such as a personal computer, tablet, or smartphone that collects and transmits voice data to a server.

[1520] Server: A computer system that receives audio data and transcribes it using speech recognition technology.

[1521] software

[1522] Audio collection application: An application that is installed on a device to collect audio data and send it to a server.

[1523] Speech recognition API: An API for converting voice data into text data. Specifically, it uses common voice recognition technology.

[1524] Database: A digital storage system for storing transcribed text data.

[1525] System Operation

[1526] server

[1527] The server receives the voice data sent from the device. The received voice data is sent to a voice recognition API and converted into text data. The converted text data is saved in a database. The saved text data is then sent to the device and displayed in real time.

[1528] Terminal

[1529] The device collects audio data via a microphone when participants speak, and sends the collected audio data to a server via a voice collection application. The text data returned from the server is displayed on the device screen, allowing users to view it in real time.

[1530] User

[1531] Users speak during the meeting. Their speech is collected as audio data through the device's microphone. Users can then see their speech being transcribed on the device's screen.

[1532] Specific examples

[1533] For example, if a user says "Let's discuss the next item on the agenda" during a meeting, the device's microphone will collect this voice and send it to the server via a voice collection application. The server will then use a speech recognition API to convert this voice data into text data, generating the text "Let's discuss the next item on the agenda." This text data will then be stored in a database and sent to the device. Finally, the text "Let's discuss the next item on the agenda" will be displayed on the device screen, allowing the user to view it in real time.

[1534] Prompt Sentence Examples

[1535] "Design a system that transcribes meeting participants' speech in real time. Include a process for collecting audio data, converting it to text using speech recognition technology, and displaying it on a device screen."

[1536] This system is very useful for improving meeting efficiency and automating the creation of meeting minutes.

[1537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1538] Step 1: Collecting audio data

[1539] The device collects audio data through a microphone when a participant speaks. Specifically, the device's audio collection application digitally captures the audio input from the microphone and saves it as an audio file. The input is the participant's speech, and the output is a digital audio file.

[1540] Step 2: Sending audio data

[1541] The device sends the collected voice data to a server. Specifically, the voice collection application uploads the voice file to the server via the network. The input is a digital voice file, and the output is the voice data sent to the server.

[1542] Step 3: Receiving audio data

[1543] The server receives the voice data sent from the terminal. Specifically, the server's receiving module receives the voice file sent over the network and stores it in the server. The input is the voice data sent from the terminal, and the output is the voice data stored in the server.

[1544] Step 4: Transcribe the audio data

[1545] The server transcribes the received voice data using speech recognition technology. Specifically, the server calls a speech recognition API and converts the voice data into text data. The input is the voice data stored on the server, and the output is the text data returned by the speech recognition API.

[1546] Step 5: Save the text data

[1547] The server saves the transcribed text data in a database. Specifically, the server's database module stores the text data in the database. The input is the text data returned from the speech recognition API, and the output is the text data saved in the database.

[1548] Step 6: Send text data

[1549] The server transmits the stored text data to the terminal. Specifically, the server's transmission module retrieves the text data from the database and transmits it to the terminal via the network. The input is the text data stored in the database, and the output is the text data transmitted to the terminal.

[1550] Step 7: Displaying Text Data

[1551] The terminal receives the text data sent from the server and displays it on the screen. Specifically, the display application of the terminal receives the text data via the network and displays it on the user interface. The input is the text data sent from the server, and the output is the text data displayed on the terminal screen.

[1552] Through this series of processes, the remarks of meeting participants are transcribed in real time and displayed on the device screen, allowing for efficient progress in the meeting and the collection of opinions.

[1553] (Application example 1)

[1554] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1555] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Furthermore, when workers give voice instructions to machines at work sites such as factories, there is no system that can accurately recognize those instructions and perform the appropriate operation. This creates the problem of insufficient progress in meeting efficiency and automation of on-site work.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1557] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included. This makes it possible to visualize the mood and energy level of a meeting, and to improve the effectiveness of meeting progress and gathering participants' opinions. Furthermore, by including a means for analyzing voice data and controlling machine operation based on specific keywords, workers in factories and other workplaces can issue voice instructions to machines, which then perform appropriate operations based on those instructions.

[1558] "Audio data" refers to digital data of audio collected at meetings, work sites, etc.

[1559] "Transcription" is the process of converting audio data into text data.

[1560] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new data and information from input data.

[1561] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1562] "Meeting mood" refers to the emotional atmosphere of all participants during the meeting.

[1563] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or at a workplace.

[1564] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[1565] "Controlling the operation of a machine" means instructing a machine to perform a specific operation based on voice data or text data.

[1566] "Voice recognition technology" is a technology that analyzes voice data and converts it into text data.

[1567] A "deep learning model" is a type of artificial intelligence that uses multi-layered neural networks to analyze and learn from data.

[1568] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, and means for analyzing the voice data and controlling the operation of a machine based on specific keywords.

[1569] The server acquires the audio data and transcribes it using speech recognition technology. For example, Google's speech recognition API is used as the speech recognition technology. The transcribed text data is then input into a generative AI system, which extracts emotion scores for the words and phrases contained within. The emotion scores are then summed or averaged to estimate the overall emotion. This makes it possible to visualize the mood and energy level of the meeting, thereby improving the effectiveness of meeting progress and gathering participant opinions.

[1570] Furthermore, the server analyzes the voice data and controls the machine's operation based on specific keywords. For example, if a worker in a factory says, "Please proceed to the next process," the robot will proceed to the next process. In this case, speech recognition technology is used to convert the voice into text, and the robot's operation is controlled based on that text.

[1571] The hardware used includes a microphone and a factory robot, while the software uses Python, the speech_recognition library, and Google's speech recognition API.

[1572] For example, in a factory, when a worker says "Please proceed to the next step," the robot will proceed to the next step. The following is an example of a prompt:

[1573] When you say "Please proceed to the next step," create a program that will make the robot proceed to the next step. Use Google's speech recognition API to convert speech into text and control the robot's actions based on that text.

[1574] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1575] Step 1:

[1576] The server captures voice data through a microphone: the input is the user's voice, and the output is digital voice data, which is used in subsequent processing steps.

[1577] Step 2:

[1578] The server transcribes the acquired voice data using speech recognition technology. Specifically, it converts the voice data into text data using Google's speech recognition API. The input is the voice data acquired in step 1, and the output is text data.

[1579] Step 3:

[1580] The server inputs the transcribed text data into a generative AI and extracts sentiment scores for the words and phrases contained within. The input is the text data obtained in step 2, and the output is the sentiment score for each word or phrase. The generative AI calculates the sentiment scores using a deep learning model.

[1581] Step 4:

[1582] The server sums or averages the extracted emotion scores to infer the overall emotion. The input is the emotion scores obtained in step 3, and the output is an overall emotion score. This overall emotion score is an indicator of the mood and energy level of the meeting.

[1583] Step 5:

[1584] The server visualizes the overall emotion score. The input is the overall emotion score obtained in step 4, and the output is a visually displayed emotion score. This helps to improve the progress of the meeting and the effectiveness of gathering participants' opinions.

[1585] Step 6:

[1586] The server analyzes the voice data and controls the machine's operation based on specific keywords. The input is the text data obtained in step 2, and the output is instructions for the machine's operation. For example, if the keyword "Please proceed to the next step" is included, the robot will proceed to the next step.

[1587] Step 7:

[1588] The server executes the machine's operations. The input is the operation instructions obtained in step 6, and the output is the actual machine operation. Specifically, the server performs operations such as the robot proceeding to the next process or installing parts.

[1589] Example 2

[1590] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1591] Conventional conference systems are required not only to transcribe audio data but also to visualize the mood and energy level of the meeting and effectively gather opinions from participants. However, there are few systems that provide these functions in an integrated manner, and in particular, they lack advanced analysis and output using generative AI models. For this reason, a new system is needed to improve the progress of meetings and the effectiveness of gathering opinions from participants.

[1592] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1593] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the effectiveness of meeting progress and participant opinion gathering, means for preprocessing the transcription data, means for generating prompt sentences for the generative AI model, means for acquiring generation results from the generative AI model, means for post-processing the generation results, and means for outputting the generation results, thereby enabling the meeting to be progressed effectively and participant opinions to be gathered.

[1594] "Audio data" refers to data in which audio is recorded in digital format.

[1595] "Transcription" is the process of converting audio data into text data.

[1596] A "generative AI model" is an artificial intelligence model that generates text data using deep learning technology.

[1597] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1598] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[1599] "Energy level" refers to the level of activity and enthusiasm of participants in a meeting or conversation.

[1600] "Preprocessing" is the initial process of preparing data in a format suitable for analysis and processing.

[1601] A "prompt sentence" is an input sentence that instructs a generative AI model on a specific task.

[1602] "Post-processing" refers to the process of preparing the generated data in a final output format.

[1603] "Output" refers to providing processed data to a user or other system.

[1604] MODE FOR CARRYING OUT THE INVENTION

[1605] This invention relates to a system that transcribes voice data and analyzes and outputs the results using a generative AI model. A specific embodiment of this system is described below.

[1606] System configuration

[1607] The server system consists of the following hardware and software:

[1608] Hardware:

[1609] High-performance processor

[1610] Large capacity memory

[1611] High-speed storage devices

[1612] software:

[1613] Voice recognition technology (e.g., Google Speech-to-Text API)

[1614] Generative AI models (e.g., BERT, GPT-3)

[1615] Scripts for data pre- and post-processing

[1616] Program processing

[1617] The server receives the voice data sent from the user or device and performs the following processes:

[1618] 1. Transcription of audio data:

[1619] The server converts the audio data into text using speech recognition technology, for example, by using the Google Speech-to-Text API to transcribe the audio file.

[1620] 2. Data preprocessing:

[1621] The server preprocesses the transcription data and removes unnecessary noise and special characters. For example, it converts the transcription data "Hello, this is a test" into "Hello, this is a test."

[1622] 3. Generative AI model selection:

[1623] The server selects the generative AI model to use, for example, a deep learning model such as BERT or GPT-3.

[1624] 4. Prompt generation:

[1625] The server generates prompts to be input to the generative AI model. Prompts instruct the model on specific tasks. For example, it generates a prompt such as "Please summarize the following sentence: 'Hello, this is a test.'"

[1626] 5. Inputting data into generative AI models:

[1627] The server inputs the generated prompt and preprocessed data into a generative AI model. For example, the server inputs the prompt "Please summarize the following sentence: 'Hello, this is a test'" into the BERT model.

[1628] 6. Obtaining the generated results:

[1629] The server receives the output from the generative AI model, which generates an appropriate output based on the input prompt. For example, the server receives the output "Hello, this is the test result" from the BERT model.

[1630] 7. Post-processing of results:

[1631] The server post-processes the generated output and formats it as needed, for example converting the output to JSON format and adding necessary metadata.

[1632] 8. Result output:

[1633] The server outputs the final results to the user or to a terminal, where the user can review the generated results and perform further processing as needed. For example, the server may display the generated results on the user's dashboard and allow the user to download the results.

[1634] Specific examples

[1635] Example prompt sentence:

[1636] "Summarize the following sentence: 'Hello, this is a test.'"

[1637] "Please translate the following sentence: 'Hello, this is a test.'"

[1638] This allows users to use generative AI models to perform a variety of tasks, such as summarizing and translating transcripts.

[1639] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1640] Step 1:

[1641] The server receives audio data sent from the user or device. The audio data is provided in an audio file format (e.g., WAV, MP3). The server inputs this audio data into transcription software (e.g., Google Speech-to-Text API) and outputs text data. Specifically, the server sends the audio file to the API and obtains the returned text data.

[1642] Step 2:

[1643] The server preprocesses the received text data. This preprocessing removes unnecessary noise and special characters and formats the data. The input is transcribed text data, and the output is formatted text data. Specifically, it uses regular expressions to remove special characters and converts the data into the required format.

[1644] Step 3:

[1645] The server selects the generative AI model to use. For example, it initializes a deep learning model such as BERT or GPT-3. The input is formatted text data, and the output is the model's initialized state. Specifically, it loads the model library and sets the necessary parameters.

[1646] Step 4:

[1647] The server generates prompt sentences to be input to the generative AI model. Prompt sentences instruct the model on specific tasks. The input is formatted text data, and the output is the generated prompt sentence. The specific operation is to create the prompt sentence using a template.

[1648] Step 5:

[1649] The server inputs the generated prompt sentence and preprocessed data into the generative AI model. The input is the prompt sentence and formatted text data, and the output is the analysis result from the generative AI model. Specifically, the server inputs the prompt sentence and text data into the model and obtains the analysis result.

[1650] Step 6:

[1651] The server receives the output from the generative AI model. The model generates appropriate output based on the input prompt. The input is the prompt and formatted text data, and the output is the generated text data. The specific operation is to receive the results returned from the model.

[1652] Step 7:

[1653] The server then post-processes the generated output, formatting it as needed and adding additional information. The input is the generated text data, and the output is the final formatted data, specifically converting it to JSON and adding any necessary metadata.

[1654] Step 8:

[1655] The server outputs the final results to the user or to a device. The user can review the generated results and perform further processing if necessary. The input is the final formatted data, and the output is the results provided to the user. Specific operations include displaying the results on the user's dashboard and providing a download link.

[1656] (Application example 2)

[1657] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1658] In today's business environment, improving meeting efficiency and gathering participant opinions are important issues. However, conventional systems have difficulty visualizing the mood and energy level of a meeting, and they have not yet realized the automatic generation of content based on the meeting content. For this reason, there is a need for a system that can improve the effectiveness of meeting progress and gathering participant opinions, as well as automatically generate content based on the meeting content.

[1659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1660] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores for included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for automatically generating content based on the voice data, and a means for displaying the generated content. This not only makes it possible to improve the efficiency of meetings and effectively gather participants' opinions, but also enables the automatic generation of content based on the meeting content.

[1661] "Audio data" refers to data in which audio is recorded in digital format.

[1662] "Transcription" is the process of converting audio data into text data.

[1663] "Generative AI" is artificial intelligence that uses deep learning models to generate new text and content.

[1664] An "emotion score" is a numerical representation of the intensity and type of emotion associated with a word or phrase in a text.

[1665] "Mood" refers to the overall atmosphere or emotional state of a meeting or conversation.

[1666] "Energy level" is an indicator of the level of activity and enthusiasm of participants in a meeting or conversation.

[1667] "Visualization" is the process of visually displaying data or information.

[1668] "Opinion gathering" is the process of summarizing the opinions of multiple participants to arrive at a single conclusion or direction.

[1669] "Automatic content generation" is the process of automatically generating text and media based on input data.

[1670] "Display" is the process of visually presenting generated content to a user.

[1671] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of the meeting and the collection of opinions of participants, means for automatically generating content based on the voice data, and means for displaying the generated content.

[1672] System configuration

[1673] Hardware

[1674] Smartphone: Used for voice input and display.

[1675] Server: Used to process generative AI models.

[1676] software

[1677] Speech recognition technology: Used to transcribe voice data.

[1678] Deep learning models (BERT and GPT-3): Used as generative AI.

[1679] Sentiment analysis tools: Used to extract sentiment scores from text data.

[1680] Data visualization tools: Used to visualize the mood and energy level of a meeting.

[1681] Data processing and calculation

[1682] 1. Speech recognition: The smartphone's microphone is used to collect the user's voice, which is then transcribed using speech recognition technology.

[1683] 2. Extract sentiment scores: The transcribed text data is fed into a sentiment analysis tool to extract sentiment scores for words and phrases.

[1684] 3. Sentiment inference: The extracted sentiment scores are summed or averaged to infer the overall sentiment.

[1685] 4. Content generation using generative AI: Transcribed text data is fed into a generative AI model to automatically generate content such as blog posts and news articles.

[1686] 5. Displaying content: The generated content is displayed on the smartphone screen.

[1687] Specific examples

[1688] When a user speaks into their smartphone, "I want to write a blog post about today's weather," the system transcribes the speech and generates text data that reads, "I want to write a blog post about today's weather." This text data is then fed into a sentiment analysis tool to extract a sentiment score. The text data is then fed into a generative AI model, which automatically generates a blog post like the one below.

[1689] "Today's forecast is for sunny skies with occasional clouds. We will have a clear blue sky in the morning, with clouds gathering in the afternoon, but there is no need to worry about rain. The temperature is expected to be a comfortable day with a maximum of 25 degrees and a minimum of 15 degrees."

[1690] Prompt Sentence Examples

[1691] If the user says "I want to write a blog post about today's weather," the prompt would be:

[1692] "I want to write a blog post about today's weather."

[1693] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1694] Step 1:

[1695] The user speaks into the smartphone.

[1696] Input: User's voice data

[1697] Output: Audio data

[1698] Specific action: The user speaks into the smartphone microphone, "I want to write a blog post about today's weather."

[1699] Step 2:

[1700] The smartphone uses voice recognition technology to transcribe the audio data.

[1701] Input: Audio data

[1702] Output: Transcribed text data

[1703] How it works: The smartphone's voice recognition software analyzes the voice data and generates text data such as, "I want to write a blog post about today's weather."

[1704] Step 3:

[1705] The server inputs the transcribed text data into a sentiment analysis tool to extract a sentiment score.

[1706] Input: Transcribed text data

[1707] Output: Sentiment score

[1708] What it does: The server sends the text data to a sentiment analysis tool, which extracts a numerical sentiment score for each word or phrase.

[1709] Step 4:

[1710] The server sums or averages the emotion scores to infer the overall emotion.

[1711] Input: Sentiment score

[1712] Output: Overall sentiment score

[1713] Specific operation: The server sums or averages the extracted emotion scores to calculate the overall emotional state as a number.

[1714] Step 5:

[1715] The server inputs the transcribed text data into a generative AI model to automatically generate content.

[1716] Input: Transcribed text data

[1717] Output: Generated content

[1718] How it works: The server sends text data to a generative AI model, which then generates content such as blog posts and news articles.

[1719] Step 6:

[1720] The smartphone displays the generated content.

[1721] Input: Generated content

[1722] Output: The displayed content

[1723] Specific operation: The generated blog post is displayed on the smartphone screen and the user can view it.

[1724] Example 3

[1725] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1726] Conventional meeting systems make it difficult to quantitatively grasp the mood and energy level of a meeting, and lack specific indicators for improving meeting progress and gathering participants' opinions. Furthermore, sentiment analysis requires a lot of manual work, making efficient operation difficult.

[1727] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1728] In this invention, the server includes: means for acquiring text data representing the voice data of each participant in the conference; means for extracting emotion scores for words or phrases included in the text data using a prompt statement and a generative AI model; means for estimating the overall emotion score by summing or averaging the extracted emotion scores; and means for visualizing changes in the estimated emotion scores. This allows for a quantitative understanding of the mood and energy level of the conference, thereby improving the progress of the conference and the effectiveness of gathering participants' opinions.

[1729] "Audio data" refers to data in which audio is recorded in digital format.

[1730] "Transcription" is the process of converting audio data into text data.

[1731] A "generative AI model" is an artificial intelligence that has the ability to generate new data based on given data.

[1732] An "emotion score" is a numerical value that indicates the intensity of emotion calculated for a word or phrase in text data.

[1733] A "sentiment analysis tool" is software or algorithms for extracting sentiment scores from text data.

[1734] A "dashboard" is an interface for visually displaying data.

[1735] "Meeting mood" refers to the overall atmosphere and emotional state felt during the meeting.

[1736] "Energy level" is an indicator of the liveliness and positivity of participants during a meeting.

[1737] A "user" is a person or organization that operates the system and inputs data.

[1738] A "server" is a computer system that processes and stores data.

[1739] "Text data" is digital data that includes character information.

[1740] "Summing or averaging" is the process of adding together or calculating the average of several numbers.

[1741] This invention is a system for quantitatively understanding the mood and energy level of a meeting, improving the effectiveness of meeting progress and gathering participants' opinions. The system transcribes audio data, extracts emotion scores using generative artificial intelligence, sums or averages the scores to estimate the overall emotion, and visualizes them on a dashboard.

[1742] Hardware and software used

[1743] Hardware: Servers (e.g., cloud computing services)

[1744] Software: Sentiment analysis tools (e.g., VADER), dashboard display tools (e.g., Grafana)

[1745] Program processing explanation

[1746] Users input text data, such as meeting minutes or chat logs, into the system. Input is done through web forms or dedicated applications. The server receives the text data sent by the user and temporarily stores it. The server then uses VADER, a sentiment analysis tool, to extract sentiment scores for each word or phrase in the received text data. VADER calculates sentiment scores as positive, negative, or neutral.

[1747] The server then sums or averages the sentiment scores for each extracted word or phrase to calculate an overall sentiment score, which quantitatively evaluates the sentiment of the entire text. Finally, the server displays the changes in the calculated overall sentiment score on a dashboard. The dashboard, built using a tool such as Grafana, visually shows the mood and energy level of the meeting.

[1748] Specific examples

[1749] An example of a prompt that a user might enter into a generative AI model is:

[1750] After inputting text data representing the voice data of each participant in the meeting, the following question is asked: "This text data is a meeting transcript. Please extract the sentiment scores of words or phrases contained in the meeting transcript."

[1751] Example of meeting minutes:

[1752] Participant A: "This project is going well."

[1753] Participant B: "There are some challenges, but we have found solutions."

[1754] Participant C: “Overall, there has been a lot of positive feedback.”

[1755] When this prompt sentence is input into the generative AI model, the emotion score of the words or phrases contained in each utterance is extracted. The server calculates the total emotion score for each utterance by summing or averaging the emotion scores of the words or phrases contained in that utterance. The server then estimates the change in the total emotion score calculated for each utterance as a change in the overall mood and displays it as a graph on the dashboard. For example, if there are many positive emotions, the graph rises, and if there are many negative emotions, the graph falls. This allows the progress of the meeting and the effectiveness of gathering participants' opinions to be visually confirmed. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1756] Step 1:

[1757] The user inputs text data.

[1758] Users enter text data such as meeting minutes and chat logs into the system through web forms or dedicated applications, and the entered text data is sent to the server.

[1759] Input: Text data such as meeting minutes and chat logs

[1760] Output: Text data sent to the server

[1761] Step 2:

[1762] The server receives the text data.

[1763] The server receives the text data sent by the user and temporarily stores it in a database, for example, a MySQL (registered trademark) database.

[1764] Input: Text data sent by the user

[1765] Output: Text data stored in the database

[1766] Step 3:

[1767] The server extracts the sentiment score using a generative AI model or sentiment analysis tool.

[1768] The server passes the stored text data to a generative AI model or sentiment analysis tool (e.g., VADER) to calculate a sentiment score for each word or phrase. VADER calculates a sentiment score as positive, negative, or neutral.

[1769] Input: Text data stored in a database

[1770] Output: Sentiment score for each word or phrase

[1771] Step 4:

[1772] The server sums or averages the sentiment scores for each utterance.

[1773] For each utterance, the server calculates an overall sentiment score by summing or averaging the sentiment scores of each word or phrase extracted from that utterance. For example, if the positive score is 3 and the negative score is 1, the overall sentiment score is 2.

[1774] Input: Sentiment scores for each word or phrase

[1775] Output: Overall sentiment score

[1776] Step 5:

[1777] The server displays the changes in the estimated emotion scores on a dashboard.

[1778] The server displays the calculated changes in the overall emotion score on a dashboard (e.g., Grafana). Users can visually check the mood and energy level of the meeting through the dashboard. For example, if there is a lot of positive emotion, the graph will rise, and if there is a lot of negative emotion, the graph will fall.

[1779] Input: Overall sentiment score

[1780] Output: A graph of the sentiment scores displayed in a dashboard The server may then perform processing to detect a peak in the change in the estimated emotion score, and when a peak is detected, may notify that it is time to summarize the conference. Alternatively, the server may estimate and present the progress of the meeting using a generative AI model and a prompt sentence that instructs the server to estimate the progress of the meeting based on changes in the estimated emotion score.

[1781] (Application example 3)

[1782] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1783] There is a lack of means to monitor the emotional state of workers in factories in real time to improve work efficiency and safety. Conventional methods make it difficult to properly grasp workers' moods and energy levels, making it difficult to optimize the work environment. Therefore, there is a need for a system that visualizes workers' emotional state and allows managers to respond quickly.

[1784] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1785] In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, and a means for extracting emotion scores for the words and phrases included in the data, which allows visualization of the mood and energy levels of workers and enables optimization of the work environment.

[1786] "Audio data" refers to data in which audio is recorded in digital format.

[1787] "Transcription" is the process of converting audio data into text form.

[1788] "Generative AI" is an artificial intelligence technology that generates new data based on input data.

[1789] An "emotion score" is a numerical representation of the intensity of emotion associated with a particular word or phrase.

[1790] "Mood" refers to the emotional state of a worker at a particular time.

[1791] "Energy level" is an indicator of a worker's vitality and fatigue level.

[1792] "Visualization" is the process of displaying data in a visual format such as a graph or chart.

[1793] "Work environment optimization" refers to adjusting the work environment to improve worker efficiency and safety.

[1794] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI system, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy levels of workers, and means for optimizing the work environment.

[1795] System configuration

[1796] This system uses the following hardware and software:

[1797] Hardware: Smart glasses (e.g., Google Glass)

[1798] Software: Python, VADER Sentiment Analysis, Matplotlib

[1799] Program processing explanation

[1800] The server first acquires the audio data and transcribes it using speech recognition technology. The transcribed text data is then fed into a generative AI system, which extracts sentiment scores for the words and phrases contained within. This sentiment score is derived using VADER Sentiment Analysis. The extracted sentiment scores are then summed or averaged to estimate the overall sentiment.

[1801] The server then visualizes the workers' moods and energy levels based on the inferred emotional data, using Matplotlib to generate graphs and charts, allowing managers to understand the workers' emotional state in real time and optimize the work environment.

[1802] Specific examples

[1803] For example, if a worker says, "I'm very tired today," the voice data is sent through the smart glasses to a server. The server uses speech recognition technology to convert this speech into text and feeds it into a generative AI system. The generative AI then uses VADER Sentiment Analysis to extract an emotion score, which then visualizes the worker's mood and energy level.

[1804] Example prompt sentence:

[1805] Calculate the emotion score when a worker says, "I'm very tired today," and display it in a graph.

[1806] By inputting this prompt into a generative AI model, an emotion score based on the worker's utterance is calculated and visualized, allowing managers to quickly understand the worker's emotional state and take appropriate action.

[1807] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1808] Step 1:

[1809] The server receives voice data from the smart glasses. The input is the worker's speech, and the output is digital voice data. The server stores this voice data.

[1810] Step 2:

[1811] The server converts the voice data into text data using speech recognition technology. The input is the voice data, and the output is the transcribed text data. Specifically, the server uses speech recognition software (e.g., Google Speech-to-Text API) to analyze the voice and generate the corresponding text.

[1812] Step 3:

[1813] The server inputs text data into the generative AI. The input is the transcribed text data, and the output is an emotion score. Specifically, the emotion score of the text data is calculated using VADER Sentiment Analysis.

[1814] Step 4:

[1815] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. Specifically, each emotion score is aggregated as a numerical value, and the sum or average is calculated.

[1816] Step 5:

[1817] The server visualizes the mood and energy levels of workers based on the estimated overall emotion score. The input is the overall emotion score, and the output is a graph or chart. Specifically, Matplotlib is used to visually display the emotion score.

[1818] Step 6:

[1819] The server provides visualized data to managers to optimize the work environment. The input is graphs and charts, and the output is notifications and reports to managers. Specifically, graphs are displayed on a dashboard, allowing managers to understand the emotional state of workers in real time.

[1820] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1821] "Example 1"

[1822] In the first embodiment, a voice recognition technology is used as a means for transcribing voice data. Specifically, the voice data is converted into text data using a voice recognition API such as Google's.

[1823] "Example 2"

[1824] In Example 2, a deep learning model is used as a means for inputting data into generative AI. Specifically, text data is analyzed using models such as BERT and GPT-3.

[1825] "Example 3"

[1826] In the third embodiment, an emotion engine is used as a means for extracting emotion scores of included words and phrases. Specifically, emotion scores are extracted from text data using an emotion analysis tool such as VADER.

[1827] "Example 4"

[1828] In the fourth embodiment, an emotion engine is used as a means for inferring the overall emotion by summing or averaging the extracted emotion scores. Specifically, the overall emotion is inferred by summing or averaging the extracted emotion scores.

[1829] "Example 5"

[1830] In the fifth embodiment, a dashboard is used as a means for visualizing the mood and energy level of the meeting. Specifically, the mood and energy level of the meeting are displayed as a graph on the dashboard using the estimated overall emotions.

[1831] "Example 6"

[1832] In the sixth embodiment, an emotion engine is used as a means for improving the efficiency of proceeding with a meeting and gathering opinions of participants. Specifically, emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, thereby improving the efficiency of proceeding with the meeting and gathering opinions.

[1833] The processing flow of each embodiment will be described below.

[1834] "Example 1"

[1835] Step 1: Step 1: Collect audio data during the meeting.

[1836] Step 2: Step 2: Convert the collected voice data into text data using Google's voice recognition API or similar.

[1837] "Example 2"

[1838] Step 1: Step 1: Convert the audio data into text data.

[1839] Step 2: Step 2: The converted text data is fed into a deep learning model such as BERT or GPT-3 and analyzed.

[1840] "Example 3"

[1841] Step 1: Step 1: Analyze the text data.

[1842] Step 2: Step 2: Extract sentiment scores from the analyzed text data using a sentiment analysis tool such as VADER.

[1843] "Example 4"

[1844] Step 1: Step 1: Extract sentiment scores from text data.

[1845] Step 2: Step 2: Sum or average the extracted emotion scores to infer the overall emotion.

[1846] "Example 5"

[1847] Step 1:Step 1: Guess the overall emotion.

[1848] Step 2: Step 2: Using the estimated overall sentiment, the mood and energy level of the meeting are displayed as a graph on a dashboard.

[1849] "Example 6"

[1850] Step 1:Step 1:Visualize the mood and energy level of the meeting.

[1851] Step 2: Step 2: The emotions extracted by the emotion engine are used to adjust the mood and energy level of the meeting, improving the efficiency of meeting progress and opinion gathering.

[1852] Example 1

[1853] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1854] Conventional meeting systems have difficulty accurately recording participants' comments and grasping the mood and energy level of the meeting in real time. They also lack the means to effectively manage the meeting and gather participants' opinions. This leads to problems such as reduced meeting efficiency and impaired decision-making quality.

[1855] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1856] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data using voice recognition technology, means for inputting the converted text data into generative artificial intelligence, means for extracting emotion scores for the words and phrases included, means for estimating the overall emotion by summing or averaging the scores, means for visualizing the mood and energy level of the meeting, and means for improving the progress of the meeting and the collection of participants' opinions. This makes it possible to accurately record comments made in the meeting and grasp the mood and energy level of the meeting in real time. Furthermore, it is possible to effectively progress the meeting and collect opinions, thereby improving the efficiency of the meeting and the quality of decision-making.

[1857] "Audio data" refers to data in which the speech of conference participants is recorded in digital form.

[1858] "Voice recognition technology" is a technology that analyzes voice data and converts it into corresponding text data.

[1859] "Character data" refers to data in text format that has been converted using voice recognition technology.

[1860] "Generative AI" is an AI technology that uses deep learning models to generate new information and predictions from input data.

[1861] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in text data.

[1862] "Mood" refers to the overall atmosphere and emotional state of the meeting.

[1863] "Energy level" is an indicator of the liveliness of a meeting and the energy of the participants.

[1864] "Visualization" is a technique that makes data easier to understand by visually displaying it.

[1865] "Convergence" is the process of summarizing the opinions of meeting participants and arriving at a common conclusion or direction.

[1866] "Effectiveness" is the optimization of processes and means to achieve a specific purpose.

[1867] MODE FOR CARRYING OUT THE INVENTION

[1868] The present invention relates to a system for acquiring speech data from conference participants and transcribing the data using speech recognition technology. Specific embodiments of this system will be described below.

[1869] Acquiring audio data

[1870] When a user speaks during a meeting, the device picks up the voice through a microphone. This voice data is sent to the server in real time. The device is equipped with a high-quality microphone and also has a noise-canceling function.

[1871] Sending audio data

[1872] The voice data acquired by the device is sent to a server via the Internet. The server receives the voice data and proceeds to the next processing step. HTTP, WebSocket, or other communication protocols are used.

[1873] Use of voice recognition technology

[1874] The server converts the received voice data into text data using speech recognition technology. Specifically, it uses Google's speech recognition API, which can convert voice to text with high accuracy.

[1875] Use of generative artificial intelligence

[1876] The converted text data is then fed into a generative AI system that uses deep learning models to extract sentiment scores for the words and phrases contained within the text data.

[1877] Extracting and visualizing emotion scores

[1878] The server sums or averages the emotion scores to infer an overall emotion. This information is used to visualize the mood and energy level of the meeting. Visualization can be done using graphs or charts.

[1879] Meeting progress and opinion gathering

[1880] The server provides a means for effectively managing the progress of meetings and gathering opinions from participants, for example, through real-time feedback and voting functions, which can improve the efficiency of meetings.

[1881] Specific examples

[1882] For example, if a user says "Let's move on to the next topic" during a meeting, the device's microphone captures this audio. The device then sends this audio data to the server in real time. The server then sends the received audio data to Google's speech recognition API, receiving the text data "Let's move on to the next topic." The server then inputs this text data into a generative artificial intelligence system to extract an emotion score. Ultimately, the system visualizes the mood and energy level of the meeting, helping to guide the meeting.

[1883] Prompt Sentence Examples

[1884] Examples of prompts to input to a generative AI model might include:

[1885] "Design a system that transcribes speech from meeting participants in real time. Include the process of capturing audio data, converting it to text using Google's speech recognition API, and storing it in a database."

[1886] In this way, by clarifying the specific embodiment of the system, it becomes possible to accurately understand and practice the technical scope of the invention.

[1887] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1888] Step 1:

[1889] When a user speaks during a conference, the device captures the voice through a microphone. The input is the user's speech, and the output is digital voice data. The device transmits this voice data to the server in real time. Specifically, the device's microphone captures the voice and generates voice data.

[1890] Step 2:

[1891] The voice data acquired by the device is sent to a server via the Internet. The input is the voice data sent from the device, and the output is the voice data received by the server. The server receives this voice data and proceeds to the next processing step. Specifically, data is sent and received using communication protocols such as HTTP or WebSocket.

[1892] Step 3:

[1893] The server converts the received voice data into text data using voice recognition technology. The input is the voice data received by the server, and the output is the text data returned from the voice recognition API. Specifically, the server sends the voice data to Google's voice recognition API and receives the text data returned from the API.

[1894] Step 4:

[1895] The server inputs the converted text data into the generative AI. The input is the text data returned from the speech recognition API, and the output is the emotion score analyzed by the generative AI. Specifically, the server inputs the text data into the generative AI and extracts the emotion score.

[1896] Step 5:

[1897] The server sums or averages the emotion scores to estimate the overall emotion. The input is the emotion score obtained from the generative AI, and the output is numerical data indicating the overall emotion. Specifically, the server aggregates the emotion scores and calculates the overall emotion.

[1898] Step 6:

[1899] The server visualizes the mood and energy level of the meeting. The input is numerical data that indicates the overall emotion, and the output is a visual representation such as a graph or chart. Specifically, the server converts the data into a graph or chart and displays it to the user.

[1900] Step 7:

[1901] The server provides a means to effectively manage the progress of the meeting and gather participants' opinions. The input is the progress of the meeting and the opinions of the participants, and the output is real-time feedback and voting results. Specifically, the server collects feedback in real time and gathers opinions through the voting function.

[1902] In this way, by clarifying the specific operations performed at each processing step and their inputs and outputs, it becomes easier to understand the overall processing flow of the system.

[1903] (Application example 1)

[1904] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1905] Conventional conference systems lack the means to effectively progress a meeting or gather opinions from participants. Also, when work instructions are given by voice in a factory, there is no way to accurately record those instructions and manage them as a work log, making it difficult to accurately grasp the progress of work and the content of instructions. For this reason, there is a need for a system that can improve the efficiency of meetings and accurately record and manage work instructions in a factory.

[1906] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes a means for transcribing voice data, a means for inputting the data into a generative AI, a means for extracting emotion scores of included words and phrases, a means for summing or averaging the scores to estimate the overall emotion, a means for visualizing the mood and energy level of the meeting, a means for improving the progress of the meeting and the effectiveness of gathering participants' opinions, a means for acquiring work instructions by voice and transcribing and recording them in real time, and a means for managing a work log. This enables more efficient meetings and accurate recording and management of work instructions in the factory.

[1907] "Audio data" refers to data in which audio information such as conference and work instructions is recorded in digital format.

[1908] "Transcription" is the process of converting audio data into text data.

[1909] "Generative AI" is an artificial intelligence technology that uses deep learning models to generate new information and results from input data.

[1910] An "emotion score" is a numerical representation of the emotional nuance of words and phrases contained in audio or text data.

[1911] "Mood" refers to the overall atmosphere and emotional state of a meeting or work environment.

[1912] "Energy level" is an indicator of the level of activity and concentration of participants in a meeting or work environment.

[1913] "Visualization" is the process of making data or information easier to understand by displaying it visually.

[1914] "Work instructions" are voice or text information that instructs the content and procedures of work to be done in a factory.

[1915] "Real-time" refers to the instantaneous processing of data and provision of information.

[1916] A "work log" is data that records work instructions and progress within a factory.

[1917] "Management" is the process of organizing data and information and storing and using them appropriately.

[1918] A system for implementing this invention includes means for transcribing voice data, means for inputting the data into a generative AI, means for extracting emotion scores for the words and phrases included, means for summing or averaging the scores to estimate the overall emotion, means for visualizing the mood and energy level of a meeting, means for improving the effectiveness of the progress of a meeting and the collection of opinions of participants, means for obtaining work instructions via voice and transcribing and recording them in real time, and means for managing a work log.

[1919] System program processing explanation

[1920] Acquiring and transcribing audio data

[1921] The server uses a microphone to capture audio data, which is recorded using the Python speech_recognition library and converted to text using the Google Cloud Speech-to-Text API.

[1922] Extracting and visualizing emotion scores

[1923] Generative AI uses natural language processing techniques to extract sentiment scores for words and phrases contained in text data. The extracted sentiment scores are summed or averaged to infer overall sentiment. The inferred sentiment is visualized as the mood or energy level of the meeting.

[1924] Real-time transcription of work instructions and work log management

[1925] The server receives voice instructions from within the factory and transcribes them in real time. This allows for accurate recording of work instructions. The recorded work instructions are managed as a work log. This makes it possible to accurately grasp the progress of work and the content of instructions.

[1926] Examples of specific examples and prompts

[1927] As a concrete example, consider work instructions in a factory. For example, when a worker issues a voice command such as "Please proceed to the next process," the command is transcribed in real time and recorded in a work log.

[1928] Example prompts to input to a generative AI model:

[1929] Transcribe voice instructions such as "Proceed to the next step" in real time and record them in your work log.

[1930] Using this prompt, the generative AI model can accurately transcribe the voice instructions and take appropriate action to record them in the work log.

[1931] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1932] Step 1: Acquire audio data

[1933] The server captures voice data using a microphone. When a user speaks into the microphone, the voice is recorded in digital format. The input is the user's voice, and the output is digital voice data. This voice data is captured using the Python speech_recognition library.

[1934] Step 2: Transcribe the audio data

[1935] The server sends the acquired voice data to the Google Cloud Speech-to-Text API for transcription. The input is digital voice data, and the output is text data. The Google Cloud Speech-to-Text API analyzes the voice data and generates corresponding text data.

[1936] Step 3: Extract sentiment scores

[1937] The server uses generative AI to extract sentiment scores for words and phrases contained in text data. The input is text data, and the output is a sentiment score. Natural language processing technology is used to quantify the emotional nuance of each word or phrase.

[1938] Step 4: Sum or average sentiment scores

[1939] The server sums or averages the extracted emotion scores to estimate the overall emotion. The input is the individual emotion scores, and the output is the overall emotion score. This allows us to numerically express the mood and energy level of the meeting.

[1940] Step 5: Visualize emotions

[1941] The server visualizes the inferred overall emotion score. The input is the overall emotion score, and the output is a visual graph or chart, which provides a visual understanding of the mood and energy level of the meeting.

[1942] Step 6: Real-time transcription of work instructions

[1943] The server receives voice instructions from the factory and transcribes them in real time. The input is the user's voice instructions, and the output is text data. Voice recognition technology is used to instantly convert voice instructions into text data.

[1944] Step 7: Manage your work log

[1945] The server manages the transcribed work instructions as a work log. The input is text data, and the output is an organized work log. This allows the progress of work and the content of instructions to be accurately recorded and managed.

[1946] Example 2

[1947] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1948] Conventional methods for recording meetings and interviews not only transcribe audio data, but also make it difficult to efficiently analyze the content and extract important information. Furthermore, there is a lack of a way to visualize the mood and energy level of the meeting and effectively summarize the opinions of participants. This leads to problems such as a decrease in the efficiency of meeting progress and decision-making.

[1949] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1950] In this invention, the server includes means for transcribing voice data, means for inputting the data into a generative AI model, means for extracting emotion scores for included words and phrases, means for summing or averaging the scores to estimate overall emotion, means for visualizing the mood and energy level of the meeting, means for improving the progress of the meeting and the collection of participant opinions, means ...

Claims

1. means for obtaining text data representative of the voice data of each participant of the conference; A means for extracting emotion scores of words or phrases included in the text data using a prompt sentence instructing to extract emotion scores of words or phrases included in the text data and a generative AI model; means for estimating an overall emotion score from the sum or average of the extracted emotion scores; A means for visualizing changes in the inferred emotion scores; and A system including:

2. The system according to claim 1 , further comprising: means for notifying when a peak in the change in the estimated emotion score is detected.

3. The system of claim 1 , further comprising a means for estimating and presenting the progress of the meeting using a prompt sentence instructing the progress of the meeting to be estimated based on the change in the inferred emotion score and the generative AI model.

Images