system

Abstract

Provide a system. A means for users to upload presentation material data and audio data; means for converting voice data into text data; means for extracting volume data and emotion data from the audio data; A means for comparing the presentation data with the converted text data; a means for evaluating speaking style based on volume data and emotion data; a means of generating feedback on the content and delivery of the presentation; means for providing feedback to a user's terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional support systems for presentations and interview practice, it has been difficult for users to obtain detailed feedback on the logical consistency of their presentation content or their speaking style. Furthermore, there are almost no systems that can extract emotional and volume information from audio data and provide feedback based on that information. This makes it difficult for users to improve both their presentation content and speaking style. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for a user to upload presentation material data and audio data, a means for converting the audio data into text data, a means for extracting volume data and emotion data from the audio data, a means for comparing the presentation material data with the converted text data, a means for evaluating speaking style based on the volume data and emotion data, a means for generating feedback on the content and speaking style of the presentation, and a means for providing the feedback to the user's terminal. This allows users to receive detailed and specific feedback on both the content and speaking style of their presentation, thereby enabling them to improve the quality of their presentations.

[0006] A "user" is an entity that uses the system to upload presentation material data and audio data and receive feedback.

[0007] "Presentation material data" refers to text data related to presentations or presentations that users upload to the system.

[0008] "Audio data" refers to audio recording data uploaded by a user to the system, including the content and speaking style of the presentation.

[0009] "Text data" is character string information converted from voice data using a voice recognition engine.

[0010] "Volume data" is information about the volume of sound extracted from audio data.

[0011] "Emotion data" is information that indicates the emotional state of a speaker analyzed from voice data.

[0012] "Feedback" refers to the evaluation and improvement information generated by the system regarding the content and style of presentation.

[0013] "System" refers to a computer program with a set of functions that allows users to upload presentation data and audio data, analyzes them, and provides feedback.

[0014] "External Sentiment Analysis Service" means an external third-party service used by the System to extract emotion data from audio data.

[0015] A "text generation algorithm" is an algorithm that uses automatic generation technology to generate feedback on the content and speaking style of a presentation. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[0038] Overall system overview

[0039] The system consists of the following main components:

[0040] 1. User Input Components

[0041] 2. Audio data processing component

[0042] 3. Analysis components of presentation materials

[0043] 4. Speaking Assessment Component

[0044] 5. Feedback Generation Component

[0045] 6. Feedback Provision Component

[0046] System Operation

[0047] User Input

[0048] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0049] Audio data processing

[0050] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[0051] Analysis of presentation materials

[0052] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[0053] Speaking evaluation

[0054] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0055] Generate feedback

[0056] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[0057] Providing feedback

[0058] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0059] Specific examples

[0060] An example of a specific operation of the system is shown below.

[0061] User Input

[0062] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0063] Audio data processing

[0064] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0065] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0066] Additionally, through external sentiment analysis services such as Empath, we receive the following data:

[0067] Volume: 60 decibels on average

[0068] Emotions: High motivation, medium tension, low joy

[0069] Analysis of presentation materials

[0070] Extract the following text data from "Documents.pdf":

[0071] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0072] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0073] Speaking evaluation

[0074] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0075] Generate feedback

[0076] The server generates feedback like this:

[0077] Overall feedback:

[0078] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0079] Increasing the volume will make it easier for the audience to hear.

[0080] Expressing your emotions will improve the overall impression of your presentation.

[0081] Providing feedback

[0082] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0083] In this way, the system of the present invention provides detailed and specific feedback to the user, helping them improve the quality of their presentation or presentation.

[0084] The processing flow will be explained below.

[0085] Step 1: The user uploads the presentation data (PDF file) and audio data (WAV file) to the system.

[0086] The user clicks on the file selection button on the interface to select the file to upload.

[0087] The user clicks the "Upload" button to send the selected file to the server.

[0088] Step 2: The server receives the uploaded file.

[0089] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[0090] Step 3: The server sends the voice data to the voice recognition engine.

[0091] The server sends the uploaded audio file (WAV file) to a speech recognition API (e.g., a cloud speech recognition service).

[0092] The server receives the converted text data from the speech recognition API.

[0093] Step 4: The server extracts volume data and emotion data from the audio data.

[0094] The server sends the audio data to an external emotion analysis API (e.g., Empath).

[0095] The server receives volume data and emotional data (e.g., enthusiasm, tension, joy, etc.) from the emotion analysis API.

[0096] Step 5: The server converts the presentation data into text data.

[0097] The server converts the uploaded PDF file into text data using a text analysis library (e.g., a PDF text extraction tool).

[0098] Step 6: The server compares the audio text with the text data of the presentation materials.

[0099] The server uses a text comparison algorithm (e.g., Jaccard similarity or Levenshtein distance) to calculate the degree of similarity between the spoken text and the source text.

[0100] Step 7: The server evaluates the speaking style based on the volume data and emotion data.

[0101] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[0102] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[0103] Step 8: The server generates comprehensive feedback.

[0104] Based on each analysis, the server uses a text generation algorithm to generate specific feedback about the content and delivery of the presentation.

[0105] Step 9: The server sends the generated feedback to the user's terminal.

[0106] The server sends an HTTP response containing the generated feedback, which is received by the user's device.

[0107] Step 10: The user's device displays the feedback.

[0108] The user's device analyzes the received feedback data and displays it in an easy-to-read format on the interface.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] Current presentation evaluation systems have limited means of specifically evaluating the logical consistency of presentation content and the effectiveness of speaking style. This makes it difficult for users to obtain specific feedback to improve the quality of their presentations. Additionally, there is a lack of use of external services for emotion analysis and volume evaluation, making it difficult to obtain highly accurate analysis results. To solve these problems, a system that can provide effective and comprehensive feedback is needed.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes means for users to upload presentation material data and audio data, means for converting the audio data into text data, and means for extracting volume data and emotion data from the audio data, thereby enabling the server to analyze the presentation material and audio data entered by the user and provide detailed and specific feedback.

[0114] "User" refers to a person who logs into the system and uploads presentation material data and audio data.

[0115] "Presentation material data" refers to document data that describes the content of the presentation, and is provided in PDF or text file format.

[0116] "Audio data" refers to audio data recorded from a user's presentation, and is provided in a format such as a WAV file.

[0117] "Upload" refers to the operation of a user sending data from their own terminal to the system's server.

[0118] "Web interface" refers to a function that provides a screen for users to access the system via the Internet and operate or transmit data.

[0119] "Speech Recognition API" refers to the application programming interface of a cloud service used to convert voice data into text data.

[0120] "Text Extraction Library" means a software library for extracting text data from PDF and other document files.

[0121] "Sentiment Analysis API" refers to an application programming interface for an external service used to analyze emotion data from audio data.

[0122] "Natural language processing technology" refers to a general term for algorithms and technologies that enable computers to understand and generate human language.

[0123] "Feedback" refers to the evaluation results that include specific advice and areas for improvement regarding the user's presentation.

[0124] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[0125] Overall system overview

[0126] The system consists of the following main components:

[0127] 1. User Input Components

[0128] 2. Audio data processing component

[0129] 3. Analysis components of presentation materials

[0130] 4. Speaking Assessment Component

[0131] 5. Feedback Generation Component

[0132] 6. Feedback Provision Component

[0133] Hardware and software used

[0134] The system uses the following hardware and software:

[0135] Cloud service speech recognition API (e.g., Google® Cloud Speech-to-Text API)

[0136] External sentiment analysis API (e.g. Empath)

[0137] Text extraction library (e.g., Apache PDFBox)

[0138] Natural language processing technology (e.g., GPT-3 (registered trademark))

[0139] System Operation

[0140] User Input

[0141] Users log in to the system using a web interface and upload presentation data (PDF or text file format) and audio data (WAV files, etc.). By clicking the upload button, the data is sent to the server.

[0142] Audio data processing

[0143] After receiving the uploaded voice data, the server converts the voice data into text data using the cloud service's voice recognition API, which is then stored on the server. Furthermore, the server uses an external emotion analysis API to obtain volume data and emotion data from the voice data.

[0144] Analysis of presentation materials

[0145] The server analyzes the presentation data uploaded by the user. In the case of PDF files, it uses a text extraction library to extract the text from the document. It then compares this text data with the text data converted from the audio data and evaluates the degree of content agreement.

[0146] Speaking evaluation

[0147] The server evaluates the appropriateness of speaking based on volume and emotion data. It analyzes average volume, volume fluctuations, and emotion labels to determine how effective a user's speech is.

[0148] Generate feedback

[0149] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology to provide the user with specific advice and suggestions for improvement.

[0150] Providing feedback

[0151] The server sends the generated feedback to the user's device, and the user's device displays the received feedback on a web interface, allowing the user to understand specific areas for improvement.

[0152] Specific examples

[0153] An example of a specific operation of the system is shown below.

[0154] User Input

[0155] User "A" uploads the presentation materials ("materials.pdf") and the presentation audio file ("presentation.wav") through the web interface. These data are accepted and stored on the server.

[0156] Audio data processing

[0157] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[0158] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0159] Additionally, we receive the following data through Empath's sentiment analysis service:

[0160] Volume: 60 decibels on average

[0161] Emotions: High motivation, medium tension, low joy

[0162] Analysis of presentation materials

[0163] Extract the following text data from "Documents.pdf":

[0164] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0165] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0166] Speaking evaluation

[0167] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0168] Generate feedback

[0169] The server generates feedback like this:

[0170] Overall feedback:

[0171] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0172] Increasing the volume will make it easier for the audience to hear.

[0173] Expressing your emotions will improve the overall impression of your presentation.

[0174] Providing feedback

[0175] The generated feedback is sent to the user's terminal and displayed on a web interface.

[0176] Prompt Sentence Examples

[0177] Here are some example prompts for using generative AI models:

[0178] Analyze user-uploaded audio files (WAV format) and generate feedback assessing:

[0179] How well does the content of the talk match the presentation materials?

[0180] Volume and its fluctuations

[0181] Emotional expression (motivation, tension, joy, etc.)

[0182] Here are the analysis results:

[0183] Voice text: "Hello, my name is A. Today I'll be talking about the latest developments in AI technology."

[0184] Source text: "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[0185] Volume: 60 decibels on average

[0186] Emotions: High motivation, medium tension, low joy

[0187] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0188] Step 1: User uploads presentation data and audio data

[0189] Users log in to the system through a web interface. After logging in, they select the presentation data (e.g., a PDF file called "Materials.pdf") and audio data (e.g., a WAV file called "Presentation.wav") and click the upload button. The input is sent from the user's device and received by the server. The output is that the uploaded files are saved on the server. Specifically, the user selects "Materials.pdf" and "Presentation.wav" on the file selection screen and clicks the "Upload" button.

[0190] Step 2: Convert audio data to text

[0191] The server sends the received audio data ("Announcement.wav") to the cloud service's speech recognition API, which converts the audio data into text data. The input is the "Announcement.wav" file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology."). Specifically, the server sends "Announcement.wav" to the Google Cloud Speech-to-Text API, receives the converted text data, and saves it.

[0192] Step 3: Extract emotion and volume data

[0193] The server sends the converted voice-to-text data to an external emotion analysis API and extracts volume and emotion data. The input is voice data ("Announcement.wav"), and the output is volume data (e.g., average 60 decibels) and emotion data (e.g., high enthusiasm, medium tension, low joy). Specifically, the server sends the data to the Empath API, obtains the analysis results, and saves them.

[0194] Step 4: Analyze the presentation

[0195] The server analyzes the uploaded presentation data ("materials.pdf"). Specifically, it uses a text extraction library to extract the text within the document. The input is a PDF file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology. Specifically, I will explain the history of AI, current research directions, and future prospects."). Specifically, the server extracts and saves the text data using Apache PDFBox.

[0196] Step 5: Compare with the converted text

[0197] The server compares the text data converted from the speech with the text data in the presentation materials and calculates the degree of similarity. The input is the text data converted from the speech and the text data extracted from the presentation materials, and the output is the degree of similarity (e.g., 75%). Specifically, the server uses an algorithm to compare both sets of text data and evaluate the degree of similarity.

[0198] Step 6: Speech assessment

[0199] The server evaluates the appropriateness of speaking style based on volume data and emotional data. The input is volume data and emotional data, and the output is the evaluation result (e.g., "low volume" or "modest emotional expression"). Specifically, the server analyzes this data and evaluates problems and areas for improvement in speaking style.

[0200] Step 7: Generate feedback

[0201] The server generates comprehensive feedback based on the results of each analysis. The inputs are the degree of match, volume data, and emotion data, and the output is feedback text (e.g., "The content matches the first chapter of the materials. Please also talk about the contents of chapters two and three. By increasing the volume, you will be able to communicate more easily with the audience. By using more emotional expression, the overall impression of the presentation will be improved."). Specifically, the server generates the feedback text using natural language processing technology.

[0202] Step 8: Provide feedback

[0203] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the display result on the user's terminal. In concrete terms, the server sends the generated feedback to the user's terminal, and the user's terminal displays it on the web interface.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Conventional presentation materials and voice analysis systems are limited to providing feedback on individual presentations and speeches, and have the problem of being unable to provide specific suggestions for improving the quality of safety briefings, especially those held within factories. Furthermore, these types of systems only allow users to analyze a portion of a presentation, making it difficult to provide comprehensive feedback that leads to improved safety awareness. The present invention aims to provide a specific system for improving the content and quality of safety briefings within factories and improving safety awareness.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes a means for users to upload presentation material data and audio data, a means for converting the audio data into text data, and a means for extracting volume data and emotion data from the audio data, which makes it possible to evaluate the degree of agreement between the presentation content and related materials, evaluate the appropriateness of the speaking style, and provide points for improving the quality of the safety briefing.

[0209] "Means for users to upload presentation material data and audio data" refers to means for users to send presentation material data and audio data to the system through a specified interface.

[0210] The "means for converting voice data into text data" refers to a means for analyzing uploaded voice data and converting it into text information.

[0211] The "means for extracting volume data and emotional data from audio data" refers to a means for extracting the volume level and emotional information from audio data using audio analysis technology.

[0212] The "means for comparing presentation material data with converted text data" is a means for analyzing the content of the presentation material and the text converted from the speech and evaluating the degree of agreement.

[0213] The "means for evaluating speaking style based on volume data and emotion data" is a means for evaluating a speaker's speaking style and presentation quality based on acquired volume data and emotion data.

[0214] The "means for generating feedback on the content and speaking style of the presentation" refers to a means for generating specific advice and points for improvement on the content and speaking style of the presentation based on the analysis results.

[0215] The "means for providing feedback to the user's terminal" refers to a means for transmitting the generated feedback to the user's device and displaying it.

[0216] The "means for collecting and analyzing audio data of safety briefings in factories and related document data" refers to a means for collecting and analyzing audio data of safety briefings held in factories and related document data.

[0217] "Means for evaluating the quality of safety briefings based on the analysis results and providing specific suggestions for improvement" refers to means for evaluating the quality of briefings based on collected and analyzed data and providing specific suggestions for improvement.

[0218] The present invention is a system that analyzes presentation material data and audio data uploaded by users and provides specific feedback on both the content and speaking style of the presentation. Furthermore, the system has the function of evaluating the quality of safety briefings in factories and providing specific points for improvement. The following describes in detail the embodiments of the present invention.

[0219] The system consists of the following main components:

[0220] 1. User Input Components

[0221] 2. Audio data processing component

[0222] 3. Analysis components of presentation materials

[0223] 4. Speaking Assessment Component

[0224] 5. Feedback Generation Component

[0225] 6. Feedback Provision Component

[0226] 7. Safety Briefing Analysis Component

[0227] System Operation

[0228] User Input

[0229] Users first log in to the system and upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0230] Audio data processing

[0231] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text). This saves the content of the voice data as text data. The server then uses an external emotion analysis service (e.g., Empath) to extract volume data and emotion data from the voice data.

[0232] Analysis of presentation materials

[0233] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document. The server then compares the converted audio text with the presentation text data to evaluate the degree of logical correspondence.

[0234] Speaking evaluation

[0235] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0236] Generate feedback

[0237] The server generates comprehensive feedback based on the results of each of the above analyses, using a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide the user with specific advice and suggestions for improvement.

[0238] Providing feedback

[0239] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0240] Safety Briefing Analysis

[0241] The server analyzes the audio data of safety briefings held in factories and related document data. The audio data is converted into text data using the procedure described above, and the related document data is also analyzed for text. The server integrates this data, evaluates the quality of the briefings, and generates specific improvements.

[0242] Specific examples

[0243] For example, if a factory manager says, "Please be sure to wear a helmet for safety reasons," but the documentation does not include that instruction, the system will point out the discrepancy between the documentation and the audio content.

[0244] Prompt Sentence Examples

[0245] Prompts for the development of an automated safety briefing analysis system:

[0246] Analyze the audio data and PDF documents and evaluate the degree of match.

[0247] Extract emotional data from audio data and evaluate the appropriateness of speaking styles.

[0248] Please provide your overall feedback based on these analysis results.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] User Input

[0252] Users upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. The input data is sent to the server, which then stores it.

[0253] Input: Presentation data and audio data

[0254] Output: Presentation data and audio data stored on the server

[0255] Specific operation: The user selects a file through a web browser and clicks the send button, which uploads the file to the server.

[0256] Step 2:

[0257] Converting audio data to text

[0258] The server receives the uploaded voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Through this process, the voice content is saved as text.

[0259] Input: Audio data

[0260] Output: Text data converted from audio data

[0261] Specific operation: The server sends the voice data to the speech recognition engine and stores the returned text data in the database.

[0262] Step 3:

[0263] Extracting volume and emotion data

[0264] The server sends the converted text data to an external emotion analysis service (e.g., Empath) to extract volume and emotion data, which then stores the speaker's emotional state and speaking style characteristics.

[0265] Input: Text data

[0266] Output: Volume data and emotion data

[0267] Specific operation: The server sends text data to the emotion analysis service, obtains the returned emotion data, and stores it.

[0268] Step 4:

[0269] Analysis of presentation data

[0270] The server analyzes the presentation data uploaded by the user. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document.

[0271] Input: Presentation data (PDF or text file)

[0272] Output: Text data of presentation materials

[0273] Specific operation: The server uses a text extraction library to extract content from PDF and text files and saves it in a database.

[0274] Step 5:

[0275] Comparison of presentation materials and audio text

[0276] The server compares the converted speech text with the presentation text, evaluates the degree of match, and stores the result as a numerical value indicating how well the presentation matches the presentation text.

[0277] Input: Converted speech text, presentation material text

[0278] Output: Match data

[0279] Specific operation: The server uses a text comparison algorithm to calculate the degree of match between the two data and saves the results.

[0280] Step 6:

[0281] Speaking evaluation

[0282] The server evaluates the appropriateness of the speaking style based on the volume data and emotion data, thereby determining how effective the speaker's speaking style is.

[0283] Input: volume data and emotion data

[0284] Output: Speaking evaluation data

[0285] Specific operation: The server analyzes the volume data and emotion data, evaluates the effectiveness of the speaking style, and saves the results.

[0286] Step 7:

[0287] Generate feedback

[0288] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide specific advice and suggestions for improvement to the user.

[0289] Input: Matching data, speaking style evaluation data

[0290] Output: Feedback data

[0291] Specific operation: The server uses a text generation algorithm to generate feedback based on a set of data and stores it in a database.

[0292] Step 8:

[0293] Providing feedback

[0294] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0295] Input: Feedback data

[0296] Output: Feedback displayed on the user's terminal

[0297] Specific operation: The server sends feedback data to the user's terminal, and the terminal displays it.

[0298] Step 9:

[0299] Collection and analysis of audio and document data from safety briefings

[0300] The server collects and analyzes the audio data of safety briefings held in factories and related materials. The collected data is converted into text and analyzed using the procedure described above.

[0301] Input: Audio data of safety briefing and related materials data

[0302] Output: Text data and analysis data

[0303] What it does: The server collects the safety briefing data and analyzes it using a speech recognition engine and text extraction library.

[0304] Step 10:

[0305] Evaluating the quality of safety briefings and providing suggestions for improvement

[0306] Based on the collected and analyzed data, the server evaluates the quality of the safety briefing and generates specific improvements.

[0307] Input: Analysis data

[0308] Output: Evaluation data and improvement proposal data

[0309] How it works: The server uses the analysis results to evaluate the quality of the safety briefing and uses a generative AI model to generate feedback suggesting improvements.

[0310] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0311] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[0312] Overall system overview

[0313] The system consists of the following main components:

[0314] 1. User Input Components

[0315] 2. Audio data processing component

[0316] 3. Analysis components of presentation materials

[0317] 4. Speaking Assessment Component

[0318] 5. Feedback Generation Component

[0319] 6. Feedback Provision Component

[0320] 7. Emotion Engine Component

[0321] System Operation

[0322] User Input

[0323] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0324] Audio data processing

[0325] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[0326] Analysis of presentation materials

[0327] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[0328] Speaking evaluation

[0329] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0330] Generate feedback

[0331] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[0332] Utilizing the Emotion Engine

[0333] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotions in real time from facial expressions and movements. The emotion data recognized in real time is combined with extracted data from the voice data and becomes an important factor in comprehensively evaluating speaking style.

[0334] Dynamic feedback adjustment

[0335] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, if the user is nervous, it might include advice like "Try to speak in a relaxed manner."

[0336] Providing feedback

[0337] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0338] Specific examples

[0339] An example of a specific operation of the system is shown below.

[0340] User Input

[0341] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0342] Audio data processing

[0343] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0344] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0345] Additionally, through external sentiment analysis services, we receive the following data:

[0346] Volume: 60 decibels on average

[0347] Emotions: High motivation, medium tension, low joy

[0348] Analysis of presentation materials

[0349] Extract the following text data from "Documents.pdf":

[0350] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0351] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0352] Speaking evaluation

[0353] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0354] Utilizing the Emotion Engine

[0355] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[0356] Dynamic feedback adjustment

[0357] The server dynamically adjusts the feedback as follows:

[0358] Overall feedback:

[0359] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0360] Increasing the volume will make it easier for the audience to hear.

[0361] Expressing your emotions will improve the overall impression of your presentation.

[0362] You seem nervous, so take a deep breath and relax.

[0363] Providing feedback

[0364] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0365] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[0366] The processing flow will be explained below.

[0367] Step 1: The user uploads the presentation data and audio data to the system.

[0368] The user clicks the file selection button on the interface and selects the presentation data (PDF file) and audio data (WAV file).

[0369] The user clicks the "Upload" button and the selected file is sent to the server.

[0370] Step 2: The server receives the uploaded file.

[0371] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[0372] Step 3: The server sends the voice data to the voice recognition engine.

[0373] The server sends the audio file (WAV file) to the speech recognition API.

[0374] The server receives the converted text data from the speech recognition API.

[0375] Step 4: The server extracts volume data and emotion data from the audio data.

[0376] The server sends the audio data to an external emotion analysis service and receives volume data and emotion data.

[0377] Step 5: The server uses the emotion engine to recognize the user's emotions in real time.

[0378] The user's camera footage is sent to a server, which then uses an emotion engine to analyze emotions in real time.

[0379] The server stores the analysis results.

[0380] Step 6: The server converts the presentation data into text data.

[0381] The server converts the PDF file into text data using a text analysis library.

[0382] Step 7: The server compares the audio text with the text data of the presentation materials.

[0383] The server uses a text comparison algorithm to calculate the degree of match between the spoken text and the source text.

[0384] Step 8: The server evaluates the speaking style based on the volume data and emotion data.

[0385] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[0386] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[0387] Step 9: The server evaluates the speaking style comprehensively based on the recognized emotion data in real time.

[0388] The server integrates the analysis results of real-time emotional data and voice data to make a comprehensive evaluation of speaking style.

[0389] Step 10: The server generates comprehensive feedback.

[0390] Based on each analysis result, the server generates specific feedback on the content and delivery of the presentation.

[0391] Using a text generation algorithm, it provides users with specific advice and areas for improvement.

[0392] Step 11: The server sends the generated feedback to the user's terminal.

[0393] The server sends an HTTP response including the generated feedback, which is received by the user's device.

[0394] Step 12: The user's device displays the feedback.

[0395] The user's terminal analyzes the received feedback data and displays it on the interface.

[0396] Specific examples

[0397] User Input

[0398] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0399] Audio data processing

[0400] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0401] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0402] Additionally, through external sentiment analysis services, we receive the following data:

[0403] Volume: 60 decibels on average

[0404] Emotions: High motivation, medium tension, low joy

[0405] Analysis of presentation materials

[0406] Extract the following text data from "Documents.pdf":

[0407] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0408] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0409] Speaking evaluation

[0410] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0411] Utilizing the Emotion Engine

[0412] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[0413] Dynamic feedback adjustment

[0414] The server dynamically adjusts the feedback as follows:

[0415] Overall feedback:

[0416] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0417] Increasing the volume will make it easier for the audience to hear.

[0418] Expressing your emotions will improve the overall impression of your presentation.

[0419] You seem nervous, so take a deep breath and relax.

[0420] Providing feedback

[0421] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0422] Example 2

[0423] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0424] In today's world, there is a demand for systems that allow users to efficiently evaluate and improve their presentation content and speaking style, but current systems have difficulty providing comprehensive feedback that combines analysis of audio data, comparison of document data, and real-time emotion recognition. Furthermore, because they are unable to generate dynamic feedback that takes into account the user's real-time emotions, they are unable to provide specific and useful suggestions for improvement.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0426] In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the content of the presentation and speaking style, means for analyzing real-time emotion data acquired from the user, means for dynamically adjusting the content of the feedback based on the real-time emotion data, and means for providing feedback to the user's terminal. This enables comprehensive analysis of the audio data and material data and provision of specific feedback that takes real-time emotion into consideration.

[0427] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[0428] "Presentation material data" refers to digital data in PDF or text file format used as presentation or explanatory materials.

[0429] "Audio Data" means digital data in the form of a WAV file or other audio file that contains a recording of content presented by a user.

[0430] A "voice recognition engine" is software or a service for converting voice data into text data.

[0431] "Text data" is data in the form of a character string extracted from voice data by a voice recognition engine.

[0432] "Emotion analysis service" is an external service that analyzes and extracts volume data and user emotion data from audio and video data.

[0433] "Volume data" is data that indicates the volume of a sound extracted from audio data.

[0434] "Emotion data" is data that indicates the user's emotional state (for example, enthusiasm, tension, joy, etc.).

[0435] "Material text" is text data in character string format extracted from presentation material data.

[0436] The "degree of agreement" is an evaluation index that indicates the degree to which the contents of the audio text and the material text match.

[0437] "Speaking style evaluation" is the process of evaluating the appropriateness and effectiveness of a user's speaking style based on volume data and emotion data.

[0438] "Feedback" is information that includes specific advice and areas for improvement regarding the content and delivery of the presentation.

[0439] "Real-time emotional data" is data that indicates the user's current emotional state, obtained by analyzing video captured by the user's camera.

[0440] "Dynamic feedback adjustment" is the process of appropriately changing the content of feedback based on emotional data obtained in real time.

[0441] A "terminal" is a device such as a computer or smartphone that a user uses to access the system.

[0442] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[0443] Hardware and Software Configuration

[0444] User Input Components

[0445] Users first log in to the system and upload their presentation data (e.g., PDF or text file format) and audio data (e.g., WAV files) to the server. Users input data using a web interface, register files through a form, and click the upload button to send the information to the server.

[0446] Audio data processing component

[0447] The server receives the uploaded voice data, converts the voice data into text data using a voice recognition API of a cloud service (e.g., Google Cloud Speech-to-Text), and extracts volume data and emotion data from the voice data using an external emotion analysis service (e.g., IBM Watson® Tone Analyzer).

[0448] Analysis components of presentation materials

[0449] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., Apache PDFBox) to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of match.

[0450] Speaking evaluation component

[0451] The server evaluates the appropriateness of the speaking style based on volume data and emotional data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0452] Feedback Generation Component

[0453] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology (e.g., a generative AI model) to provide the user with specific advice and suggestions for improvement.

[0454] Utilizing the Emotion Engine Component

[0455] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotional data in real time from facial expressions and movements. This real-time emotional data is also an important factor in comprehensively evaluating the speaking style.

[0456] Dynamic feedback adjustment

[0457] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, by including advice like "Try to speak in a relaxed manner" if the user is nervous.

[0458] Provide Feedback Component

[0459] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0460] Specific examples

[0461] An example of a specific operation of the system is shown below.

[0462] User Input

[0463] User "A" uploads the presentation materials "Materials.pdf" and the presentation audio file "Presentation.wav". The uploaded files are accepted and saved on the server.

[0464] Audio data processing

[0465] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[0466] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0467] Additionally, through IBM Watson Tone Analyzer, you receive the following data:

[0468] Volume: 60 decibels on average

[0469] Emotions: High motivation, medium tension, low joy

[0470] Analysis of presentation materials

[0471] Extract the following text data from "document.pdf" using Apache PDFBox:

[0472] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0473] The server compares the audio text with the source text and calculates a match of 75%.

[0474] Speaking evaluation

[0475] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0476] Utilizing the Emotion Engine

[0477] The emotion engine recognizes that the user is nervous.

[0478] Dynamic feedback adjustment

[0479] The server dynamically adjusts the feedback as follows:

[0480] Overall feedback:

[0481] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0482] Increasing the volume will make it easier for the audience to hear.

[0483] Expressing your emotions will improve the overall impression of your presentation.

[0484] You seem nervous, so take a deep breath and relax.

[0485] Providing feedback

[0486] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0487] Example prompts for generative AI models

[0488] Please analyze the data below and provide feedback on the content and delivery of your presentation:

[0489] Presentation materials:

[0490] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0491] Announcement audio text:

[0492] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0493] Volume data:

[0494] Average 60 decibels

[0495] Emotional Data:

[0496] High motivation, medium tension, low joy

[0497] Real-time emotions:

[0498] I'm nervous

[0499] When this prompt sentence is fed into a generative AI model, appropriate feedback is generated.

[0500] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0501] Step 1:

[0502] A user logs into the system. The user opens the web interface, enters their credentials (username and password) into the login form, and clicks the login button. The server receives this and checks the credentials against its database. If authentication is successful, the server issues a login session to the user and displays a welcome page.

[0503] Input: Username and Password

[0504] Output: Login session and welcome page

[0505] Step 2:

[0506] The user uploads the presentation materials and audio data. The user clicks the "Select File" button on the web interface and selects the presentation materials (e.g., materials.pdf) and audio data (e.g., presentation.wav). Once the selection is complete, the user clicks the "Upload" button, which sends the files to the server. The server receives the files and saves them in the specified folder.

[0507] Input: Presentation files and audio files

[0508] Output: Saved presentation materials and audio files

[0509] Step 3:

[0510] The server converts the audio data into text data. First, the server sends the saved audio file (Announcement.wav) to a cloud service's speech recognition API (e.g., Google Cloud Speech-to-Text). The API analyzes the audio data and returns the corresponding text data. The server receives this text data and stores it in a database.

[0511] Input: Audio file

[0512] Output: Text data converted from audio

[0513] Step 4:

[0514] The server extracts volume and emotion data from the audio data. The server sends the audio file to an emotion analysis service API (e.g., IBM Watson Tone Analyzer). The API analyzes the volume and emotion data and returns them. The server receives this data and stores it in a database.

[0515] Input: Audio file

[0516] Output: Volume data and emotion data

[0517] Step 5:

[0518] The server analyzes the presentation data as text data. The server analyzes the presentation file (e.g., material.pdf) using a text extraction library (e.g., Apache PDFBox) and extracts the text data. The extracted text data is stored in a database.

[0519] Input: Presentation materials file

[0520] Output: Text data extracted from the material

[0521] Step 6:

[0522] The server compares the audio text with the source text. The server retrieves the audio text and source text from the database and uses a text comparison algorithm to calculate the degree of match between the two. The result of the match is stored in the database.

[0523] Input: Audio text and source text

[0524] Output: Match evaluation result

[0525] Step 7:

[0526] The server evaluates the speaking style based on the volume data and emotion data. The server obtains the volume data and emotion data from the database and analyzes them using an algorithm to evaluate the appropriateness of the speaking style. The evaluation results are stored in the database.

[0527] Input: volume data and emotion data

[0528] Output: Speaking evaluation results

[0529] Step 8:

[0530] The server generates feedback on the content and speaking style of the presentation. The server uses natural language processing technology (e.g., generative AI model) to create feedback for the user based on the evaluation results. The feedback is stored in a database.

[0531] Input: Evaluation result

[0532] Output: Feedback

[0533] Step 9:

[0534] The server analyzes real-time emotion data acquired from the user. The server analyzes video captured from the user's camera in real time and extracts emotion data. This emotion data is stored in a database.

[0535] Input: Real-time video of the user

[0536] Output: Real-time emotion data

[0537] Step 10:

[0538] The server dynamically adjusts the feedback content based on real-time emotional data. The server uses the emotional data obtained in real time to change the feedback content as appropriate. For example, if the user is nervous, the server will include advice such as "Try to speak in a relaxed manner."

[0539] Input: Real-time emotion data

[0540] Output: Dynamically adjusted feedback

[0541] Step 11:

[0542] The server sends the generated feedback to the user's terminal, the server sends the final feedback to the user's terminal for the user to review, and the terminal displays the feedback on a web page.

[0543] Input: Feedback

[0544] Output: Feedback displayed on the terminal

[0545] (Application example 2)

[0546] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0547] In conventional presentation practice systems, when users receive feedback to evaluate their presentation content and speaking style, it is difficult for them to receive specific advice that appropriately takes into account changes in emotion and volume fluctuations. Furthermore, because feedback is not provided in real time, it is difficult for users to receive advice that corresponds to their momentary emotional state. The present invention aims to solve these problems.

[0548] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the presentation content and speaking style, means for recognizing the user's emotions in real time and dynamically adjusting the feedback, and means for providing feedback to the user's terminal. This enables the user to receive specific and dynamic feedback based on emotion recognition in real time.

[0549] "Means for users to upload presentation material data and audio data" refers to an interface that allows users to send their own presentation materials and audio to the system.

[0550] "Means for converting voice data into text data" refers to a function that analyzes input voice data and converts it into a corresponding text format.

[0551] "Means for extracting volume data and emotion data from audio data" refers to technology for identifying and extracting information related to volume and the speaker's emotion from collected audio data.

[0552] "Means for comparing presentation material data with converted text data" refers to a method for comparing the text of uploaded presentation materials with the text of converted audio data and evaluating the degree of agreement between them.

[0553] "Means for evaluating speaking style based on volume data and emotion data" refers to a mechanism for evaluating the effectiveness of a user's speaking style based on volume and emotion characteristics.

[0554] "Means for generating feedback on the content and speaking style of the presentation" refers to a function that generates specific advice and areas for improvement for the user regarding the content and speaking style of the presentation based on analyzed data.

[0555] "Means for recognizing a user's emotions in real time and dynamically adjusting feedback" refers to technology that recognizes a user's emotional state in real time and instantly changes the feedback content accordingly.

[0556] "Means for providing feedback to a user's terminal" refers to a system for transmitting the generated feedback to the user's device for display.

[0557] The following describes in detail the mode for carrying out the present invention. The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. This system recognizes emotions in real time and provides dynamic feedback to the user to improve the quality of the presentation.

[0558] Overall system overview

[0559] The system consists of the following main components:

[0560] 1. User Input Components

[0561] 2. Audio data processing component

[0562] 3. Analysis components of presentation materials

[0563] 4. Speaking Assessment Component

[0564] 5. Feedback Generation Component

[0565] 6. Real-time Emotion Recognition Component

[0566] 7. Feedback Provision Component

[0567] System Operation

[0568] User Input

[0569] Users first log in to the system, then upload their presentation materials (PDF or text file format) and audio data (WAV file format) through the web interface, allowing the system to receive the user's presentation materials and audio data.

[0570] Audio data processing

[0571] The server receives the uploaded voice data and converts it into text using a speech recognition engine such as the Google Cloud Speech-to-Text API, and also extracts volume and emotion data using an external emotion analysis service such as the Microsoft® Azure® Emotion Recognition API.

[0572] Analysis of presentation materials

[0573] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[0574] Speaking evaluation

[0575] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0576] Generate feedback

[0577] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[0578] Real-time Emotion Recognition

[0579] The server analyzes the video data acquired from the camera and recognizes the user's emotions in real time, which makes it possible to immediately provide advice such as "Try to speak in a relaxed manner" if the user is nervous.

[0580] Providing Feedback

[0581] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0582] Specific examples

[0583] An example of a specific operation of the system is shown below.

[0584] User Input

[0585] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0586] Audio data processing

[0587] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0588] "Hello, my name is A. Today I'd like to talk about the latest developments in AI technology."

[0589] Additionally, through external sentiment analysis services, we receive the following data:

[0590] Volume: 60 decibels on average

[0591] Emotions: High motivation, medium tension, low joy

[0592] Analysis of presentation materials

[0593] Extract the following text data from "Documents.pdf":

[0594] "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[0595] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0596] Speaking evaluation

[0597] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0598] Real-time Emotion Recognition

[0599] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[0600] Generating and Providing Feedback

[0601] The server dynamically adjusts the feedback as follows:

[0602] "General feedback:

[0603] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0604] Increasing the volume will make it easier for the audience to hear.

[0605] Expressing your emotions will improve the overall impression of your presentation.

[0606] You seem nervous, so take a deep breath and relax."

[0607] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0608] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[0609] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0610] Step 1:

[0611] A user first logs into the system, where they enter their account information and are authenticated. The information entered is securely stored by Firebase Authentication. The output is that the user is authenticated and granted access to the system.

[0612] Step 2:

[0613] Users upload presentation data (PDF or text file format) and audio data (WAV file format) to the system. The data selected by the user is uploaded to Firebase Storage and saved on the server side. The input data are presentation data and audio files, and these data are saved on the server as output.

[0614] Step 3:

[0615] The server sends the uploaded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. A speech recognition algorithm is used to convert the voice data (input) into text data (output). The text data is generated and saved.

[0616] Step 4:

[0617] The server uses the Microsoft Azure Emotion Recognition API to extract volume and emotion data from the audio data. Audio data is provided as input, and volume and emotion information is obtained as output. This data is used to analyze user emotions and volume fluctuations.

[0618] Step 5:

[0619] The server analyzes the presentation data as text data. In the case of PDF files, it uses a text extraction library to extract the text within the document. The input data is the presentation data, and the output is the extracted text data. This data is used to compare with the audio text.

[0620] Step 6:

[0621] The server compares the converted speech text with the text data of the presentation materials and evaluates the degree of match. The speech text and the material text are provided as input, and the degree of match is calculated as output. The degree of match indicates how faithfully the user followed the materials in their presentation.

[0622] Step 7:

[0623] The server evaluates the appropriateness of speaking style based on volume data and emotional data. Volume and emotional information are used as input data, and the evaluation results of speaking style are obtained as output. The evaluation results include the appropriateness of volume and the degree of emotional expression.

[0624] Step 8:

[0625] The server uses a text generation algorithm to generate feedback based on the analysis results. The input data is the speaking style evaluation results and the degree of agreement with the presentation content, and the output is specific feedback, including suggestions for improvement and advice.

[0626] Step 9:

[0627] The server recognizes the user's emotions in real time and dynamically adjusts the feedback. The input is real-time video data, and the output is dynamically adjusted feedback. The video data is captured through a camera and analyzed by the emotion engine.

[0628] Step 10:

[0629] The generated feedback is sent to the user's device, which then displays the received feedback. Specific improvements and advice are provided to the user as output, allowing the user to improve their presentation. The user can then review the feedback and use it in their next practice.

[0630] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0631] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0632] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0633] [Second embodiment]

[0634] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0635] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0636] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0637] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0638] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0639] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0640] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0641] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0642] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0643] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0644] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0645] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0646] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[0647] Overall system overview

[0648] The system consists of the following main components:

[0649] 1. User Input Components

[0650] 2. Audio data processing component

[0651] 3. Analysis components of presentation materials

[0652] 4. Speaking Assessment Component

[0653] 5. Feedback Generation Component

[0654] 6. Feedback Provision Component

[0655] System Operation

[0656] User Input

[0657] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0658] Audio data processing

[0659] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[0660] Analysis of presentation materials

[0661] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[0662] Speaking evaluation

[0663] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0664] Generate feedback

[0665] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[0666] Providing feedback

[0667] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0668] Specific examples

[0669] An example of a specific operation of the system is shown below.

[0670] User Input

[0671] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0672] Audio data processing

[0673] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0674] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0675] Additionally, through external sentiment analysis services such as Empath, we receive the following data:

[0676] Volume: 60 decibels on average

[0677] Emotions: High motivation, medium tension, low joy

[0678] Analysis of presentation materials

[0679] Extract the following text data from "Documents.pdf":

[0680] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0681] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0682] Speaking evaluation

[0683] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0684] Generate feedback

[0685] The server generates feedback like this:

[0686] Overall feedback:

[0687] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0688] Increasing the volume will make it easier for the audience to hear.

[0689] Expressing your emotions will improve the overall impression of your presentation.

[0690] Providing feedback

[0691] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0692] In this way, the system of the present invention provides detailed and specific feedback to the user, helping them improve the quality of their presentation or presentation.

[0693] The processing flow will be explained below.

[0694] Step 1: The user uploads the presentation data (PDF file) and audio data (WAV file) to the system.

[0695] The user clicks on the file selection button on the interface to select the file to upload.

[0696] The user clicks the "Upload" button to send the selected file to the server.

[0697] Step 2: The server receives the uploaded file.

[0698] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[0699] Step 3: The server sends the voice data to the voice recognition engine.

[0700] The server sends the uploaded audio file (WAV file) to a speech recognition API (e.g., a cloud speech recognition service).

[0701] The server receives the converted text data from the speech recognition API.

[0702] Step 4: The server extracts volume data and emotion data from the audio data.

[0703] The server sends the audio data to an external emotion analysis API (e.g., Empath).

[0704] The server receives volume data and emotional data (e.g., enthusiasm, tension, joy, etc.) from the emotion analysis API.

[0705] Step 5: The server converts the presentation data into text data.

[0706] The server converts the uploaded PDF file into text data using a text analysis library (e.g., a PDF text extraction tool).

[0707] Step 6: The server compares the audio text with the text data of the presentation materials.

[0708] The server uses a text comparison algorithm (e.g., Jaccard similarity or Levenshtein distance) to calculate the degree of similarity between the spoken text and the source text.

[0709] Step 7: The server evaluates the speaking style based on the volume data and emotion data.

[0710] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[0711] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[0712] Step 8: The server generates comprehensive feedback.

[0713] Based on each analysis, the server uses a text generation algorithm to generate specific feedback about the content and delivery of the presentation.

[0714] Step 9: The server sends the generated feedback to the user's terminal.

[0715] The server sends an HTTP response containing the generated feedback, which is received by the user's device.

[0716] Step 10: The user's device displays the feedback.

[0717] The user's device analyzes the received feedback data and displays it in an easy-to-read format on the interface.

[0718] Example 1

[0719] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0720] Current presentation evaluation systems have limited means of specifically evaluating the logical consistency of presentation content and the effectiveness of speaking style. This makes it difficult for users to obtain specific feedback to improve the quality of their presentations. Additionally, there is a lack of use of external services for emotion analysis and volume evaluation, making it difficult to obtain highly accurate analysis results. To solve these problems, a system that can provide effective and comprehensive feedback is needed.

[0721] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0722] In this invention, the server includes means for users to upload presentation material data and audio data, means for converting the audio data into text data, and means for extracting volume data and emotion data from the audio data, thereby enabling the server to analyze the presentation material and audio data entered by the user and provide detailed and specific feedback.

[0723] "User" refers to a person who logs into the system and uploads presentation material data and audio data.

[0724] "Presentation material data" refers to document data that describes the content of the presentation, and is provided in PDF or text file format.

[0725] "Audio data" refers to audio data recorded from a user's presentation, and is provided in a format such as a WAV file.

[0726] "Upload" refers to the operation of a user sending data from their own terminal to the system's server.

[0727] "Web interface" refers to a function that provides a screen for users to access the system via the Internet and operate or transmit data.

[0728] "Speech Recognition API" refers to the application programming interface of a cloud service used to convert voice data into text data.

[0729] "Text Extraction Library" means a software library for extracting text data from PDF and other document files.

[0730] "Sentiment Analysis API" refers to an application programming interface for an external service used to analyze emotion data from audio data.

[0731] "Natural language processing technology" refers to a general term for algorithms and technologies that enable computers to understand and generate human language.

[0732] "Feedback" refers to the evaluation results that include specific advice and areas for improvement regarding the user's presentation.

[0733] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[0734] Overall system overview

[0735] The system consists of the following main components:

[0736] 1. User Input Components

[0737] 2. Audio data processing component

[0738] 3. Analysis components of presentation materials

[0739] 4. Speaking Assessment Component

[0740] 5. Feedback Generation Component

[0741] 6. Feedback Provision Component

[0742] Hardware and software used

[0743] The system uses the following hardware and software:

[0744] Cloud service speech recognition API (e.g. Google Cloud Speech-to-Text API)

[0745] External sentiment analysis API (e.g. Empath)

[0746] A text extraction library (e.g., Apache PDFBox)

[0747] Natural language processing technology (e.g., GPT-3)

[0748] System Operation

[0749] User Input

[0750] Users log in to the system using a web interface and upload presentation data (PDF or text file format) and audio data (WAV files, etc.). By clicking the upload button, the data is sent to the server.

[0751] Audio data processing

[0752] After receiving the uploaded voice data, the server converts the voice data into text data using the cloud service's voice recognition API, which is then stored on the server. Furthermore, the server uses an external emotion analysis API to obtain volume data and emotion data from the voice data.

[0753] Analysis of presentation materials

[0754] The server analyzes the presentation data uploaded by the user. In the case of PDF files, it uses a text extraction library to extract the text from the document. It then compares this text data with the text data converted from the audio data and evaluates the degree of content agreement.

[0755] Speaking evaluation

[0756] The server evaluates the appropriateness of speaking based on volume and emotion data. It analyzes average volume, volume fluctuations, and emotion labels to determine how effective a user's speech is.

[0757] Generate feedback

[0758] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology to provide the user with specific advice and suggestions for improvement.

[0759] Providing feedback

[0760] The server sends the generated feedback to the user's device, and the user's device displays the received feedback on a web interface, allowing the user to understand specific areas for improvement.

[0761] Specific examples

[0762] An example of a specific operation of the system is shown below.

[0763] User Input

[0764] User "A" uploads the presentation materials ("materials.pdf") and the presentation audio file ("presentation.wav") through the web interface. These data are accepted and stored on the server.

[0765] Audio data processing

[0766] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[0767] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0768] Additionally, we receive the following data through Empath's sentiment analysis service:

[0769] Volume: 60 decibels on average

[0770] Emotions: High motivation, medium tension, low joy

[0771] Analysis of presentation materials

[0772] Extract the following text data from "Documents.pdf":

[0773] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0774] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0775] Speaking evaluation

[0776] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0777] Generate feedback

[0778] The server generates feedback like this:

[0779] Overall feedback:

[0780] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0781] Increasing the volume will make it easier for the audience to hear.

[0782] Expressing your emotions will improve the overall impression of your presentation.

[0783] Providing feedback

[0784] The generated feedback is sent to the user's terminal and displayed on a web interface.

[0785] Prompt Sentence Examples

[0786] Here are some example prompts for using generative AI models:

[0787] Analyze user-uploaded audio files (WAV format) and generate feedback assessing:

[0788] How well does the content of the talk match the presentation materials?

[0789] Volume and its fluctuations

[0790] Emotional expression (motivation, tension, joy, etc.)

[0791] Here are the analysis results:

[0792] Voice text: "Hello, my name is A. Today I'll be talking about the latest developments in AI technology."

[0793] Source text: "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[0794] Volume: 60 decibels on average

[0795] Emotions: High motivation, medium tension, low joy

[0796] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0797] Step 1: User uploads presentation data and audio data

[0798] Users log in to the system through a web interface. After logging in, they select the presentation data (e.g., a PDF file called "Materials.pdf") and audio data (e.g., a WAV file called "Presentation.wav") and click the upload button. The input is sent from the user's device and received by the server. The output is that the uploaded files are saved on the server. Specifically, the user selects "Materials.pdf" and "Presentation.wav" on the file selection screen and clicks the "Upload" button.

[0799] Step 2: Convert audio data to text

[0800] The server sends the received audio data ("Announcement.wav") to the cloud service's speech recognition API, which converts the audio data into text data. The input is the "Announcement.wav" file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology."). Specifically, the server sends "Announcement.wav" to the Google Cloud Speech-to-Text API, receives the converted text data, and saves it.

[0801] Step 3: Extract emotion and volume data

[0802] The server sends the converted voice-to-text data to an external emotion analysis API and extracts volume and emotion data. The input is voice data ("Announcement.wav"), and the output is volume data (e.g., average 60 decibels) and emotion data (e.g., high enthusiasm, medium tension, low joy). Specifically, the server sends the data to the Empath API, obtains the analysis results, and saves them.

[0803] Step 4: Analyze the presentation

[0804] The server analyzes the uploaded presentation data ("materials.pdf"). Specifically, it uses a text extraction library to extract the text within the document. The input is a PDF file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology. Specifically, I will explain the history of AI, current research directions, and future prospects."). Specifically, the server extracts and saves the text data using Apache PDFBox.

[0805] Step 5: Compare with the converted text

[0806] The server compares the text data converted from the speech with the text data in the presentation materials and calculates the degree of similarity. The input is the text data converted from the speech and the text data extracted from the presentation materials, and the output is the degree of similarity (e.g., 75%). Specifically, the server uses an algorithm to compare both sets of text data and evaluate the degree of similarity.

[0807] Step 6: Speech assessment

[0808] The server evaluates the appropriateness of speaking style based on volume data and emotional data. The input is volume data and emotional data, and the output is the evaluation result (e.g., "low volume" or "modest emotional expression"). Specifically, the server analyzes this data and evaluates problems and areas for improvement in speaking style.

[0809] Step 7: Generate feedback

[0810] The server generates comprehensive feedback based on the results of each analysis. The inputs are the degree of match, volume data, and emotion data, and the output is feedback text (e.g., "The content matches the first chapter of the materials. Please also talk about the contents of chapters two and three. By increasing the volume, you will be able to communicate more easily with the audience. By using more emotional expression, the overall impression of the presentation will be improved."). Specifically, the server generates the feedback text using natural language processing technology.

[0811] Step 8: Provide feedback

[0812] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the display result on the user's terminal. In concrete terms, the server sends the generated feedback to the user's terminal, and the user's terminal displays it on the web interface.

[0813] (Application example 1)

[0814] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] Conventional presentation materials and voice analysis systems are limited to providing feedback on individual presentations and speeches, and have the problem of being unable to provide specific suggestions for improving the quality of safety briefings, especially those held within factories. Furthermore, these types of systems only allow users to analyze a portion of a presentation, making it difficult to provide comprehensive feedback that leads to improved safety awareness. The present invention aims to provide a specific system for improving the content and quality of safety briefings within factories and improving safety awareness.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0817] In this invention, the server includes a means for users to upload presentation material data and audio data, a means for converting the audio data into text data, and a means for extracting volume data and emotion data from the audio data, which makes it possible to evaluate the degree of agreement between the presentation content and related materials, evaluate the appropriateness of the speaking style, and provide points for improving the quality of the safety briefing.

[0818] "Means for users to upload presentation material data and audio data" refers to means for users to send presentation material data and audio data to the system through a specified interface.

[0819] The "means for converting voice data into text data" refers to a means for analyzing uploaded voice data and converting it into text information.

[0820] The "means for extracting volume data and emotional data from audio data" refers to a means for extracting the volume level and emotional information from audio data using audio analysis technology.

[0821] The "means for comparing presentation material data with converted text data" is a means for analyzing the content of the presentation material and the text converted from the speech and evaluating the degree of agreement.

[0822] The "means for evaluating speaking style based on volume data and emotion data" is a means for evaluating a speaker's speaking style and presentation quality based on acquired volume data and emotion data.

[0823] The "means for generating feedback on the content and speaking style of the presentation" refers to a means for generating specific advice and points for improvement on the content and speaking style of the presentation based on the analysis results.

[0824] The "means for providing feedback to the user's terminal" refers to a means for transmitting the generated feedback to the user's device and displaying it.

[0825] The "means for collecting and analyzing audio data of safety briefings in factories and related document data" refers to a means for collecting and analyzing audio data of safety briefings held in factories and related document data.

[0826] "Means for evaluating the quality of safety briefings based on the analysis results and providing specific suggestions for improvement" refers to means for evaluating the quality of briefings based on collected and analyzed data and providing specific suggestions for improvement.

[0827] The present invention is a system that analyzes presentation material data and audio data uploaded by users and provides specific feedback on both the content and speaking style of the presentation. Furthermore, the system has the function of evaluating the quality of safety briefings in factories and providing specific points for improvement. The following describes in detail the embodiments of the present invention.

[0828] The system consists of the following main components:

[0829] 1. User Input Components

[0830] 2. Audio data processing component

[0831] 3. Analysis components of presentation materials

[0832] 4. Speaking Assessment Component

[0833] 5. Feedback Generation Component

[0834] 6. Feedback Provision Component

[0835] 7. Safety Briefing Analysis Component

[0836] System Operation

[0837] User Input

[0838] Users first log in to the system and upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0839] Audio data processing

[0840] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text). This saves the content of the voice data as text data. The server then uses an external emotion analysis service (e.g., Empath) to extract volume data and emotion data from the voice data.

[0841] Analysis of presentation materials

[0842] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document. The server then compares the converted audio text with the presentation text data to evaluate the degree of logical correspondence.

[0843] Speaking evaluation

[0844] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0845] Generate feedback

[0846] The server generates comprehensive feedback based on the results of each of the above analyses, using a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide the user with specific advice and suggestions for improvement.

[0847] Providing feedback

[0848] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0849] Safety Briefing Analysis

[0850] The server analyzes the audio data of safety briefings held in factories and related document data. The audio data is converted into text data using the procedure described above, and the related document data is also analyzed for text. The server integrates this data, evaluates the quality of the briefings, and generates specific improvements.

[0851] Specific examples

[0852] For example, if a factory manager says, "Please be sure to wear a helmet for safety reasons," but the documentation does not include that instruction, the system will point out the discrepancy between the documentation and the audio content.

[0853] Prompt Sentence Examples

[0854] Prompts for the development of an automated safety briefing analysis system:

[0855] Analyze the audio data and PDF documents and evaluate the degree of match.

[0856] Extract emotional data from audio data and evaluate the appropriateness of speaking styles.

[0857] Please provide your overall feedback based on these analysis results.

[0858] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0859] Step 1:

[0860] User Input

[0861] Users upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. The input data is sent to the server, which then stores it.

[0862] Input: Presentation data and audio data

[0863] Output: Presentation data and audio data stored on the server

[0864] Specific operation: The user selects a file through a web browser and clicks the send button, which uploads the file to the server.

[0865] Step 2:

[0866] Converting audio data to text

[0867] The server receives the uploaded voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Through this process, the voice content is saved as text.

[0868] Input: Audio data

[0869] Output: Text data converted from audio data

[0870] Specific operation: The server sends the voice data to the speech recognition engine and stores the returned text data in the database.

[0871] Step 3:

[0872] Extracting volume and emotion data

[0873] The server sends the converted text data to an external emotion analysis service (e.g., Empath) to extract volume and emotion data, which then stores the speaker's emotional state and speaking style characteristics.

[0874] Input: Text data

[0875] Output: Volume data and emotion data

[0876] Specific operation: The server sends text data to the emotion analysis service, obtains the returned emotion data, and stores it.

[0877] Step 4:

[0878] Analysis of presentation data

[0879] The server analyzes the presentation data uploaded by the user. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document.

[0880] Input: Presentation data (PDF or text file)

[0881] Output: Text data of presentation materials

[0882] Specific operation: The server uses a text extraction library to extract content from PDF and text files and saves it in a database.

[0883] Step 5:

[0884] Comparison of presentation materials and audio text

[0885] The server compares the converted speech text with the presentation text, evaluates the degree of match, and stores the result as a numerical value indicating how well the presentation matches the presentation text.

[0886] Input: Converted speech text, presentation material text

[0887] Output: Match data

[0888] Specific operation: The server uses a text comparison algorithm to calculate the degree of match between the two data and saves the results.

[0889] Step 6:

[0890] Speaking evaluation

[0891] The server evaluates the appropriateness of the speaking style based on the volume data and emotion data, thereby determining how effective the speaker's speaking style is.

[0892] Input: volume data and emotion data

[0893] Output: Speaking evaluation data

[0894] Specific operation: The server analyzes the volume data and emotion data, evaluates the effectiveness of the speaking style, and saves the results.

[0895] Step 7:

[0896] Generate feedback

[0897] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide specific advice and suggestions for improvement to the user.

[0898] Input: Matching data, speaking style evaluation data

[0899] Output: Feedback data

[0900] Specific operation: The server uses a text generation algorithm to generate feedback based on a set of data and stores it in a database.

[0901] Step 8:

[0902] Providing feedback

[0903] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0904] Input: Feedback data

[0905] Output: Feedback displayed on the user's terminal

[0906] Specific operation: The server sends feedback data to the user's terminal, and the terminal displays it.

[0907] Step 9:

[0908] Collection and analysis of audio and document data from safety briefings

[0909] The server collects and analyzes the audio data of safety briefings held in factories and related materials. The collected data is converted into text and analyzed using the procedure described above.

[0910] Input: Audio data of safety briefing and related materials data

[0911] Output: Text data and analysis data

[0912] What it does: The server collects the safety briefing data and analyzes it using a speech recognition engine and text extraction library.

[0913] Step 10:

[0914] Evaluating the quality of safety briefings and providing suggestions for improvement

[0915] Based on the collected and analyzed data, the server evaluates the quality of the safety briefing and generates specific improvements.

[0916] Input: Analysis data

[0917] Output: Evaluation data and improvement proposal data

[0918] How it works: The server uses the analysis results to evaluate the quality of the safety briefing and uses a generative AI model to generate feedback suggesting improvements.

[0919] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0920] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[0921] Overall system overview

[0922] The system consists of the following main components:

[0923] 1. User Input Components

[0924] 2. Audio data processing component

[0925] 3. Analysis components of presentation materials

[0926] 4. Speaking Assessment Component

[0927] 5. Feedback Generation Component

[0928] 6. Feedback Provision Component

[0929] 7. Emotion Engine Component

[0930] System Operation

[0931] User Input

[0932] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[0933] Audio data processing

[0934] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[0935] Analysis of presentation materials

[0936] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[0937] Speaking evaluation

[0938] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[0939] Generate feedback

[0940] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[0941] Utilizing the Emotion Engine

[0942] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotions in real time from facial expressions and movements. The emotion data recognized in real time is combined with extracted data from the voice data and becomes an important factor in comprehensively evaluating speaking style.

[0943] Dynamic feedback adjustment

[0944] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, if the user is nervous, it might include advice like "Try to speak in a relaxed manner."

[0945] Providing feedback

[0946] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[0947] Specific examples

[0948] An example of a specific operation of the system is shown below.

[0949] User Input

[0950] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[0951] Audio data processing

[0952] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[0953] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[0954] Additionally, through external sentiment analysis services, we receive the following data:

[0955] Volume: 60 decibels on average

[0956] Emotions: High motivation, medium tension, low joy

[0957] Analysis of presentation materials

[0958] Extract the following text data from "Documents.pdf":

[0959] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[0960] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[0961] Speaking evaluation

[0962] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[0963] Utilizing the Emotion Engine

[0964] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[0965] Dynamic feedback adjustment

[0966] The server dynamically adjusts the feedback as follows:

[0967] Overall feedback:

[0968] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[0969] Increasing the volume will make it easier for the audience to hear.

[0970] Expressing your emotions will improve the overall impression of your presentation.

[0971] You seem nervous, so take a deep breath and relax.

[0972] Providing feedback

[0973] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[0974] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[0975] The processing flow will be explained below.

[0976] Step 1: The user uploads the presentation data and audio data to the system.

[0977] The user clicks the file selection button on the interface and selects the presentation data (PDF file) and audio data (WAV file).

[0978] The user clicks the "Upload" button and the selected file is sent to the server.

[0979] Step 2: The server receives the uploaded file.

[0980] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[0981] Step 3: The server sends the voice data to the voice recognition engine.

[0982] The server sends the audio file (WAV file) to the speech recognition API.

[0983] The server receives the converted text data from the speech recognition API.

[0984] Step 4: The server extracts volume data and emotion data from the audio data.

[0985] The server sends the audio data to an external emotion analysis service and receives volume data and emotion data.

[0986] Step 5: The server uses the emotion engine to recognize the user's emotions in real time.

[0987] The user's camera footage is sent to a server, which then uses an emotion engine to analyze emotions in real time.

[0988] The server stores the analysis results.

[0989] Step 6: The server converts the presentation data into text data.

[0990] The server converts the PDF file into text data using a text analysis library.

[0991] Step 7: The server compares the audio text with the text data of the presentation materials.

[0992] The server uses a text comparison algorithm to calculate the degree of match between the spoken text and the source text.

[0993] Step 8: The server evaluates the speaking style based on the volume data and emotion data.

[0994] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[0995] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[0996] Step 9: The server evaluates the speaking style comprehensively based on the recognized emotion data in real time.

[0997] The server integrates the analysis results of real-time emotional data and voice data to make a comprehensive evaluation of speaking style.

[0998] Step 10: The server generates comprehensive feedback.

[0999] Based on each analysis result, the server generates specific feedback on the content and delivery of the presentation.

[1000] Using a text generation algorithm, it provides users with specific advice and areas for improvement.

[1001] Step 11: The server sends the generated feedback to the user's terminal.

[1002] The server sends an HTTP response including the generated feedback, which is received by the user's device.

[1003] Step 12: The user's device displays the feedback.

[1004] The user's terminal analyzes the received feedback data and displays it on the interface.

[1005] Specific examples

[1006] User Input

[1007] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1008] Audio data processing

[1009] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1010] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1011] Additionally, through external sentiment analysis services, we receive the following data:

[1012] Volume: 60 decibels on average

[1013] Emotions: High motivation, medium tension, low joy

[1014] Analysis of presentation materials

[1015] Extract the following text data from "Documents.pdf":

[1016] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1017] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1018] Speaking evaluation

[1019] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1020] Utilizing the Emotion Engine

[1021] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[1022] Dynamic feedback adjustment

[1023] The server dynamically adjusts the feedback as follows:

[1024] Overall feedback:

[1025] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1026] Increasing the volume will make it easier for the audience to hear.

[1027] Expressing your emotions will improve the overall impression of your presentation.

[1028] You seem nervous, so take a deep breath and relax.

[1029] Providing feedback

[1030] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1031] Example 2

[1032] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1033] In today's world, there is a demand for systems that allow users to efficiently evaluate and improve their presentation content and speaking style, but current systems have difficulty providing comprehensive feedback that combines analysis of audio data, comparison of document data, and real-time emotion recognition. Furthermore, because they are unable to generate dynamic feedback that takes into account the user's real-time emotions, they are unable to provide specific and useful suggestions for improvement.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1035] In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the content of the presentation and speaking style, means for analyzing real-time emotion data acquired from the user, means for dynamically adjusting the content of the feedback based on the real-time emotion data, and means for providing feedback to the user's terminal. This enables comprehensive analysis of the audio data and material data and provision of specific feedback that takes real-time emotion into consideration.

[1036] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[1037] "Presentation material data" refers to digital data in PDF or text file format used as presentation or explanatory materials.

[1038] "Audio Data" means digital data in the form of a WAV file or other audio file that contains a recording of content presented by a user.

[1039] A "voice recognition engine" is software or a service for converting voice data into text data.

[1040] "Text data" is data in the form of a character string extracted from voice data by a voice recognition engine.

[1041] "Emotion analysis service" is an external service that analyzes and extracts volume data and user emotion data from audio and video data.

[1042] "Volume data" is data that indicates the volume of a sound extracted from audio data.

[1043] "Emotion data" is data that indicates the user's emotional state (for example, enthusiasm, tension, joy, etc.).

[1044] "Material text" is text data in character string format extracted from presentation material data.

[1045] The "degree of agreement" is an evaluation index that indicates the degree to which the contents of the audio text and the material text match.

[1046] "Speaking style evaluation" is the process of evaluating the appropriateness and effectiveness of a user's speaking style based on volume data and emotion data.

[1047] "Feedback" is information that includes specific advice and areas for improvement regarding the content and delivery of the presentation.

[1048] "Real-time emotional data" is data that indicates the user's current emotional state, obtained by analyzing video captured by the user's camera.

[1049] "Dynamic feedback adjustment" is the process of appropriately changing the content of feedback based on emotional data obtained in real time.

[1050] A "terminal" is a device such as a computer or smartphone that a user uses to access the system.

[1051] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[1052] Hardware and Software Configuration

[1053] User Input Components

[1054] Users first log in to the system and upload their presentation data (e.g., PDF or text file format) and audio data (e.g., WAV files) to the server. Users input data using a web interface, register files through a form, and click the upload button to send the information to the server.

[1055] Audio data processing component

[1056] The server receives the uploaded voice data, converts the voice data into text data using a speech recognition API of a cloud service (e.g., Google Cloud Speech-to-Text), and extracts volume data and emotion data from the voice data using an external emotion analysis service (e.g., IBM Watson Tone Analyzer).

[1057] Analysis components of presentation materials

[1058] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., Apache PDFBox) to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of match.

[1059] Speaking evaluation component

[1060] The server evaluates the appropriateness of the speaking style based on volume data and emotional data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1061] Feedback Generation Component

[1062] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology (e.g., a generative AI model) to provide the user with specific advice and suggestions for improvement.

[1063] Utilizing the Emotion Engine Component

[1064] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotional data in real time from facial expressions and movements. This real-time emotional data is also an important factor in comprehensively evaluating the speaking style.

[1065] Dynamic feedback adjustment

[1066] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, by including advice like "Try to speak in a relaxed manner" if the user is nervous.

[1067] Provide Feedback Component

[1068] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1069] Specific examples

[1070] An example of a specific operation of the system is shown below.

[1071] User Input

[1072] User "A" uploads the presentation materials "Materials.pdf" and the presentation audio file "Presentation.wav". The uploaded files are accepted and saved on the server.

[1073] Audio data processing

[1074] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[1075] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1076] Additionally, through IBM Watson Tone Analyzer, you receive the following data:

[1077] Volume: 60 decibels on average

[1078] Emotions: High motivation, medium tension, low joy

[1079] Analysis of presentation materials

[1080] Extract the following text data from "document.pdf" using Apache PDFBox:

[1081] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1082] The server compares the audio text with the source text and calculates a match of 75%.

[1083] Speaking evaluation

[1084] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1085] Utilizing the Emotion Engine

[1086] The emotion engine recognizes that the user is nervous.

[1087] Dynamic feedback adjustment

[1088] The server dynamically adjusts the feedback as follows:

[1089] Overall feedback:

[1090] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1091] Increasing the volume will make it easier for the audience to hear.

[1092] Expressing your emotions will improve the overall impression of your presentation.

[1093] You seem nervous, so take a deep breath and relax.

[1094] Providing feedback

[1095] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1096] Example prompts for generative AI models

[1097] Please analyze the data below and provide feedback on the content and delivery of your presentation:

[1098] Presentation materials:

[1099] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1100] Announcement audio text:

[1101] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1102] Volume data:

[1103] Average 60 decibels

[1104] Emotional Data:

[1105] High motivation, medium tension, low joy

[1106] Real-time emotions:

[1107] I'm nervous

[1108] When this prompt sentence is fed into a generative AI model, appropriate feedback is generated.

[1109] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1110] Step 1:

[1111] A user logs into the system. The user opens the web interface, enters their credentials (username and password) into the login form, and clicks the login button. The server receives this and checks the credentials against its database. If authentication is successful, the server issues a login session to the user and displays a welcome page.

[1112] Input: Username and Password

[1113] Output: Login session and welcome page

[1114] Step 2:

[1115] The user uploads the presentation materials and audio data. The user clicks the "Select File" button on the web interface and selects the presentation materials (e.g., materials.pdf) and audio data (e.g., presentation.wav). Once the selection is complete, the user clicks the "Upload" button, which sends the files to the server. The server receives the files and saves them in the specified folder.

[1116] Input: Presentation files and audio files

[1117] Output: Saved presentation materials and audio files

[1118] Step 3:

[1119] The server converts the audio data into text data. First, the server sends the saved audio file (Announcement.wav) to a cloud service's speech recognition API (e.g., Google Cloud Speech-to-Text). The API analyzes the audio data and returns the corresponding text data. The server receives this text data and stores it in a database.

[1120] Input: Audio file

[1121] Output: Text data converted from audio

[1122] Step 4:

[1123] The server extracts volume and emotion data from the audio data. The server sends the audio file to an emotion analysis service API (e.g., IBM Watson Tone Analyzer). The API analyzes the volume and emotion data and returns them. The server receives this data and stores it in a database.

[1124] Input: Audio file

[1125] Output: Volume data and emotion data

[1126] Step 5:

[1127] The server analyzes the presentation data as text data. The server analyzes the presentation file (e.g., material.pdf) using a text extraction library (e.g., Apache PDFBox) and extracts the text data. The extracted text data is stored in a database.

[1128] Input: Presentation materials file

[1129] Output: Text data extracted from the material

[1130] Step 6:

[1131] The server compares the audio text with the source text. The server retrieves the audio text and source text from the database and uses a text comparison algorithm to calculate the degree of match between the two. The result of the match is stored in the database.

[1132] Input: Audio text and source text

[1133] Output: Match evaluation result

[1134] Step 7:

[1135] The server evaluates the speaking style based on the volume data and emotion data. The server obtains the volume data and emotion data from the database and analyzes them using an algorithm to evaluate the appropriateness of the speaking style. The evaluation results are stored in the database.

[1136] Input: volume data and emotion data

[1137] Output: Speaking evaluation results

[1138] Step 8:

[1139] The server generates feedback on the content and speaking style of the presentation. The server uses natural language processing technology (e.g., generative AI model) to create feedback for the user based on the evaluation results. The feedback is stored in a database.

[1140] Input: Evaluation result

[1141] Output: Feedback

[1142] Step 9:

[1143] The server analyzes real-time emotion data acquired from the user. The server analyzes video captured from the user's camera in real time and extracts emotion data. This emotion data is stored in a database.

[1144] Input: Real-time video of the user

[1145] Output: Real-time emotion data

[1146] Step 10:

[1147] The server dynamically adjusts the feedback content based on real-time emotional data. The server uses the emotional data obtained in real time to change the feedback content as appropriate. For example, if the user is nervous, the server will include advice such as "Try to speak in a relaxed manner."

[1148] Input: Real-time emotion data

[1149] Output: Dynamically adjusted feedback

[1150] Step 11:

[1151] The server sends the generated feedback to the user's terminal, the server sends the final feedback to the user's terminal for the user to review, and the terminal displays the feedback on a web page.

[1152] Input: Feedback

[1153] Output: Feedback displayed on the terminal

[1154] (Application example 2)

[1155] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1156] In conventional presentation practice systems, when users receive feedback to evaluate their presentation content and speaking style, it is difficult for them to receive specific advice that appropriately takes into account changes in emotion and volume fluctuations. Furthermore, because feedback is not provided in real time, it is difficult for users to receive advice that corresponds to their momentary emotional state. The present invention aims to solve these problems.

[1157] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the presentation content and speaking style, means for recognizing the user's emotions in real time and dynamically adjusting the feedback, and means for providing feedback to the user's terminal. This enables the user to receive specific and dynamic feedback based on emotion recognition in real time.

[1158] "Means for users to upload presentation material data and audio data" refers to an interface that allows users to send their own presentation materials and audio to the system.

[1159] "Means for converting voice data into text data" refers to a function that analyzes input voice data and converts it into a corresponding text format.

[1160] "Means for extracting volume data and emotion data from audio data" refers to technology for identifying and extracting information related to volume and the speaker's emotion from collected audio data.

[1161] "Means for comparing presentation material data with converted text data" refers to a method for comparing the text of uploaded presentation materials with the text of converted audio data and evaluating the degree of agreement between them.

[1162] "Means for evaluating speaking style based on volume data and emotion data" refers to a mechanism for evaluating the effectiveness of a user's speaking style based on volume and emotion characteristics.

[1163] "Means for generating feedback on the content and speaking style of the presentation" refers to a function that generates specific advice and areas for improvement for the user regarding the content and speaking style of the presentation based on analyzed data.

[1164] "Means for recognizing a user's emotions in real time and dynamically adjusting feedback" refers to technology that recognizes a user's emotional state in real time and instantly changes the feedback content accordingly.

[1165] "Means for providing feedback to a user's terminal" refers to a system for transmitting the generated feedback to the user's device for display.

[1166] The following describes in detail the mode for carrying out the present invention. The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. This system recognizes emotions in real time and provides dynamic feedback to the user to improve the quality of the presentation.

[1167] Overall system overview

[1168] The system consists of the following main components:

[1169] 1. User Input Components

[1170] 2. Audio data processing component

[1171] 3. Analysis components of presentation materials

[1172] 4. Speaking Assessment Component

[1173] 5. Feedback Generation Component

[1174] 6. Real-time Emotion Recognition Component

[1175] 7. Feedback Provision Component

[1176] System Operation

[1177] User Input

[1178] Users first log in to the system, then upload their presentation materials (PDF or text file format) and audio data (WAV file format) through the web interface, allowing the system to receive the user's presentation materials and audio data.

[1179] Audio data processing

[1180] The server receives the uploaded audio data and converts it into text using a speech recognition engine such as the Google Cloud Speech-to-Text API, and extracts volume and emotion data using external emotion analysis services such as the Microsoft Azure Emotion Recognition API.

[1181] Analysis of presentation materials

[1182] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[1183] Speaking evaluation

[1184] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1185] Generate feedback

[1186] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[1187] Real-time Emotion Recognition

[1188] The server analyzes the video data acquired from the camera and recognizes the user's emotions in real time, which makes it possible to immediately provide advice such as "Try to speak in a relaxed manner" if the user is nervous.

[1189] Providing Feedback

[1190] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1191] Specific examples

[1192] An example of a specific operation of the system is shown below.

[1193] User Input

[1194] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1195] Audio data processing

[1196] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1197] "Hello, my name is A. Today I'd like to talk about the latest developments in AI technology."

[1198] Additionally, through external sentiment analysis services, we receive the following data:

[1199] Volume: 60 decibels on average

[1200] Emotions: High motivation, medium tension, low joy

[1201] Analysis of presentation materials

[1202] Extract the following text data from "Documents.pdf":

[1203] "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[1204] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1205] Speaking evaluation

[1206] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1207] Real-time Emotion Recognition

[1208] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[1209] Generating and Providing Feedback

[1210] The server dynamically adjusts the feedback as follows:

[1211] "General feedback:

[1212] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1213] Increasing the volume will make it easier for the audience to hear.

[1214] Expressing your emotions will improve the overall impression of your presentation.

[1215] You seem nervous, so take a deep breath and relax."

[1216] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1217] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[1218] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1219] Step 1:

[1220] A user first logs into the system, where they enter their account information and are authenticated. The information entered is securely stored by Firebase Authentication. The output is that the user is authenticated and granted access to the system.

[1221] Step 2:

[1222] Users upload presentation data (PDF or text file format) and audio data (WAV file format) to the system. The data selected by the user is uploaded to Firebase Storage and saved on the server side. The input data are presentation data and audio files, and these data are saved on the server as output.

[1223] Step 3:

[1224] The server sends the uploaded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. A speech recognition algorithm is used to convert the voice data (input) into text data (output). The text data is generated and saved.

[1225] Step 4:

[1226] The server uses the Microsoft Azure Emotion Recognition API to extract volume and emotion data from the audio data. Audio data is provided as input, and volume and emotion information is obtained as output. This data is used to analyze user emotions and volume fluctuations.

[1227] Step 5:

[1228] The server analyzes the presentation data as text data. In the case of PDF files, it uses a text extraction library to extract the text within the document. The input data is the presentation data, and the output is the extracted text data. This data is used to compare with the audio text.

[1229] Step 6:

[1230] The server compares the converted speech text with the text data of the presentation materials and evaluates the degree of match. The speech text and the material text are provided as input, and the degree of match is calculated as output. The degree of match indicates how faithfully the user followed the materials in their presentation.

[1231] Step 7:

[1232] The server evaluates the appropriateness of speaking style based on volume data and emotional data. Volume and emotional information are used as input data, and the evaluation results of speaking style are obtained as output. The evaluation results include the appropriateness of volume and the degree of emotional expression.

[1233] Step 8:

[1234] The server uses a text generation algorithm to generate feedback based on the analysis results. The input data is the speaking style evaluation results and the degree of agreement with the presentation content, and the output is specific feedback, including suggestions for improvement and advice.

[1235] Step 9:

[1236] The server recognizes the user's emotions in real time and dynamically adjusts the feedback. The input is real-time video data, and the output is dynamically adjusted feedback. The video data is captured through a camera and analyzed by the emotion engine.

[1237] Step 10:

[1238] The generated feedback is sent to the user's device, which then displays the received feedback. Specific improvements and advice are provided to the user as output, allowing the user to improve their presentation. The user can then review the feedback and use it in their next practice.

[1239] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1240] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1241] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1242] [Third embodiment]

[1243] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1244] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1245] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1246] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1247] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1248] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1249] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1250] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1251] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1252] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1253] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1254] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1255] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[1256] Overall system overview

[1257] The system consists of the following main components:

[1258] 1. User Input Components

[1259] 2. Audio data processing component

[1260] 3. Analysis components of presentation materials

[1261] 4. Speaking Assessment Component

[1262] 5. Feedback Generation Component

[1263] 6. Feedback Provision Component

[1264] System Operation

[1265] User Input

[1266] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[1267] Audio data processing

[1268] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[1269] Analysis of presentation materials

[1270] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[1271] Speaking evaluation

[1272] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1273] Generate feedback

[1274] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[1275] Providing feedback

[1276] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1277] Specific examples

[1278] An example of a specific operation of the system is shown below.

[1279] User Input

[1280] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1281] Audio data processing

[1282] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1283] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1284] Additionally, through external sentiment analysis services such as Empath, we receive the following data:

[1285] Volume: 60 decibels on average

[1286] Emotions: High motivation, medium tension, low joy

[1287] Analysis of presentation materials

[1288] Extract the following text data from "Documents.pdf":

[1289] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1290] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1291] Speaking evaluation

[1292] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1293] Generate feedback

[1294] The server generates feedback like this:

[1295] Overall feedback:

[1296] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1297] Increasing the volume will make it easier for the audience to hear.

[1298] Expressing your emotions will improve the overall impression of your presentation.

[1299] Providing feedback

[1300] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1301] In this way, the system of the present invention provides detailed and specific feedback to the user, helping them improve the quality of their presentation or presentation.

[1302] The processing flow will be explained below.

[1303] Step 1: The user uploads the presentation data (PDF file) and audio data (WAV file) to the system.

[1304] The user clicks on the file selection button on the interface to select the file to upload.

[1305] The user clicks the "Upload" button to send the selected file to the server.

[1306] Step 2: The server receives the uploaded file.

[1307] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[1308] Step 3: The server sends the voice data to the voice recognition engine.

[1309] The server sends the uploaded audio file (WAV file) to a speech recognition API (e.g., a cloud speech recognition service).

[1310] The server receives the converted text data from the speech recognition API.

[1311] Step 4: The server extracts volume data and emotion data from the audio data.

[1312] The server sends the audio data to an external emotion analysis API (e.g., Empath).

[1313] The server receives volume data and emotional data (e.g., enthusiasm, tension, joy, etc.) from the emotion analysis API.

[1314] Step 5: The server converts the presentation data into text data.

[1315] The server converts the uploaded PDF file into text data using a text analysis library (e.g., a PDF text extraction tool).

[1316] Step 6: The server compares the audio text with the text data of the presentation materials.

[1317] The server uses a text comparison algorithm (e.g., Jaccard similarity or Levenshtein distance) to calculate the degree of similarity between the spoken text and the source text.

[1318] Step 7: The server evaluates the speaking style based on the volume data and emotion data.

[1319] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[1320] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[1321] Step 8: The server generates comprehensive feedback.

[1322] Based on each analysis, the server uses a text generation algorithm to generate specific feedback about the content and delivery of the presentation.

[1323] Step 9: The server sends the generated feedback to the user's terminal.

[1324] The server sends an HTTP response containing the generated feedback, which is received by the user's device.

[1325] Step 10: The user's device displays the feedback.

[1326] The user's device analyzes the received feedback data and displays it in an easy-to-read format on the interface.

[1327] Example 1

[1328] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1329] Current presentation evaluation systems have limited means of specifically evaluating the logical consistency of presentation content and the effectiveness of speaking style. This makes it difficult for users to obtain specific feedback to improve the quality of their presentations. Additionally, there is a lack of use of external services for emotion analysis and volume evaluation, making it difficult to obtain highly accurate analysis results. To solve these problems, a system that can provide effective and comprehensive feedback is needed.

[1330] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1331] In this invention, the server includes means for users to upload presentation material data and audio data, means for converting the audio data into text data, and means for extracting volume data and emotion data from the audio data, thereby enabling the server to analyze the presentation material and audio data entered by the user and provide detailed and specific feedback.

[1332] "User" refers to a person who logs into the system and uploads presentation material data and audio data.

[1333] "Presentation material data" refers to document data that describes the content of the presentation, and is provided in PDF or text file format.

[1334] "Audio data" refers to audio data recorded from a user's presentation, and is provided in a format such as a WAV file.

[1335] "Upload" refers to the operation of a user sending data from their own terminal to the system's server.

[1336] "Web interface" refers to a function that provides a screen for users to access the system via the Internet and operate or transmit data.

[1337] "Speech Recognition API" refers to the application programming interface of a cloud service used to convert voice data into text data.

[1338] "Text Extraction Library" means a software library for extracting text data from PDF and other document files.

[1339] "Sentiment Analysis API" refers to an application programming interface for an external service used to analyze emotion data from audio data.

[1340] "Natural language processing technology" refers to a general term for algorithms and technologies that enable computers to understand and generate human language.

[1341] "Feedback" refers to the evaluation results that include specific advice and areas for improvement regarding the user's presentation.

[1342] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[1343] Overall system overview

[1344] The system consists of the following main components:

[1345] 1. User Input Components

[1346] 2. Audio data processing component

[1347] 3. Analysis components of presentation materials

[1348] 4. Speaking Assessment Component

[1349] 5. Feedback Generation Component

[1350] 6. Feedback Provision Component

[1351] Hardware and software used

[1352] The system uses the following hardware and software:

[1353] Cloud service speech recognition API (e.g. Google Cloud Speech-to-Text API)

[1354] External sentiment analysis API (e.g. Empath)

[1355] A text extraction library (e.g., Apache PDFBox)

[1356] Natural language processing technology (e.g., GPT-3)

[1357] System Operation

[1358] User Input

[1359] Users log in to the system using a web interface and upload presentation data (PDF or text file format) and audio data (WAV files, etc.). By clicking the upload button, the data is sent to the server.

[1360] Audio data processing

[1361] After receiving the uploaded voice data, the server converts the voice data into text data using the cloud service's voice recognition API, which is then stored on the server. Furthermore, the server uses an external emotion analysis API to obtain volume data and emotion data from the voice data.

[1362] Analysis of presentation materials

[1363] The server analyzes the presentation data uploaded by the user. In the case of PDF files, it uses a text extraction library to extract the text from the document. It then compares this text data with the text data converted from the audio data and evaluates the degree of content agreement.

[1364] Speaking evaluation

[1365] The server evaluates the appropriateness of speaking based on volume and emotion data. It analyzes average volume, volume fluctuations, and emotion labels to determine how effective a user's speech is.

[1366] Generate feedback

[1367] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology to provide the user with specific advice and suggestions for improvement.

[1368] Providing feedback

[1369] The server sends the generated feedback to the user's device, and the user's device displays the received feedback on a web interface, allowing the user to understand specific areas for improvement.

[1370] Specific examples

[1371] An example of a specific operation of the system is shown below.

[1372] User Input

[1373] User "A" uploads the presentation materials ("materials.pdf") and the presentation audio file ("presentation.wav") through the web interface. These data are accepted and stored on the server.

[1374] Audio data processing

[1375] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[1376] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1377] Additionally, we receive the following data through Empath's sentiment analysis service:

[1378] Volume: 60 decibels on average

[1379] Emotions: High motivation, medium tension, low joy

[1380] Analysis of presentation materials

[1381] Extract the following text data from "Documents.pdf":

[1382] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1383] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1384] Speaking evaluation

[1385] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1386] Generate feedback

[1387] The server generates feedback like this:

[1388] Overall feedback:

[1389] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1390] Increasing the volume will make it easier for the audience to hear.

[1391] Expressing your emotions will improve the overall impression of your presentation.

[1392] Providing feedback

[1393] The generated feedback is sent to the user's terminal and displayed on a web interface.

[1394] Prompt Sentence Examples

[1395] Here are some example prompts for using generative AI models:

[1396] Analyze user-uploaded audio files (WAV format) and generate feedback assessing:

[1397] How well does the content of the talk match the presentation materials?

[1398] Volume and its fluctuations

[1399] Emotional expression (motivation, tension, joy, etc.)

[1400] Here are the analysis results:

[1401] Voice text: "Hello, my name is A. Today I'll be talking about the latest developments in AI technology."

[1402] Source text: "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[1403] Volume: 60 decibels on average

[1404] Emotions: High motivation, medium tension, low joy

[1405] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1406] Step 1: User uploads presentation data and audio data

[1407] Users log in to the system through a web interface. After logging in, they select the presentation data (e.g., a PDF file called "Materials.pdf") and audio data (e.g., a WAV file called "Presentation.wav") and click the upload button. The input is sent from the user's device and received by the server. The output is that the uploaded files are saved on the server. Specifically, the user selects "Materials.pdf" and "Presentation.wav" on the file selection screen and clicks the "Upload" button.

[1408] Step 2: Convert audio data to text

[1409] The server sends the received audio data ("Announcement.wav") to the cloud service's speech recognition API, which converts the audio data into text data. The input is the "Announcement.wav" file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology."). Specifically, the server sends "Announcement.wav" to the Google Cloud Speech-to-Text API, receives the converted text data, and saves it.

[1410] Step 3: Extract emotion and volume data

[1411] The server sends the converted voice-to-text data to an external emotion analysis API and extracts volume and emotion data. The input is voice data ("Announcement.wav"), and the output is volume data (e.g., average 60 decibels) and emotion data (e.g., high enthusiasm, medium tension, low joy). Specifically, the server sends the data to the Empath API, obtains the analysis results, and saves them.

[1412] Step 4: Analyze the presentation

[1413] The server analyzes the uploaded presentation data ("materials.pdf"). Specifically, it uses a text extraction library to extract the text within the document. The input is a PDF file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology. Specifically, I will explain the history of AI, current research directions, and future prospects."). Specifically, the server extracts and saves the text data using Apache PDFBox.

[1414] Step 5: Compare with the converted text

[1415] The server compares the text data converted from the speech with the text data in the presentation materials and calculates the degree of similarity. The input is the text data converted from the speech and the text data extracted from the presentation materials, and the output is the degree of similarity (e.g., 75%). Specifically, the server uses an algorithm to compare both sets of text data and evaluate the degree of similarity.

[1416] Step 6: Speech assessment

[1417] The server evaluates the appropriateness of speaking style based on volume data and emotional data. The input is volume data and emotional data, and the output is the evaluation result (e.g., "low volume" or "modest emotional expression"). Specifically, the server analyzes this data and evaluates problems and areas for improvement in speaking style.

[1418] Step 7: Generate feedback

[1419] The server generates comprehensive feedback based on the results of each analysis. The inputs are the degree of match, volume data, and emotion data, and the output is feedback text (e.g., "The content matches the first chapter of the materials. Please also talk about the contents of chapters two and three. By increasing the volume, you will be able to communicate more easily with the audience. By using more emotional expression, the overall impression of the presentation will be improved."). Specifically, the server generates the feedback text using natural language processing technology.

[1420] Step 8: Provide feedback

[1421] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the display result on the user's terminal. In concrete terms, the server sends the generated feedback to the user's terminal, and the user's terminal displays it on the web interface.

[1422] (Application example 1)

[1423] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1424] Conventional presentation materials and voice analysis systems are limited to providing feedback on individual presentations and speeches, and have the problem of being unable to provide specific suggestions for improving the quality of safety briefings, especially those held within factories. Furthermore, these types of systems only allow users to analyze a portion of a presentation, making it difficult to provide comprehensive feedback that leads to improved safety awareness. The present invention aims to provide a specific system for improving the content and quality of safety briefings within factories and improving safety awareness.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1426] In this invention, the server includes a means for users to upload presentation material data and audio data, a means for converting the audio data into text data, and a means for extracting volume data and emotion data from the audio data, which makes it possible to evaluate the degree of agreement between the presentation content and related materials, evaluate the appropriateness of the speaking style, and provide points for improving the quality of the safety briefing.

[1427] "Means for users to upload presentation material data and audio data" refers to means for users to send presentation material data and audio data to the system through a specified interface.

[1428] The "means for converting voice data into text data" refers to a means for analyzing uploaded voice data and converting it into text information.

[1429] The "means for extracting volume data and emotional data from audio data" refers to a means for extracting the volume level and emotional information from audio data using audio analysis technology.

[1430] The "means for comparing presentation material data with converted text data" is a means for analyzing the content of the presentation material and the text converted from the speech and evaluating the degree of agreement.

[1431] The "means for evaluating speaking style based on volume data and emotion data" is a means for evaluating a speaker's speaking style and presentation quality based on acquired volume data and emotion data.

[1432] The "means for generating feedback on the content and speaking style of the presentation" refers to a means for generating specific advice and points for improvement on the content and speaking style of the presentation based on the analysis results.

[1433] The "means for providing feedback to the user's terminal" refers to a means for transmitting the generated feedback to the user's device and displaying it.

[1434] The "means for collecting and analyzing audio data of safety briefings in factories and related document data" refers to a means for collecting and analyzing audio data of safety briefings held in factories and related document data.

[1435] "Means for evaluating the quality of safety briefings based on the analysis results and providing specific suggestions for improvement" refers to means for evaluating the quality of briefings based on collected and analyzed data and providing specific suggestions for improvement.

[1436] The present invention is a system that analyzes presentation material data and audio data uploaded by users and provides specific feedback on both the content and speaking style of the presentation. Furthermore, the system has the function of evaluating the quality of safety briefings in factories and providing specific points for improvement. The following describes in detail the embodiments of the present invention.

[1437] The system consists of the following main components:

[1438] 1. User Input Components

[1439] 2. Audio data processing component

[1440] 3. Analysis components of presentation materials

[1441] 4. Speaking Assessment Component

[1442] 5. Feedback Generation Component

[1443] 6. Feedback Provision Component

[1444] 7. Safety Briefing Analysis Component

[1445] System Operation

[1446] User Input

[1447] Users first log in to the system and upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[1448] Audio data processing

[1449] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text). This saves the content of the voice data as text data. The server then uses an external emotion analysis service (e.g., Empath) to extract volume data and emotion data from the voice data.

[1450] Analysis of presentation materials

[1451] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document. The server then compares the converted audio text with the presentation text data to evaluate the degree of logical correspondence.

[1452] Speaking evaluation

[1453] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1454] Generate feedback

[1455] The server generates comprehensive feedback based on the results of each of the above analyses, using a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide the user with specific advice and suggestions for improvement.

[1456] Providing feedback

[1457] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1458] Safety Briefing Analysis

[1459] The server analyzes the audio data of safety briefings held in factories and related document data. The audio data is converted into text data using the procedure described above, and the related document data is also analyzed for text. The server integrates this data, evaluates the quality of the briefings, and generates specific improvements.

[1460] Specific examples

[1461] For example, if a factory manager says, "Please be sure to wear a helmet for safety reasons," but the documentation does not include that instruction, the system will point out the discrepancy between the documentation and the audio content.

[1462] Prompt Sentence Examples

[1463] Prompts for the development of an automated safety briefing analysis system:

[1464] Analyze the audio data and PDF documents and evaluate the degree of match.

[1465] Extract emotional data from audio data and evaluate the appropriateness of speaking styles.

[1466] Please provide your overall feedback based on these analysis results.

[1467] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1468] Step 1:

[1469] User Input

[1470] Users upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. The input data is sent to the server, which then stores it.

[1471] Input: Presentation data and audio data

[1472] Output: Presentation data and audio data stored on the server

[1473] Specific operation: The user selects a file through a web browser and clicks the send button, which uploads the file to the server.

[1474] Step 2:

[1475] Converting audio data to text

[1476] The server receives the uploaded voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Through this process, the voice content is saved as text.

[1477] Input: Audio data

[1478] Output: Text data converted from audio data

[1479] Specific operation: The server sends the voice data to the speech recognition engine and stores the returned text data in the database.

[1480] Step 3:

[1481] Extracting volume and emotion data

[1482] The server sends the converted text data to an external emotion analysis service (e.g., Empath) to extract volume and emotion data, which then stores the speaker's emotional state and speaking style characteristics.

[1483] Input: Text data

[1484] Output: Volume data and emotion data

[1485] Specific operation: The server sends text data to the emotion analysis service, obtains the returned emotion data, and stores it.

[1486] Step 4:

[1487] Analysis of presentation data

[1488] The server analyzes the presentation data uploaded by the user. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document.

[1489] Input: Presentation data (PDF or text file)

[1490] Output: Text data of presentation materials

[1491] Specific operation: The server uses a text extraction library to extract content from PDF and text files and saves it in a database.

[1492] Step 5:

[1493] Comparison of presentation materials and audio text

[1494] The server compares the converted speech text with the presentation text, evaluates the degree of match, and stores the result as a numerical value indicating how well the presentation matches the presentation text.

[1495] Input: Converted speech text, presentation material text

[1496] Output: Match data

[1497] Specific operation: The server uses a text comparison algorithm to calculate the degree of match between the two data and saves the results.

[1498] Step 6:

[1499] Speaking evaluation

[1500] The server evaluates the appropriateness of the speaking style based on the volume data and emotion data, thereby determining how effective the speaker's speaking style is.

[1501] Input: volume data and emotion data

[1502] Output: Speaking evaluation data

[1503] Specific operation: The server analyzes the volume data and emotion data, evaluates the effectiveness of the speaking style, and saves the results.

[1504] Step 7:

[1505] Generate feedback

[1506] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide specific advice and suggestions for improvement to the user.

[1507] Input: Matching data, speaking style evaluation data

[1508] Output: Feedback data

[1509] Specific operation: The server uses a text generation algorithm to generate feedback based on a set of data and stores it in a database.

[1510] Step 8:

[1511] Providing feedback

[1512] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1513] Input: Feedback data

[1514] Output: Feedback displayed on the user's terminal

[1515] Specific operation: The server sends feedback data to the user's terminal, and the terminal displays it.

[1516] Step 9:

[1517] Collection and analysis of audio and document data from safety briefings

[1518] The server collects and analyzes the audio data of safety briefings held in factories and related materials. The collected data is converted into text and analyzed using the procedure described above.

[1519] Input: Audio data of safety briefing and related materials data

[1520] Output: Text data and analysis data

[1521] What it does: The server collects the safety briefing data and analyzes it using a speech recognition engine and text extraction library.

[1522] Step 10:

[1523] Evaluating the quality of safety briefings and providing suggestions for improvement

[1524] Based on the collected and analyzed data, the server evaluates the quality of the safety briefing and generates specific improvements.

[1525] Input: Analysis data

[1526] Output: Evaluation data and improvement proposal data

[1527] How it works: The server uses the analysis results to evaluate the quality of the safety briefing and uses a generative AI model to generate feedback suggesting improvements.

[1528] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1529] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[1530] Overall system overview

[1531] The system consists of the following main components:

[1532] 1. User Input Components

[1533] 2. Audio data processing component

[1534] 3. Analysis components of presentation materials

[1535] 4. Speaking Assessment Component

[1536] 5. Feedback Generation Component

[1537] 6. Feedback Provision Component

[1538] 7. Emotion Engine Component

[1539] System Operation

[1540] User Input

[1541] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[1542] Audio data processing

[1543] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[1544] Analysis of presentation materials

[1545] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[1546] Speaking evaluation

[1547] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1548] Generate feedback

[1549] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[1550] Utilizing the Emotion Engine

[1551] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotions in real time from facial expressions and movements. The emotion data recognized in real time is combined with extracted data from the voice data and becomes an important factor in comprehensively evaluating speaking style.

[1552] Dynamic feedback adjustment

[1553] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, if the user is nervous, it might include advice like "Try to speak in a relaxed manner."

[1554] Providing feedback

[1555] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1556] Specific examples

[1557] An example of a specific operation of the system is shown below.

[1558] User Input

[1559] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1560] Audio data processing

[1561] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1562] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1563] Additionally, through external sentiment analysis services, we receive the following data:

[1564] Volume: 60 decibels on average

[1565] Emotions: High motivation, medium tension, low joy

[1566] Analysis of presentation materials

[1567] Extract the following text data from "Documents.pdf":

[1568] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1569] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1570] Speaking evaluation

[1571] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1572] Utilizing the Emotion Engine

[1573] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[1574] Dynamic feedback adjustment

[1575] The server dynamically adjusts the feedback as follows:

[1576] Overall feedback:

[1577] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1578] Increasing the volume will make it easier for the audience to hear.

[1579] Expressing your emotions will improve the overall impression of your presentation.

[1580] You seem nervous, so take a deep breath and relax.

[1581] Providing feedback

[1582] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1583] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[1584] The processing flow will be explained below.

[1585] Step 1: The user uploads the presentation data and audio data to the system.

[1586] The user clicks the file selection button on the interface and selects the presentation data (PDF file) and audio data (WAV file).

[1587] The user clicks the "Upload" button and the selected file is sent to the server.

[1588] Step 2: The server receives the uploaded file.

[1589] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[1590] Step 3: The server sends the voice data to the voice recognition engine.

[1591] The server sends the audio file (WAV file) to the speech recognition API.

[1592] The server receives the converted text data from the speech recognition API.

[1593] Step 4: The server extracts volume data and emotion data from the audio data.

[1594] The server sends the audio data to an external emotion analysis service and receives volume data and emotion data.

[1595] Step 5: The server uses the emotion engine to recognize the user's emotions in real time.

[1596] The user's camera footage is sent to a server, which then uses an emotion engine to analyze emotions in real time.

[1597] The server stores the analysis results.

[1598] Step 6: The server converts the presentation data into text data.

[1599] The server converts the PDF file into text data using a text analysis library.

[1600] Step 7: The server compares the audio text with the text data of the presentation materials.

[1601] The server uses a text comparison algorithm to calculate the degree of match between the spoken text and the source text.

[1602] Step 8: The server evaluates the speaking style based on the volume data and emotion data.

[1603] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[1604] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[1605] Step 9: The server evaluates the speaking style comprehensively based on the recognized emotion data in real time.

[1606] The server integrates the analysis results of real-time emotional data and voice data to make a comprehensive evaluation of speaking style.

[1607] Step 10: The server generates comprehensive feedback.

[1608] Based on each analysis result, the server generates specific feedback on the content and delivery of the presentation.

[1609] Using a text generation algorithm, it provides users with specific advice and areas for improvement.

[1610] Step 11: The server sends the generated feedback to the user's terminal.

[1611] The server sends an HTTP response including the generated feedback, which is received by the user's device.

[1612] Step 12: The user's device displays the feedback.

[1613] The user's terminal analyzes the received feedback data and displays it on the interface.

[1614] Specific examples

[1615] User Input

[1616] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1617] Audio data processing

[1618] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1619] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1620] Additionally, through external sentiment analysis services, we receive the following data:

[1621] Volume: 60 decibels on average

[1622] Emotions: High motivation, medium tension, low joy

[1623] Analysis of presentation materials

[1624] Extract the following text data from "Documents.pdf":

[1625] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1626] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1627] Speaking evaluation

[1628] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1629] Utilizing the Emotion Engine

[1630] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[1631] Dynamic feedback adjustment

[1632] The server dynamically adjusts the feedback as follows:

[1633] Overall feedback:

[1634] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1635] Increasing the volume will make it easier for the audience to hear.

[1636] Expressing your emotions will improve the overall impression of your presentation.

[1637] You seem nervous, so take a deep breath and relax.

[1638] Providing feedback

[1639] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1640] Example 2

[1641] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1642] In today's world, there is a demand for systems that allow users to efficiently evaluate and improve their presentation content and speaking style, but current systems have difficulty providing comprehensive feedback that combines analysis of audio data, comparison of document data, and real-time emotion recognition. Furthermore, because they are unable to generate dynamic feedback that takes into account the user's real-time emotions, they are unable to provide specific and useful suggestions for improvement.

[1643] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1644] In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the content of the presentation and speaking style, means for analyzing real-time emotion data acquired from the user, means for dynamically adjusting the content of the feedback based on the real-time emotion data, and means for providing feedback to the user's terminal. This enables comprehensive analysis of the audio data and material data and provision of specific feedback that takes real-time emotion into consideration.

[1645] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[1646] "Presentation material data" refers to digital data in PDF or text file format used as presentation or explanatory materials.

[1647] "Audio Data" means digital data in the form of a WAV file or other audio file that contains a recording of content presented by a user.

[1648] A "voice recognition engine" is software or a service for converting voice data into text data.

[1649] "Text data" is data in the form of a character string extracted from voice data by a voice recognition engine.

[1650] "Emotion analysis service" is an external service that analyzes and extracts volume data and user emotion data from audio and video data.

[1651] "Volume data" is data that indicates the volume of a sound extracted from audio data.

[1652] "Emotion data" is data that indicates the user's emotional state (for example, enthusiasm, tension, joy, etc.).

[1653] "Material text" is text data in character string format extracted from presentation material data.

[1654] The "degree of agreement" is an evaluation index that indicates the degree to which the contents of the audio text and the material text match.

[1655] "Speaking style evaluation" is the process of evaluating the appropriateness and effectiveness of a user's speaking style based on volume data and emotion data.

[1656] "Feedback" is information that includes specific advice and areas for improvement regarding the content and delivery of the presentation.

[1657] "Real-time emotional data" is data that indicates the user's current emotional state, obtained by analyzing video captured by the user's camera.

[1658] "Dynamic feedback adjustment" is the process of appropriately changing the content of feedback based on emotional data obtained in real time.

[1659] A "terminal" is a device such as a computer or smartphone that a user uses to access the system.

[1660] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[1661] Hardware and Software Configuration

[1662] User Input Components

[1663] Users first log in to the system and upload their presentation data (e.g., PDF or text file format) and audio data (e.g., WAV files) to the server. Users input data using a web interface, register files through a form, and click the upload button to send the information to the server.

[1664] Audio data processing component

[1665] The server receives the uploaded voice data, converts the voice data into text data using a speech recognition API of a cloud service (e.g., Google Cloud Speech-to-Text), and extracts volume data and emotion data from the voice data using an external emotion analysis service (e.g., IBM Watson Tone Analyzer).

[1666] Analysis components of presentation materials

[1667] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., Apache PDFBox) to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of match.

[1668] Speaking evaluation component

[1669] The server evaluates the appropriateness of the speaking style based on volume data and emotional data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1670] Feedback Generation Component

[1671] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology (e.g., a generative AI model) to provide the user with specific advice and suggestions for improvement.

[1672] Utilizing the Emotion Engine Component

[1673] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotional data in real time from facial expressions and movements. This real-time emotional data is also an important factor in comprehensively evaluating the speaking style.

[1674] Dynamic feedback adjustment

[1675] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, by including advice like "Try to speak in a relaxed manner" if the user is nervous.

[1676] Provide Feedback Component

[1677] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1678] Specific examples

[1679] An example of a specific operation of the system is shown below.

[1680] User Input

[1681] User "A" uploads the presentation materials "Materials.pdf" and the presentation audio file "Presentation.wav". The uploaded files are accepted and saved on the server.

[1682] Audio data processing

[1683] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[1684] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1685] Additionally, through IBM Watson Tone Analyzer, you receive the following data:

[1686] Volume: 60 decibels on average

[1687] Emotions: High motivation, medium tension, low joy

[1688] Analysis of presentation materials

[1689] Extract the following text data from "document.pdf" using Apache PDFBox:

[1690] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1691] The server compares the audio text with the source text and calculates a match of 75%.

[1692] Speaking evaluation

[1693] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1694] Utilizing the Emotion Engine

[1695] The emotion engine recognizes that the user is nervous.

[1696] Dynamic feedback adjustment

[1697] The server dynamically adjusts the feedback as follows:

[1698] Overall feedback:

[1699] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1700] Increasing the volume will make it easier for the audience to hear.

[1701] Expressing your emotions will improve the overall impression of your presentation.

[1702] You seem nervous, so take a deep breath and relax.

[1703] Providing feedback

[1704] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1705] Example prompts for generative AI models

[1706] Please analyze the data below and provide feedback on the content and delivery of your presentation:

[1707] Presentation materials:

[1708] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1709] Announcement audio text:

[1710] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1711] Volume data:

[1712] Average 60 decibels

[1713] Emotional Data:

[1714] High motivation, medium tension, low joy

[1715] Real-time emotions:

[1716] I'm nervous

[1717] When this prompt sentence is fed into a generative AI model, appropriate feedback is generated.

[1718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1719] Step 1:

[1720] A user logs into the system. The user opens the web interface, enters their credentials (username and password) into the login form, and clicks the login button. The server receives this and checks the credentials against its database. If authentication is successful, the server issues a login session to the user and displays a welcome page.

[1721] Input: Username and Password

[1722] Output: Login session and welcome page

[1723] Step 2:

[1724] The user uploads the presentation materials and audio data. The user clicks the "Select File" button on the web interface and selects the presentation materials (e.g., materials.pdf) and audio data (e.g., presentation.wav). Once the selection is complete, the user clicks the "Upload" button, which sends the files to the server. The server receives the files and saves them in the specified folder.

[1725] Input: Presentation files and audio files

[1726] Output: Saved presentation materials and audio files

[1727] Step 3:

[1728] The server converts the audio data into text data. First, the server sends the saved audio file (Announcement.wav) to a cloud service's speech recognition API (e.g., Google Cloud Speech-to-Text). The API analyzes the audio data and returns the corresponding text data. The server receives this text data and stores it in a database.

[1729] Input: Audio file

[1730] Output: Text data converted from audio

[1731] Step 4:

[1732] The server extracts volume and emotion data from the audio data. The server sends the audio file to an emotion analysis service API (e.g., IBM Watson Tone Analyzer). The API analyzes the volume and emotion data and returns them. The server receives this data and stores it in a database.

[1733] Input: Audio file

[1734] Output: Volume data and emotion data

[1735] Step 5:

[1736] The server analyzes the presentation data as text data. The server analyzes the presentation file (e.g., material.pdf) using a text extraction library (e.g., Apache PDFBox) and extracts the text data. The extracted text data is stored in a database.

[1737] Input: Presentation materials file

[1738] Output: Text data extracted from the material

[1739] Step 6:

[1740] The server compares the audio text with the source text. The server retrieves the audio text and source text from the database and uses a text comparison algorithm to calculate the degree of match between the two. The result of the match is stored in the database.

[1741] Input: Audio text and source text

[1742] Output: Match evaluation result

[1743] Step 7:

[1744] The server evaluates the speaking style based on the volume data and emotion data. The server obtains the volume data and emotion data from the database and analyzes them using an algorithm to evaluate the appropriateness of the speaking style. The evaluation results are stored in the database.

[1745] Input: volume data and emotion data

[1746] Output: Speaking evaluation results

[1747] Step 8:

[1748] The server generates feedback on the content and speaking style of the presentation. The server uses natural language processing technology (e.g., generative AI model) to create feedback for the user based on the evaluation results. The feedback is stored in a database.

[1749] Input: Evaluation result

[1750] Output: Feedback

[1751] Step 9:

[1752] The server analyzes real-time emotion data acquired from the user. The server analyzes video captured from the user's camera in real time and extracts emotion data. This emotion data is stored in a database.

[1753] Input: Real-time video of the user

[1754] Output: Real-time emotion data

[1755] Step 10:

[1756] The server dynamically adjusts the feedback content based on real-time emotional data. The server uses the emotional data obtained in real time to change the feedback content as appropriate. For example, if the user is nervous, the server will include advice such as "Try to speak in a relaxed manner."

[1757] Input: Real-time emotion data

[1758] Output: Dynamically adjusted feedback

[1759] Step 11:

[1760] The server sends the generated feedback to the user's terminal, the server sends the final feedback to the user's terminal for the user to review, and the terminal displays the feedback on a web page.

[1761] Input: Feedback

[1762] Output: Feedback displayed on the terminal

[1763] (Application example 2)

[1764] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1765] In conventional presentation practice systems, when users receive feedback to evaluate their presentation content and speaking style, it is difficult for them to receive specific advice that appropriately takes into account changes in emotion and volume fluctuations. Furthermore, because feedback is not provided in real time, it is difficult for users to receive advice that corresponds to their momentary emotional state. The present invention aims to solve these problems.

[1766] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the presentation content and speaking style, means for recognizing the user's emotions in real time and dynamically adjusting the feedback, and means for providing feedback to the user's terminal. This enables the user to receive specific and dynamic feedback based on emotion recognition in real time.

[1767] "Means for users to upload presentation material data and audio data" refers to an interface that allows users to send their own presentation materials and audio to the system.

[1768] "Means for converting voice data into text data" refers to a function that analyzes input voice data and converts it into a corresponding text format.

[1769] "Means for extracting volume data and emotion data from audio data" refers to technology for identifying and extracting information related to volume and the speaker's emotion from collected audio data.

[1770] "Means for comparing presentation material data with converted text data" refers to a method for comparing the text of uploaded presentation materials with the text of converted audio data and evaluating the degree of agreement between them.

[1771] "Means for evaluating speaking style based on volume data and emotion data" refers to a mechanism for evaluating the effectiveness of a user's speaking style based on volume and emotion characteristics.

[1772] "Means for generating feedback on the content and speaking style of the presentation" refers to a function that generates specific advice and areas for improvement for the user regarding the content and speaking style of the presentation based on analyzed data.

[1773] "Means for recognizing a user's emotions in real time and dynamically adjusting feedback" refers to technology that recognizes a user's emotional state in real time and instantly changes the feedback content accordingly.

[1774] "Means for providing feedback to a user's terminal" refers to a system for transmitting the generated feedback to the user's device for display.

[1775] The following describes in detail the mode for carrying out the present invention. The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. This system recognizes emotions in real time and provides dynamic feedback to the user to improve the quality of the presentation.

[1776] Overall system overview

[1777] The system consists of the following main components:

[1778] 1. User Input Components

[1779] 2. Audio data processing component

[1780] 3. Analysis components of presentation materials

[1781] 4. Speaking Assessment Component

[1782] 5. Feedback Generation Component

[1783] 6. Real-time Emotion Recognition Component

[1784] 7. Feedback Provision Component

[1785] System Operation

[1786] User Input

[1787] Users first log in to the system, then upload their presentation materials (PDF or text file format) and audio data (WAV file format) through the web interface, allowing the system to receive the user's presentation materials and audio data.

[1788] Audio data processing

[1789] The server receives the uploaded audio data and converts it into text using a speech recognition engine such as the Google Cloud Speech-to-Text API, and extracts volume and emotion data using external emotion analysis services such as the Microsoft Azure Emotion Recognition API.

[1790] Analysis of presentation materials

[1791] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[1792] Speaking evaluation

[1793] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1794] Generate feedback

[1795] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[1796] Real-time Emotion Recognition

[1797] The server analyzes the video data acquired from the camera and recognizes the user's emotions in real time, which makes it possible to immediately provide advice such as "Try to speak in a relaxed manner" if the user is nervous.

[1798] Providing Feedback

[1799] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1800] Specific examples

[1801] An example of a specific operation of the system is shown below.

[1802] User Input

[1803] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1804] Audio data processing

[1805] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1806] "Hello, my name is A. Today I'd like to talk about the latest developments in AI technology."

[1807] Additionally, through external sentiment analysis services, we receive the following data:

[1808] Volume: 60 decibels on average

[1809] Emotions: High motivation, medium tension, low joy

[1810] Analysis of presentation materials

[1811] Extract the following text data from "Documents.pdf":

[1812] "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[1813] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1814] Speaking evaluation

[1815] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1816] Real-time Emotion Recognition

[1817] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[1818] Generating and Providing Feedback

[1819] The server dynamically adjusts the feedback as follows:

[1820] "General feedback:

[1821] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1822] Increasing the volume will make it easier for the audience to hear.

[1823] Expressing your emotions will improve the overall impression of your presentation.

[1824] You seem nervous, so take a deep breath and relax."

[1825] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1826] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[1827] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1828] Step 1:

[1829] A user first logs into the system, where they enter their account information and are authenticated. The information entered is securely stored by Firebase Authentication. The output is that the user is authenticated and granted access to the system.

[1830] Step 2:

[1831] Users upload presentation data (PDF or text file format) and audio data (WAV file format) to the system. The data selected by the user is uploaded to Firebase Storage and saved on the server side. The input data are presentation data and audio files, and these data are saved on the server as output.

[1832] Step 3:

[1833] The server sends the uploaded voice data to the Google Cloud Speech-to-Text API, which converts the voice data into text data. A speech recognition algorithm is used to convert the voice data (input) into text data (output). The text data is generated and saved.

[1834] Step 4:

[1835] The server uses the Microsoft Azure Emotion Recognition API to extract volume and emotion data from the audio data. Audio data is provided as input, and volume and emotion information is obtained as output. This data is used to analyze user emotions and volume fluctuations.

[1836] Step 5:

[1837] The server analyzes the presentation data as text data. In the case of PDF files, it uses a text extraction library to extract the text within the document. The input data is the presentation data, and the output is the extracted text data. This data is used to compare with the audio text.

[1838] Step 6:

[1839] The server compares the converted speech text with the text data of the presentation materials and evaluates the degree of match. The speech text and the material text are provided as input, and the degree of match is calculated as output. The degree of match indicates how faithfully the user followed the materials in their presentation.

[1840] Step 7:

[1841] The server evaluates the appropriateness of speaking style based on volume data and emotional data. Volume and emotional information are used as input data, and the evaluation results of speaking style are obtained as output. The evaluation results include the appropriateness of volume and the degree of emotional expression.

[1842] Step 8:

[1843] The server uses a text generation algorithm to generate feedback based on the analysis results. The input data is the speaking style evaluation results and the degree of agreement with the presentation content, and the output is specific feedback, including suggestions for improvement and advice.

[1844] Step 9:

[1845] The server recognizes the user's emotions in real time and dynamically adjusts the feedback. The input is real-time video data, and the output is dynamically adjusted feedback. The video data is captured through a camera and analyzed by the emotion engine.

[1846] Step 10:

[1847] The generated feedback is sent to the user's device, which then displays the received feedback. Specific improvements and advice are provided to the user as output, allowing the user to improve their presentation. The user can then review the feedback and use it in their next practice.

[1848] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1849] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1850] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1851] [Fourth embodiment]

[1852] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1853] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1854] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1855] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1856] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1857] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1858] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1859] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1860] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1861] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1862] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1863] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1864] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1865] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[1866] Overall system overview

[1867] The system consists of the following main components:

[1868] 1. User Input Components

[1869] 2. Audio data processing component

[1870] 3. Analysis components of presentation materials

[1871] 4. Speaking Assessment Component

[1872] 5. Feedback Generation Component

[1873] 6. Feedback Provision Component

[1874] System Operation

[1875] User Input

[1876] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[1877] Audio data processing

[1878] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[1879] Analysis of presentation materials

[1880] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[1881] Speaking evaluation

[1882] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[1883] Generate feedback

[1884] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[1885] Providing feedback

[1886] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[1887] Specific examples

[1888] An example of a specific operation of the system is shown below.

[1889] User Input

[1890] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[1891] Audio data processing

[1892] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[1893] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1894] Additionally, through external sentiment analysis services such as Empath, we receive the following data:

[1895] Volume: 60 decibels on average

[1896] Emotions: High motivation, medium tension, low joy

[1897] Analysis of presentation materials

[1898] Extract the following text data from "Documents.pdf":

[1899] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1900] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1901] Speaking evaluation

[1902] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1903] Generate feedback

[1904] The server generates feedback like this:

[1905] Overall feedback:

[1906] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[1907] Increasing the volume will make it easier for the audience to hear.

[1908] Expressing your emotions will improve the overall impression of your presentation.

[1909] Providing feedback

[1910] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[1911] In this way, the system of the present invention provides detailed and specific feedback to the user, helping them improve the quality of their presentation or presentation.

[1912] The processing flow will be explained below.

[1913] Step 1: The user uploads the presentation data (PDF file) and audio data (WAV file) to the system.

[1914] The user clicks on the file selection button on the interface to select the file to upload.

[1915] The user clicks the "Upload" button to send the selected file to the server.

[1916] Step 2: The server receives the uploaded file.

[1917] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[1918] Step 3: The server sends the voice data to the voice recognition engine.

[1919] The server sends the uploaded audio file (WAV file) to a speech recognition API (e.g., a cloud speech recognition service).

[1920] The server receives the converted text data from the speech recognition API.

[1921] Step 4: The server extracts volume data and emotion data from the audio data.

[1922] The server sends the audio data to an external emotion analysis API (e.g., Empath).

[1923] The server receives volume data and emotional data (e.g., enthusiasm, tension, joy, etc.) from the emotion analysis API.

[1924] Step 5: The server converts the presentation data into text data.

[1925] The server converts the uploaded PDF file into text data using a text analysis library (e.g., a PDF text extraction tool).

[1926] Step 6: The server compares the audio text with the text data of the presentation materials.

[1927] The server uses a text comparison algorithm (e.g., Jaccard similarity or Levenshtein distance) to calculate the degree of similarity between the spoken text and the source text.

[1928] Step 7: The server evaluates the speaking style based on the volume data and emotion data.

[1929] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[1930] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[1931] Step 8: The server generates comprehensive feedback.

[1932] Based on each analysis, the server uses a text generation algorithm to generate specific feedback about the content and delivery of the presentation.

[1933] Step 9: The server sends the generated feedback to the user's terminal.

[1934] The server sends an HTTP response containing the generated feedback, which is received by the user's device.

[1935] Step 10: The user's device displays the feedback.

[1936] The user's device analyzes the received feedback data and displays it in an easy-to-read format on the interface.

[1937] Example 1

[1938] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1939] Current presentation evaluation systems have limited means of specifically evaluating the logical consistency of presentation content and the effectiveness of speaking style. This makes it difficult for users to obtain specific feedback to improve the quality of their presentations. Additionally, there is a lack of use of external services for emotion analysis and volume evaluation, making it difficult to obtain highly accurate analysis results. To solve these problems, a system that can provide effective and comprehensive feedback is needed.

[1940] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1941] In this invention, the server includes means for users to upload presentation material data and audio data, means for converting the audio data into text data, and means for extracting volume data and emotion data from the audio data, thereby enabling the server to analyze the presentation material and audio data entered by the user and provide detailed and specific feedback.

[1942] "User" refers to a person who logs into the system and uploads presentation material data and audio data.

[1943] "Presentation material data" refers to document data that describes the content of the presentation, and is provided in PDF or text file format.

[1944] "Audio data" refers to audio data recorded from a user's presentation, and is provided in a format such as a WAV file.

[1945] "Upload" refers to the operation of a user sending data from their own terminal to the system's server.

[1946] "Web interface" refers to a function that provides a screen for users to access the system via the Internet and operate or transmit data.

[1947] "Speech Recognition API" refers to the application programming interface of a cloud service used to convert voice data into text data.

[1948] "Text Extraction Library" means a software library for extracting text data from PDF and other document files.

[1949] "Sentiment Analysis API" refers to an application programming interface for an external service used to analyze emotion data from audio data.

[1950] "Natural language processing technology" refers to a general term for algorithms and technologies that enable computers to understand and generate human language.

[1951] "Feedback" refers to the evaluation results that include specific advice and areas for improvement regarding the user's presentation.

[1952] The present invention is a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation. Hereinafter, an embodiment of the present invention will be described in detail.

[1953] Overall system overview

[1954] The system consists of the following main components:

[1955] 1. User Input Components

[1956] 2. Audio data processing component

[1957] 3. Analysis components of presentation materials

[1958] 4. Speaking Assessment Component

[1959] 5. Feedback Generation Component

[1960] 6. Feedback Provision Component

[1961] Hardware and software used

[1962] The system uses the following hardware and software:

[1963] Cloud service speech recognition API (e.g. Google Cloud Speech-to-Text API)

[1964] External sentiment analysis API (e.g. Empath)

[1965] A text extraction library (e.g., Apache PDFBox)

[1966] Natural language processing technology (e.g., GPT-3)

[1967] System Operation

[1968] User Input

[1969] Users log in to the system using a web interface and upload presentation data (PDF or text file format) and audio data (WAV files, etc.). By clicking the upload button, the data is sent to the server.

[1970] Audio data processing

[1971] After receiving the uploaded voice data, the server converts the voice data into text data using the cloud service's voice recognition API, which is then stored on the server. Furthermore, the server uses an external emotion analysis API to obtain volume data and emotion data from the voice data.

[1972] Analysis of presentation materials

[1973] The server analyzes the presentation data uploaded by the user. In the case of PDF files, it uses a text extraction library to extract the text from the document. It then compares this text data with the text data converted from the audio data and evaluates the degree of content agreement.

[1974] Speaking evaluation

[1975] The server evaluates the appropriateness of speaking based on volume and emotion data. It analyzes average volume, volume fluctuations, and emotion labels to determine how effective a user's speech is.

[1976] Generate feedback

[1977] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology to provide the user with specific advice and suggestions for improvement.

[1978] Providing feedback

[1979] The server sends the generated feedback to the user's device, and the user's device displays the received feedback on a web interface, allowing the user to understand specific areas for improvement.

[1980] Specific examples

[1981] An example of a specific operation of the system is shown below.

[1982] User Input

[1983] User "A" uploads the presentation materials ("materials.pdf") and the presentation audio file ("presentation.wav") through the web interface. These data are accepted and stored on the server.

[1984] Audio data processing

[1985] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[1986] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[1987] Additionally, we receive the following data through Empath's sentiment analysis service:

[1988] Volume: 60 decibels on average

[1989] Emotions: High motivation, medium tension, low joy

[1990] Analysis of presentation materials

[1991] Extract the following text data from "Documents.pdf":

[1992] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[1993] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[1994] Speaking evaluation

[1995] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[1996] Generate feedback

[1997] The server generates feedback like this:

[1998] Overall feedback:

[1999] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[2000] Increasing the volume will make it easier for the audience to hear.

[2001] Expressing your emotions will improve the overall impression of your presentation.

[2002] Providing feedback

[2003] The generated feedback is sent to the user's terminal and displayed on a web interface.

[2004] Prompt Sentence Examples

[2005] Here are some example prompts for using generative AI models:

[2006] Analyze user-uploaded audio files (WAV format) and generate feedback assessing:

[2007] How well does the content of the talk match the presentation materials?

[2008] Volume and its fluctuations

[2009] Emotional expression (motivation, tension, joy, etc.)

[2010] Here are the analysis results:

[2011] Voice text: "Hello, my name is A. Today I'll be talking about the latest developments in AI technology."

[2012] Source text: "Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects."

[2013] Volume: 60 decibels on average

[2014] Emotions: High motivation, medium tension, low joy

[2015] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2016] Step 1: User uploads presentation data and audio data

[2017] Users log in to the system through a web interface. After logging in, they select the presentation data (e.g., a PDF file called "Materials.pdf") and audio data (e.g., a WAV file called "Presentation.wav") and click the upload button. The input is sent from the user's device and received by the server. The output is that the uploaded files are saved on the server. Specifically, the user selects "Materials.pdf" and "Presentation.wav" on the file selection screen and clicks the "Upload" button.

[2018] Step 2: Convert audio data to text

[2019] The server sends the received audio data ("Announcement.wav") to the cloud service's speech recognition API, which converts the audio data into text data. The input is the "Announcement.wav" file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology."). Specifically, the server sends "Announcement.wav" to the Google Cloud Speech-to-Text API, receives the converted text data, and saves it.

[2020] Step 3: Extract emotion and volume data

[2021] The server sends the converted voice-to-text data to an external emotion analysis API and extracts volume and emotion data. The input is voice data ("Announcement.wav"), and the output is volume data (e.g., average 60 decibels) and emotion data (e.g., high enthusiasm, medium tension, low joy). Specifically, the server sends the data to the Empath API, obtains the analysis results, and saves them.

[2022] Step 4: Analyze the presentation

[2023] The server analyzes the uploaded presentation data ("materials.pdf"). Specifically, it uses a text extraction library to extract the text within the document. The input is a PDF file, and the output is text data (e.g., "Hello, my name is A. Today I will talk about the latest trends in AI technology. Specifically, I will explain the history of AI, current research directions, and future prospects."). Specifically, the server extracts and saves the text data using Apache PDFBox.

[2024] Step 5: Compare with the converted text

[2025] The server compares the text data converted from the speech with the text data in the presentation materials and calculates the degree of similarity. The input is the text data converted from the speech and the text data extracted from the presentation materials, and the output is the degree of similarity (e.g., 75%). Specifically, the server uses an algorithm to compare both sets of text data and evaluate the degree of similarity.

[2026] Step 6: Speech assessment

[2027] The server evaluates the appropriateness of speaking style based on volume data and emotional data. The input is volume data and emotional data, and the output is the evaluation result (e.g., "low volume" or "modest emotional expression"). Specifically, the server analyzes this data and evaluates problems and areas for improvement in speaking style.

[2028] Step 7: Generate feedback

[2029] The server generates comprehensive feedback based on the results of each analysis. The inputs are the degree of match, volume data, and emotion data, and the output is feedback text (e.g., "The content matches the first chapter of the materials. Please also talk about the contents of chapters two and three. By increasing the volume, you will be able to communicate more easily with the audience. By using more emotional expression, the overall impression of the presentation will be improved."). Specifically, the server generates the feedback text using natural language processing technology.

[2030] Step 8: Provide feedback

[2031] The server sends the generated feedback to the user's terminal. The input is the feedback text, and the output is the display result on the user's terminal. In concrete terms, the server sends the generated feedback to the user's terminal, and the user's terminal displays it on the web interface.

[2032] (Application example 1)

[2033] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2034] Conventional presentation materials and voice analysis systems are limited to providing feedback on individual presentations and speeches, and have the problem of being unable to provide specific suggestions for improving the quality of safety briefings, especially those held within factories. Furthermore, these types of systems only allow users to analyze a portion of a presentation, making it difficult to provide comprehensive feedback that leads to improved safety awareness. The present invention aims to provide a specific system for improving the content and quality of safety briefings within factories and improving safety awareness.

[2035] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2036] In this invention, the server includes a means for users to upload presentation material data and audio data, a means for converting the audio data into text data, and a means for extracting volume data and emotion data from the audio data, which makes it possible to evaluate the degree of agreement between the presentation content and related materials, evaluate the appropriateness of the speaking style, and provide points for improving the quality of the safety briefing.

[2037] "Means for users to upload presentation material data and audio data" refers to means for users to send presentation material data and audio data to the system through a specified interface.

[2038] The "means for converting voice data into text data" refers to a means for analyzing uploaded voice data and converting it into text information.

[2039] The "means for extracting volume data and emotional data from audio data" refers to a means for extracting the volume level and emotional information from audio data using audio analysis technology.

[2040] The "means for comparing presentation material data with converted text data" is a means for analyzing the content of the presentation material and the text converted from the speech and evaluating the degree of agreement.

[2041] The "means for evaluating speaking style based on volume data and emotion data" is a means for evaluating a speaker's speaking style and presentation quality based on acquired volume data and emotion data.

[2042] The "means for generating feedback on the content and speaking style of the presentation" refers to a means for generating specific advice and points for improvement on the content and speaking style of the presentation based on the analysis results.

[2043] The "means for providing feedback to the user's terminal" refers to a means for transmitting the generated feedback to the user's device and displaying it.

[2044] The "means for collecting and analyzing audio data of safety briefings in factories and related document data" refers to a means for collecting and analyzing audio data of safety briefings held in factories and related document data.

[2045] "Means for evaluating the quality of safety briefings based on the analysis results and providing specific suggestions for improvement" refers to means for evaluating the quality of briefings based on collected and analyzed data and providing specific suggestions for improvement.

[2046] The present invention is a system that analyzes presentation material data and audio data uploaded by users and provides specific feedback on both the content and speaking style of the presentation. Furthermore, the system has the function of evaluating the quality of safety briefings in factories and providing specific points for improvement. The following describes in detail the embodiments of the present invention.

[2047] The system consists of the following main components:

[2048] 1. User Input Components

[2049] 2. Audio data processing component

[2050] 3. Analysis components of presentation materials

[2051] 4. Speaking Assessment Component

[2052] 5. Feedback Generation Component

[2053] 6. Feedback Provision Component

[2054] 7. Safety Briefing Analysis Component

[2055] System Operation

[2056] User Input

[2057] Users first log in to the system and upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[2058] Audio data processing

[2059] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text). This saves the content of the voice data as text data. The server then uses an external emotion analysis service (e.g., Empath) to extract volume data and emotion data from the voice data.

[2060] Analysis of presentation materials

[2061] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document. The server then compares the converted audio text with the presentation text data to evaluate the degree of logical correspondence.

[2062] Speaking evaluation

[2063] The server evaluates the appropriateness of the speaking style based on volume data and emotional data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[2064] Generate feedback

[2065] The server generates comprehensive feedback based on the results of each of the above analyses, using a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide the user with specific advice and suggestions for improvement.

[2066] Providing feedback

[2067] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[2068] Safety Briefing Analysis

[2069] The server analyzes the audio data of safety briefings held in factories and related document data. The audio data is converted into text data using the procedure described above, and the related document data is also analyzed for text. The server integrates this data, evaluates the quality of the briefings, and generates specific improvements.

[2070] Specific examples

[2071] For example, if a factory manager says, "Please be sure to wear a helmet for safety reasons," but the documentation does not include that instruction, the system will point out the discrepancy between the documentation and the audio content.

[2072] Prompt Sentence Examples

[2073] Prompts for the development of an automated safety briefing analysis system:

[2074] Analyze the audio data and PDF documents and evaluate the degree of match.

[2075] Extract emotional data from audio data and evaluate the appropriateness of speaking styles.

[2076] Please provide your overall feedback based on these analysis results.

[2077] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2078] Step 1:

[2079] User Input

[2080] Users upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. The input data is sent to the server, which then stores it.

[2081] Input: Presentation data and audio data

[2082] Output: Presentation data and audio data stored on the server

[2083] Specific operation: The user selects a file through a web browser and clicks the send button, which uploads the file to the server.

[2084] Step 2:

[2085] Converting audio data to text

[2086] The server receives the uploaded voice data and converts it into text using a speech recognition engine (e.g., Google Cloud Speech-to-Text). Through this process, the voice content is saved as text.

[2087] Input: Audio data

[2088] Output: Text data converted from audio data

[2089] Specific operation: The server sends the voice data to the speech recognition engine and stores the returned text data in the database.

[2090] Step 3:

[2091] Extracting volume and emotion data

[2092] The server sends the converted text data to an external emotion analysis service (e.g., Empath) to extract volume and emotion data, which then stores the speaker's emotional state and speaking style characteristics.

[2093] Input: Text data

[2094] Output: Volume data and emotion data

[2095] Specific operation: The server sends text data to the emotion analysis service, obtains the returned emotion data, and stores it.

[2096] Step 4:

[2097] Analysis of presentation data

[2098] The server analyzes the presentation data uploaded by the user. For example, in the case of a PDF file, it uses a text extraction library (e.g., PyMuPDF) to extract the text within the document.

[2099] Input: Presentation data (PDF or text file)

[2100] Output: Text data of presentation materials

[2101] Specific operation: The server uses a text extraction library to extract content from PDF and text files and saves it in a database.

[2102] Step 5:

[2103] Comparison of presentation materials and audio text

[2104] The server compares the converted speech text with the presentation text, evaluates the degree of match, and stores the result as a numerical value indicating how well the presentation matches the presentation text.

[2105] Input: Converted speech text, presentation material text

[2106] Output: Match data

[2107] Specific operation: The server uses a text comparison algorithm to calculate the degree of match between the two data and saves the results.

[2108] Step 6:

[2109] Speaking evaluation

[2110] The server evaluates the appropriateness of the speaking style based on the volume data and emotion data, thereby determining how effective the speaker's speaking style is.

[2111] Input: volume data and emotion data

[2112] Output: Speaking evaluation data

[2113] Specific operation: The server analyzes the volume data and emotion data, evaluates the effectiveness of the speaking style, and saves the results.

[2114] Step 7:

[2115] Generate feedback

[2116] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., a generative AI model such as GPT-3) to provide specific advice and suggestions for improvement to the user.

[2117] Input: Matching data, speaking style evaluation data

[2118] Output: Feedback data

[2119] Specific operation: The server uses a text generation algorithm to generate feedback based on a set of data and stores it in a database.

[2120] Step 8:

[2121] Providing feedback

[2122] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[2123] Input: Feedback data

[2124] Output: Feedback displayed on the user's terminal

[2125] Specific operation: The server sends feedback data to the user's terminal, and the terminal displays it.

[2126] Step 9:

[2127] Collection and analysis of audio and document data from safety briefings

[2128] The server collects and analyzes the audio data of safety briefings held in factories and related materials. The collected data is converted into text and analyzed using the procedure described above.

[2129] Input: Audio data of safety briefing and related materials data

[2130] Output: Text data and analysis data

[2131] What it does: The server collects the safety briefing data and analyzes it using a speech recognition engine and text extraction library.

[2132] Step 10:

[2133] Evaluating the quality of safety briefings and providing suggestions for improvement

[2134] Based on the collected and analyzed data, the server evaluates the quality of the safety briefing and generates specific improvements.

[2135] Input: Analysis data

[2136] Output: Evaluation data and improvement proposal data

[2137] How it works: The server uses the analysis results to evaluate the quality of the safety briefing and uses a generative AI model to generate feedback suggesting improvements.

[2138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2139] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[2140] Overall system overview

[2141] The system consists of the following main components:

[2142] 1. User Input Components

[2143] 2. Audio data processing component

[2144] 3. Analysis components of presentation materials

[2145] 4. Speaking Assessment Component

[2146] 5. Feedback Generation Component

[2147] 6. Feedback Provision Component

[2148] 7. Emotion Engine Component

[2149] System Operation

[2150] User Input

[2151] Users first log in to the system and then upload their presentation data (PDF or text file format) and audio data (WAV files, etc.) through a web interface. User input is accepted through a form, and the information is sent to the server by clicking the upload button.

[2152] Audio data processing

[2153] The server receives the uploaded voice data. It converts the voice data into text data using a voice recognition engine (e.g., a voice recognition API of a cloud service). This allows the content of the voice data to be saved as text data. The server then uses an external emotion analysis service (e.g., a third-party emotion analysis API) to extract volume data and emotion data from the voice data.

[2154] Analysis of presentation materials

[2155] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of logical agreement.

[2156] Speaking evaluation

[2157] The server evaluates the appropriateness of the speaking style based on volume and emotion data extracted from the voice data. Specifically, it analyzes the average volume, volume fluctuations, and emotion labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[2158] Generate feedback

[2159] The server generates comprehensive feedback based on the results of each of the above analyses, utilizing a text generation algorithm (e.g., natural language processing technology) to provide the user with specific advice and suggestions for improvement.

[2160] Utilizing the Emotion Engine

[2161] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotions in real time from facial expressions and movements. The emotion data recognized in real time is combined with extracted data from the voice data and becomes an important factor in comprehensively evaluating speaking style.

[2162] Dynamic feedback adjustment

[2163] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, if the user is nervous, it might include advice like "Try to speak in a relaxed manner."

[2164] Providing feedback

[2165] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[2166] Specific examples

[2167] An example of a specific operation of the system is shown below.

[2168] User Input

[2169] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[2170] Audio data processing

[2171] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[2172] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[2173] Additionally, through external sentiment analysis services, we receive the following data:

[2174] Volume: 60 decibels on average

[2175] Emotions: High motivation, medium tension, low joy

[2176] Analysis of presentation materials

[2177] Extract the following text data from "Documents.pdf":

[2178] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[2179] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[2180] Speaking evaluation

[2181] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[2182] Utilizing the Emotion Engine

[2183] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[2184] Dynamic feedback adjustment

[2185] The server dynamically adjusts the feedback as follows:

[2186] Overall feedback:

[2187] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[2188] Increasing the volume will make it easier for the audience to hear.

[2189] Expressing your emotions will improve the overall impression of your presentation.

[2190] You seem nervous, so take a deep breath and relax.

[2191] Providing feedback

[2192] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[2193] In this way, the system of the present invention provides detailed and specific feedback to the user, and by adding real-time emotion recognition, helps improve presentations for greater effectiveness.

[2194] The processing flow will be explained below.

[2195] Step 1: The user uploads the presentation data and audio data to the system.

[2196] The user clicks the file selection button on the interface and selects the presentation data (PDF file) and audio data (WAV file).

[2197] The user clicks the "Upload" button and the selected file is sent to the server.

[2198] Step 2: The server receives the uploaded file.

[2199] The server receives the HTTP request and temporarily stores the uploaded file in local or cloud storage.

[2200] Step 3: The server sends the voice data to the voice recognition engine.

[2201] The server sends the audio file (WAV file) to the speech recognition API.

[2202] The server receives the converted text data from the speech recognition API.

[2203] Step 4: The server extracts volume data and emotion data from the audio data.

[2204] The server sends the audio data to an external emotion analysis service and receives volume data and emotion data.

[2205] Step 5: The server uses the emotion engine to recognize the user's emotions in real time.

[2206] The user's camera footage is sent to a server, which then uses an emotion engine to analyze emotions in real time.

[2207] The server stores the analysis results.

[2208] Step 6: The server converts the presentation data into text data.

[2209] The server converts the PDF file into text data using a text analysis library.

[2210] Step 7: The server compares the audio text with the text data of the presentation materials.

[2211] The server uses a text comparison algorithm to calculate the degree of match between the spoken text and the source text.

[2212] Step 8: The server evaluates the speaking style based on the volume data and emotion data.

[2213] The server analyzes the extracted volume data and evaluates the average volume and volume fluctuations.

[2214] The server analyzes the emotional data and evaluates the appropriateness of the speaker's emotional expression.

[2215] Step 9: The server evaluates the speaking style comprehensively based on the recognized emotion data in real time.

[2216] The server integrates the analysis results of real-time emotional data and voice data to make a comprehensive evaluation of speaking style.

[2217] Step 10: The server generates comprehensive feedback.

[2218] Based on each analysis result, the server generates specific feedback on the content and delivery of the presentation.

[2219] Using a text generation algorithm, it provides users with specific advice and areas for improvement.

[2220] Step 11: The server sends the generated feedback to the user's terminal.

[2221] The server sends an HTTP response including the generated feedback, which is received by the user's device.

[2222] Step 12: The user's device displays the feedback.

[2223] The user's terminal analyzes the received feedback data and displays it on the interface.

[2224] Specific examples

[2225] User Input

[2226] User "A" uploads the presentation materials ("Materials.pdf") and the presentation audio file ("Presentation.wav"). The uploaded files are accepted and saved on the server.

[2227] Audio data processing

[2228] The server sends "Announcement.wav" to the speech recognition engine and receives the following text data:

[2229] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[2230] Additionally, through external sentiment analysis services, we receive the following data:

[2231] Volume: 60 decibels on average

[2232] Emotions: High motivation, medium tension, low joy

[2233] Analysis of presentation materials

[2234] Extract the following text data from "Documents.pdf":

[2235] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[2236] The server compares the audio text with the source text and calculates the degree of match (for example, 75% match).

[2237] Speaking evaluation

[2238] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[2239] Utilizing the Emotion Engine

[2240] The emotion engine analyzes the user's real-time emotions and recognizes, for example, that "the user is nervous."

[2241] Dynamic feedback adjustment

[2242] The server dynamically adjusts the feedback as follows:

[2243] Overall feedback:

[2244] The content is consistent with Chapter 1 of the handout. Please also discuss the contents of Chapters 2 and 3.

[2245] Increasing the volume will make it easier for the audience to hear.

[2246] Expressing your emotions will improve the overall impression of your presentation.

[2247] You seem nervous, so take a deep breath and relax.

[2248] Providing feedback

[2249] The generated feedback is sent to the user's terminal, which displays the feedback on a web page.

[2250] Example 2

[2251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2252] In today's world, there is a demand for systems that allow users to efficiently evaluate and improve their presentation content and speaking style, but current systems have difficulty providing comprehensive feedback that combines analysis of audio data, comparison of document data, and real-time emotion recognition. Furthermore, because they are unable to generate dynamic feedback that takes into account the user's real-time emotions, they are unable to provide specific and useful suggestions for improvement.

[2253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2254] In this invention, the server includes means for a user to upload presentation material data and audio data, means for converting the audio data into text data, means for extracting volume data and emotion data from the audio data, means for comparing the presentation material data with the converted text data, means for evaluating speaking style based on the volume data and emotion data, means for generating feedback on the content of the presentation and speaking style, means for analyzing real-time emotion data acquired from the user, means for dynamically adjusting the content of the feedback based on the real-time emotion data, and means for providing feedback to the user's terminal. This enables comprehensive analysis of the audio data and material data and provision of specific feedback that takes real-time emotion into consideration.

[2255] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[2256] "Presentation material data" refers to digital data in PDF or text file format used as presentation or explanatory materials.

[2257] "Audio Data" means digital data in the form of a WAV file or other audio file that contains a recording of content presented by a user.

[2258] A "voice recognition engine" is software or a service for converting voice data into text data.

[2259] "Text data" is data in the form of a character string extracted from voice data by a voice recognition engine.

[2260] "Emotion analysis service" is an external service that analyzes and extracts volume data and user emotion data from audio and video data.

[2261] "Volume data" is data that indicates the volume of a sound extracted from audio data.

[2262] "Emotion data" is data that indicates the user's emotional state (for example, enthusiasm, tension, joy, etc.).

[2263] "Material text" is text data in character string format extracted from presentation material data.

[2264] The "degree of agreement" is an evaluation index that indicates the degree to which the contents of the audio text and the material text match.

[2265] "Speaking style evaluation" is the process of evaluating the appropriateness and effectiveness of a user's speaking style based on volume data and emotion data.

[2266] "Feedback" is information that includes specific advice and areas for improvement regarding the content and delivery of the presentation.

[2267] "Real-time emotional data" is data that indicates the user's current emotional state, obtained by analyzing video captured by the user's camera.

[2268] "Dynamic feedback adjustment" is the process of appropriately changing the content of feedback based on emotional data obtained in real time.

[2269] A "terminal" is a device such as a computer or smartphone that a user uses to access the system.

[2270] The present invention combines a system that analyzes presentation material data and audio data uploaded by a user and provides specific feedback on both the content and speaking style of the presentation with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[2271] Hardware and Software Configuration

[2272] User Input Components

[2273] Users first log in to the system and upload their presentation data (e.g., PDF or text file format) and audio data (e.g., WAV files) to the server. Users input data using a web interface, register files through a form, and click the upload button to send the information to the server.

[2274] Audio data processing component

[2275] The server receives the uploaded voice data, converts the voice data into text data using a speech recognition API of a cloud service (e.g., Google Cloud Speech-to-Text), and extracts volume data and emotion data from the voice data using an external emotion analysis service (e.g., IBM Watson Tone Analyzer).

[2276] Analysis components of presentation materials

[2277] The server analyzes the presentation data uploaded by the user as text data. For example, in the case of a PDF file, it uses a text extraction library (e.g., Apache PDFBox) to extract the text within the document. The server then compares the converted speech text with the presentation text data to evaluate the degree of match.

[2278] Speaking evaluation component

[2279] The server evaluates the appropriateness of the speaking style based on volume data and emotional data. Specifically, it analyzes the average volume, volume fluctuations, and emotional labels (e.g., enthusiasm, tension, joy, etc.) to determine how effective the user's presentation is.

[2280] Feedback Generation Component

[2281] The server generates comprehensive feedback based on each analysis result, utilizing natural language processing technology (e.g., a generative AI model) to provide the user with specific advice and suggestions for improvement.

[2282] Utilizing the Emotion Engine Component

[2283] The server also utilizes an emotion engine that recognizes the user's emotions in real time. It analyzes the video captured by the user's camera and recognizes emotional data in real time from facial expressions and movements. This real-time emotional data is also an important factor in comprehensively evaluating the speaking style.

[2284] Dynamic feedback adjustment

[2285] Based on real-time emotion recognition, the server dynamically adjusts the feedback it provides, for example, by including advice like "Try to speak in a relaxed manner" if the user is nervous.

[2286] Provide Feedback Component

[2287] The server finally sends the generated feedback to the user's terminal, which displays the received feedback and provides specific guidelines for the user to improve.

[2288] Specific examples

[2289] An example of a specific operation of the system is shown below.

[2290] User Input

[2291] User "A" uploads the presentation materials "Materials.pdf" and the presentation audio file "Presentation.wav". The uploaded files are accepted and saved on the server.

[2292] Audio data processing

[2293] The server sends "Announcement.wav" to the Google Cloud Speech-to-Text API and receives the following text data:

[2294] Hello, my name is A. Today I will be talking about the latest trends in AI technology.

[2295] Additionally, through IBM Watson Tone Analyzer, you receive the following data:

[2296] Volume: 60 decibels on average

[2297] Emotions: High motivation, medium tension, low joy

[2298] Analysis of presentation materials

[2299] Extract the following text data from "document.pdf" using Apache PDFBox:

[2300] Hello, my name is A. Today I'd like to talk about the latest trends in AI technology. Specifically, I'll explain the history of AI, current research directions, and future prospects.

[2301] The server compares the audio text with the source text and calculates a match of 75%.

[2302] Speaking evaluation

[2303] The server analyzes the volume data and emotional data and determines that the volume is low and the emotional expression is subdued.

[2304] Utilizing the Emotion Engine

[2305] The emotion engine recognizes that the user is nervous.

[2306] Dynamic feedback adjustment

[2307] The server dynamically adjusts the feedback as follows:

[2308] Overall feedback:

[2309] The content is consistent with Chapter 1 ...

Claims

1. A means for users to upload presentation material data and audio data; means for converting voice data into text data; means for extracting volume data and emotion data from the audio data; A means for comparing the presentation data with the converted text data; a means for evaluating speaking style based on volume data and emotion data; a means of generating feedback on the content and delivery of the presentation; means for providing feedback to a user's terminal; A system including:

2. 2. The system of claim 1, wherein the means for extracting the volume data and emotion data utilizes an external emotion analysis service.

3. 10. The system of claim 1, wherein the means for generating feedback uses a text generation algorithm.

Images