System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image file of a presentation material; means for receiving presentation voice data; means for analyzing each of the image file and the voice data; means for generating feedback based on a result of the analysis; and means for displaying the feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, improving effective presentation skills is crucial, yet many people have few opportunities to receive appropriate feedback. As a result, the quality of their presentations declines, often leading to lost business opportunities. Furthermore, different approaches are required depending on the reviewer and presentation content, so feedback that adapts to each individual is necessary. There is a need for a method to efficiently improve presentation skills that addresses these individual needs. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system including a means for receiving image files of presentation materials, a means for receiving presentation audio data, a means for analyzing the image files and the audio data, a means for generating feedback based on the analysis results, and a means for displaying the feedback. Furthermore, by providing a means for weighting the analyzed image files and audio data based on specific review recipients and presentation content, and a means for optimizing the feedback through individual learning, it is possible to provide appropriate feedback tailored to each individual's needs. Furthermore, the analysis means includes a means for converting image files to text data, a means for converting audio data to text, and a means for analyzing the converted text data using natural language processing technology, making it possible to effectively identify redundancies, unclear expressions, and logical inconsistencies in the presentation content and suggest areas for improvement.

[0006] "Presentation materials" refers to image files and slides containing the content of the presentation.

[0007] "Presentation audio data" refers to an audio file that records the speaker's voice during a presentation.

[0008] "Analysis means" refers to means for converting image files or audio data into text data or analyzing the content using natural language processing technology.

[0009] "Feedback" refers to information generated based on the analysis results, including suggestions for improvement and comments on the presentation.

[0010] "Weighting" refers to a method of adjusting the importance and priority of analysis results depending on the review recipient and presentation content.

[0011] "Personalized learning" refers to learning algorithms and methods that optimize feedback for specific reviewers or users.

[0012] "Natural language processing" refers to the technology of performing syntactic and semantic analysis of text data to understand and analyze its content.

[0013] "Image analysis module" refers to software or algorithms for extracting text and graphic information from image files and converting it into text data.

[0014] "Speech analysis module" refers to software and algorithms used to convert voice data into text and analyze voice characteristics. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that analyzes image files and audio data of presentation materials and provides feedback. The program processing of this system is explained in detail below in natural language.

[0037] System Overview

[0038] Users upload image files to be used in their presentations and audio data recording the presentation to the system. The server receives these files and performs image and audio analysis. The analyzed text data is further analyzed using natural language processing technology to generate specific feedback based on the presentation's content and speaking style. The feedback is weighted based on the specific reviewer and presentation content and provided to the user.

[0039] Program processing

[0040] Receiving and storing data

[0041] 1. After the user logs in, they upload the image files of the presentation materials and the presentation audio data through a dedicated interface.

[0042] 2. The terminal reads the file specified by the user and sends it to the server.

[0043] 3. The server receives these files and stores them in a secure temporary storage directory.

[0044] Image analysis

[0045] 1. The server loads the image file from the temporary storage directory.

[0046] 2. The server starts the image analysis module and performs processing to convert the image file into text data.

[0047] 3. The image analysis module extracts the text in the image file and returns the data to the server.

[0048] 4. The server formats the retrieved text data and prepares it for passing to the natural language processing module.

[0049] Audio analysis

[0050] 1. The server loads the audio file from the temporary storage directory.

[0051] 2. The server launches the speech analysis module and processes the speech data to convert it into text.

[0052] 3. The speech analysis module converts the speech into text data and returns the data to the server.

[0053] 4. The server formats the acquired speech-to-text data and prepares it for passing to the natural language processing module.

[0054] Natural Language Processing and Feedback Generation

[0055] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[0056] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[0057] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0058] Weighting and Individual Optimization

[0059] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[0060] 2. The server applies a weighting algorithm based on the received information to adjust the analysis results.

[0061] 3. The server uses a personalized learning module to look at past feedback data and generate feedback optimized for the specific reviewer.

[0062] Specific examples

[0063] Example 1: Proposal presentation

[0064] The user uploads slide images and audio data for new product proposals.

[0065] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[0066] The server generates specific feedback based on the analysis, such as identifying redundancies in the proposal or adding specific use cases to attract the client's interest.

[0067] Improve your presentation materials and delivery based on the feedback provided by users.

[0068] Example 2: Apology presentation

[0069] A user uploads slide images and audio data apologizing for the project delay.

[0070] The server performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[0071] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[0072] Improve your apology presentation based on user-provided feedback.

[0073] This concludes the description of the "Mode for Carrying Out the Invention" of the present invention. This system provides specific and detailed feedback to help users give effective presentations.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] Users upload image files of presentation materials and presentation audio data through a dedicated interface.

[0077] Step 2:

[0078] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[0079] Step 3:

[0080] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[0081] Step 4:

[0082] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0083] Step 5:

[0084] The image analysis module extracts the text within the image file and returns the text data to the server.

[0085] Step 6:

[0086] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[0087] Step 7:

[0088] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0089] Step 8:

[0090] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[0091] Step 9:

[0092] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0093] Step 10:

[0094] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0095] Step 11:

[0096] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0097] Step 12:

[0098] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[0099] Step 13:

[0100] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0101] Step 14:

[0102] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0103] Step 15:

[0104] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[0105] Step 16:

[0106] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[0107] Step 17:

[0108] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[0109] Step 18:

[0110] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] Conventional presentation feedback systems mainly analyze image files and audio data individually, which has the drawback of insufficient comprehensive analysis and specific feedback to users. Furthermore, since it is not possible to weight or optimize based on specific reviewers or presentation content, it is difficult to obtain the specific and accurate feedback that users need.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for analyzing the analyzed text data using natural language processing technology, means for generating feedback based on the analysis results, means for displaying the feedback, means for the user to input specific review recipients and presentation content, means for weighting the feedback based on the input information, and means for optimizing the feedback using past feedback data. This makes it possible to comprehensively analyze the presentation materials and audio data and provide specific and accurate feedback based on the specific review recipients and presentation content.

[0116] "Presentation materials" are a collection of images and slides that visually represent the contents of a presentation.

[0117] "Image files" refer to still image data that has been digitized from presentation materials.

[0118] "Presentation audio data" refers to audio data recorded from a presentation.

[0119] "Server" refers to the central computer system used for analysis, data processing, and feedback generation.

[0120] A "terminal" refers to a computer or device operated by a user, and is responsible for exchanging data with a server.

[0121] "User" refers to a person who uses the system to upload presentation data and receive feedback.

[0122] "Analysis means" refers to the function of converting image files and audio data into text data and analyzing it using natural language processing technology.

[0123] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0124] "Feedback" refers to information, including suggestions for improvement and recommendations, provided to users based on analyzed data.

[0125] "Weighting" refers to the process of adjusting the analysis results based on the specific review destination or presentation content.

[0126] "Optimization" refers to using past feedback data to make adjustments to improve the accuracy and usefulness of feedback.

[0127] The present invention relates to a system for analyzing image files and audio data of presentation materials and providing feedback to users. An embodiment of this system will be described in detail below.

[0128] System Overview

[0129] The system of the present invention allows users to upload image files used as presentation materials and audio data for the presentation, and then analyzes the materials and audio and provides feedback. Specifically, the server receives and analyzes these files, and uses natural language processing technology to generate feedback appropriate for the user.

[0130] Hardware and software used

[0131] Server: A central computer system that analyzes and processes data and generates feedback. The server is equipped with image analysis modules, audio analysis modules, and natural language processing modules.

[0132] Device: A computer or device operated by a user to upload files or display feedback.

[0133] Image Analysis Module: Analyzes image files using image analysis libraries such as Python, OpenCV, and Tesseract OCR.

[0134] Speech analysis module: Analyzes speech data using speech analysis APIs such as Google Speech-to-Text and IBM Watson Speech to Text.

[0135] Natural Language Processing module: Analyzes and processes text data using the NLTK library and generative AI models.

[0136] Data processing and calculation

[0137] When a user uploads image files and audio data for a presentation, the device sends these files to the server. The server receives these files and stores them in a secure temporary directory. The server then launches image and audio analysis modules to convert the image files into text data and the audio data into text.

[0138] The server passes the acquired text data to a natural language processing module to detect redundancies, unclear expressions, and logical inconsistencies. The server generates feedback based on the analysis results and provides it to the user.

[0139] Additionally, when a user inputs a specific review destination or presentation content (e.g., proposal, apology), the server applies a weighting algorithm and references past feedback data to optimize the feedback.

[0140] Specific examples

[0141] Proposal presentation example

[0142] User: Uploads slide images and audio data for a new product proposal presentation to the system.

[0143] Server: Performs image and audio analysis to analyze the content of slides and speaking style.

[0144] Server: Generates specific feedback based on the analysis, such as redundancies in the proposal or adding specific use cases to capture the client's interest.

[0145] Users: Improve their presentation materials and delivery based on the feedback provided.

[0146] Example of an apology presentation

[0147] User: Upload slide images and audio data to the system apologizing for the project delay.

[0148] Server: Performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[0149] Server: Based on the analysis results, provide feedback on specific improvement measures and how to choose the right words to use in your apology.

[0150] User: Improve your apology presentation based on the feedback provided.

[0151] This is the embodiment of the present invention. This system allows users to receive specific and detailed feedback on how to give effective presentations.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1:

[0154] A user logs in.

[0155] Input: User ID and password.

[0156] How it works: A user accesses the system's login page and enters their user ID and password.

[0157] Output: If authentication is successful, the user can access the main interface of the system.

[0158] Step 2:

[0159] The user uploads image files of the presentation materials and audio data of the presentation.

[0160] Input: Image files and audio data files.

[0161] How it works: The user selects image and audio files in a dedicated interface and clicks the upload button.

[0162] Output: The selected file is prepared by the terminal.

[0163] Step 3:

[0164] The terminal reads the file specified by the user and sends it to the server.

[0165] Input: Image files and audio data files.

[0166] How it works: The device gets the path of the selected file and sends it to the server using an HTTP POST request.

[0167] Output: The file data sent to the server.

[0168] Step 4:

[0169] The server receives the file and stores it in a secure temporary storage directory.

[0170] Input: Submitted image and audio data files.

[0171] How it works: The server saves the received file in a temporary directory, such as the / tmp directory.

[0172] Output: File saved in temporary directory.

[0173] Step 5:

[0174] The server loads the image file from the temporary storage directory.

[0175] Input: The path to the saved image file.

[0176] What it does: The server loads the image file from the temporary storage directory into memory.

[0177] Output: The loaded image data.

[0178] Step 6:

[0179] The server starts the image analysis module and converts the image file into text data.

[0180] Input: The loaded image data.

[0181] How it works: The server uses image analysis libraries such as OpenCV and Tesseract OCR to extract text from images.

[0182] Output: The extracted text data.

[0183] Step 7:

[0184] The server reads the audio file from the temporary storage directory.

[0185] Input: The path to the saved audio file.

[0186] How it works: The server loads the audio file from the temporary storage directory into memory.

[0187] Output: The loaded audio data.

[0188] Step 8:

[0189] The server starts a voice analysis module and converts the voice data into text data.

[0190] Input: The loaded audio data.

[0191] How it works: The server uses a speech analysis API such as Google Speech-to-Text or IBM Watson Speech to Text to convert the audio data into text.

[0192] Output: The converted speech-to-text data.

[0193] Step 9:

[0194] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0195] Input: Extracted image-text data and audio-text data.

[0196] How it works: The server sends text data to the natural language processing module via an API.

[0197] Output: Text data sent to the natural language processing module.

[0198] Step 10:

[0199] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0200] Input: The text data sent.

[0201] How it works: The natural language processing module uses the NLTK library and generative AI models to analyze the quality of text.

[0202] Output: Analysis result data.

[0203] Step 11:

[0204] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[0205] Input: Analysis result data.

[0206] How it works: The server uses templates based on the analysis results to summarize specific improvements and suggestions in natural language.

[0207] Output: The generated feedback sentence.

[0208] Step 12:

[0209] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0210] Input: Specific reviewer and presentation content.

[0211] How it works: The user selects and inputs the review recipient and presentation purpose in the interface.

[0212] Output: The input review destination and presentation content data.

[0213] Step 13:

[0214] Based on the information received, the server applies a weighting algorithm to adjust the analysis results.

[0215] Input: Review destination and presentation content data, analysis result data.

[0216] How it works: The server applies weighting parameters appropriate to the reviewer and presentation content to adjust the feedback.

[0217] Output: Weighted analysis result data.

[0218] Step 14:

[0219] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[0220] Input: Weighted analysis result data, past feedback data.

[0221] How it works: The server queries a database of past feedback and optimizes the feedback based on feedback performance in similar situations.

[0222] Output: Optimized feedback text.

[0223] The above is the specific flow of program processing in this system.

[0224] (Application example 1)

[0225] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0226] Conventional presentation training systems simply require users to upload presentation materials and audio data, which the system then analyzes and provides feedback. This makes it difficult to provide feedback optimized for specific environments (e.g., in a factory) or situations (e.g., safety measures presentations). They also lack the ability to detect redundant sentences or unclear expressions in real time and immediately suggest specific areas for improvement. This leaves users facing specific challenges in conducting effective presentation training.

[0227] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0228] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for displaying the feedback, and means for coordinating with a robot that performs presentation training in a factory. This allows users to receive feedback in real time that is optimized for specific environments and situations in the factory, enabling more effective presentation training.

[0229] An "image file of presentation materials" is a file in which information such as slides and charts used in a presentation is saved in image format.

[0230] "Presentation audio data" refers to data recorded in audio format of what was said during a presentation.

[0231] "Means for analysis" refers to the technology or device that converts received image files or audio data into text information and performs analysis.

[0232] "Means for generating feedback" refers to technology or devices that generate suggestions for improvement or comments to be provided to users based on the content of the presentation and the results of data analysis.

[0233] "Means for displaying feedback" refers to techniques or devices for visually or audibly presenting the generated feedback to the user.

[0234] A "robot for in-factory presentation training" is an autonomous or remotely controlled mechanical device that supports presentation training in a factory and provides feedback.

[0235] A "weighting means" is a technique or device that adjusts the evaluation or results based on the analyzed text data according to specific criteria.

[0236] "Means for optimizing feedback through individual learning" refers to technology or devices that generate feedback optimized for each user based on past analysis results and feedback data.

[0237] "Means for converting voice data to text" refers to technology or devices that use voice recognition technology to convert the spoken content in an audio file into text form.

[0238] "Means for analyzing using natural language processing technology" refers to technology or devices that perform semantic analysis and error checking of text data converted using natural language processing technology.

[0239] "Means for detecting redundant sentences and unclear expressions" refers to technology or devices that identify unimportant long sentences and difficult-to-understand expressions from the analyzed text data and point them out to the user.

[0240] This invention relates to a system that analyzes image files of presentation materials and audio data of presentations and provides feedback. This system is intended to support presentation training, particularly in factories, and to provide users with optimized feedback.

[0241] System configuration and program processing

[0242] Receiving and storing data

[0243] The server receives and temporarily stores image files and audio data for presentation materials uploaded by users via the interface of the robots in the factory. The server stores the received files in a secure temporary storage directory and prepares them for analysis.

[0244] Image analysis

[0245] The server uses PIL (Python Imaging Library) and pytesseract libraries to convert image files to text, extracting the text data within the image files, which is then formatted by the server and prepared for natural language processing.

[0246] Audio analysis

[0247] The server uses the speech_recognition library to convert the audio file to text, which is then formatted and prepared for natural language processing as well.

[0248] Natural Language Processing and Feedback Generation

[0249] The server analyzes the text data obtained from image and audio analysis using natural language processing technology. This process uses the spacy library. During the analysis, redundant sentences and unclear expressions are identified in the text data, and specific feedback on improvements is generated.

[0250] Weighting and Individual Optimization

[0251] The server weights specific review destinations and presentation content based on the analyzed text data, and also provides optimized feedback for each user by referencing past feedback data through personalized learning.

[0252] Specific examples

[0253] Example 1: Proposal presentation

[0254] When a user gives a presentation to propose a new product in a factory, they upload slide images and audio data. The server analyzes these files and generates specific feedback such as advice like, "The text on this slide is redundant. Please explain it more succinctly."

[0255] Example 2: Safety Presentation

[0256] When giving a presentation on safety measures in a factory, the user uploads the materials and audio data. The server analyzes the data and provides feedback such as, "The explanation of safety precautions tends to be redundant. In particular, make the section on 'checking safety devices' more concise."

[0257] Example prompt sentence:

[0258] "The text on this slide is redundant. Please explain it more succinctly."

[0259] "Safety precautions can be redundant. Please be more concise, especially the 'check safety equipment' section."

[0260] This invention allows users to receive real-time feedback optimized for specific environments and situations within a factory, enabling more effective presentation training. This system is an important tool for improving the effectiveness of presentations in specific situations.

[0261] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0262] Step 1:

[0263] A user logs in and uploads image files and audio data for presentation materials. The user uses the robot interface in the factory to select these files and send them to the server. The input is the image files and audio files, and the output is the server receiving the files.

[0264] Step 2:

[0265] After the server receives the uploaded files, it stores them in a secure temporary directory. The input is the image and audio files uploaded by the user, and the output is that these files are stored in the temporary directory.

[0266] Step 3:

[0267] The server reads the saved image file and converts it to text data using image analysis modules (PIL and pytesseract). The input is the image file saved in the temporary storage directory, and the output is the converted text data. This text data is returned to the server.

[0268] Step 4:

[0269] The server reads the saved audio file and converts it into text using a speech analysis module (speech_recognition library). The input is the audio file saved in the temporary storage directory, and the output is the converted text. This text data is also returned to the server.

[0270] Step 5:

[0271] The server passes the text data obtained from image and audio analysis to a natural language processing module (the spacy library) for analysis. The input is the converted text data, and the output is the analysis results. Based on these analysis results, redundant sentences and unclear expressions are detected.

[0272] Step 6:

[0273] The server generates specific feedback based on the analysis results of the natural language processing. The input is the analysis results from the natural language processing module, and the output is the generated feedback. This feedback includes specific advice on redundant sentences and areas for improvement.

[0274] Step 7:

[0275] The server weights the generated feedback based on the specific reviewer and presentation content. The input is feedback and reviewer information, and the output is weighted feedback.

[0276] Step 8:

[0277] The server uses an individual learning module to generate optimized feedback by referring to past feedback data, with the input being the past feedback data and the current feedback, and the output being the optimized feedback.

[0278] Step 9:

[0279] The server visually displays the generated feedback to the user: the input is the optimized feedback, and the output is the user seeing the feedback on the screen.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] overview

[0282] This invention relates to a system that analyzes image files and audio data from presentation materials and provides feedback. It also has the function of generating feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions.

[0283] System configuration

[0284] The system consists of the following main components:

[0285] 1. Data Receiving Method

[0286] 2. Image analysis methods

[0287] 3. Audio analysis methods

[0288] 4. Natural Language Processing Methods

[0289] 5. Feedback Generation Methods

[0290] 6. Weighting Methods

[0291] 7. Personalized learning tools

[0292] 8. Emotion Engine (Voice Emotion Analysis and Image Emotion Analysis)

[0293] 9. Feedback Display Methods

[0294] Program processing

[0295] The program processing of this system will be specifically explained below.

[0296] Receiving and storing data

[0297] 1. The user uploads the image files of the presentation materials and the presentation audio data through a dedicated interface.

[0298] 2. The terminal reads the file specified by the user and sends it to the server.

[0299] 3. The server receives the file and stores it in a secure temporary storage directory.

[0300] Image analysis

[0301] 1. The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0302] 2. The image analysis module extracts the text from the image file and returns the text data to the server.

[0303] 3. The server formats the text data it receives and stores it in a data store for the natural language processing module.

[0304] Audio analysis

[0305] 1. The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0306] 2. The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[0307] 3. The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0308] Natural Language Processing and Feedback Generation

[0309] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[0310] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[0311] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0312] Emotion engine processing

[0313] 1. The server passes the voice data to the voice emotion analysis means and analyzes the user's emotions.

[0314] 2. The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[0315] 3. The server passes the user's facial image contained in the image file to the image emotion analysis means, and analyzes the emotion.

[0316] 4. The image emotion analysis means extracts emotions from the facial image and returns the emotion data to the server.

[0317] 5. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[0318] Weighting and Individual Optimization

[0319] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[0320] 2. Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0321] 3. The server uses a personalized learning module to generate feedback optimized for the specific reviewer, referencing past feedback data.

[0322] Specific examples

[0323] Example 1: Proposal presentation

[0324] The user uploads slide images and audio data for new product proposals.

[0325] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[0326] The server uses an emotion engine to analyze the user's emotions and provides feedback based on that. For example, if the user is nervous, the server adds advice to ease the tension.

[0327] Based on the analysis, the server generates specific feedback, such as redundancy in the proposal or adding use cases to attract the client's interest.

[0328] Improve your presentation materials and delivery based on the feedback provided by users.

[0329] Example 2: Apology presentation

[0330] A user uploads slide images and audio data apologizing for the project delay.

[0331] The server performs image and audio analysis to point out inappropriate comments and unclear improvement suggestions.

[0332] The server uses an emotion engine to analyze the user's emotions and provide feedback based on that. For example, if the user is feeling down, it might add words of encouragement or an appropriate apology.

[0333] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[0334] Improve your apology presentation based on user-provided feedback.

[0335] This concludes the "Mode for Carrying Out the Invention" of the present invention. This system not only provides specific and detailed feedback to help users give effective presentations, but also enables support based on the user's emotional state.

[0336] The processing flow will be explained below.

[0337] Step 1:

[0338] The user uploads image files of presentation materials and presentation audio data from a dedicated interface.

[0339] Step 2:

[0340] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[0341] Step 3:

[0342] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[0343] Step 4:

[0344] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0345] Step 5:

[0346] The image analysis module extracts the text within the image file and returns the text data to the server.

[0347] Step 6:

[0348] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[0349] Step 7:

[0350] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0351] Step 8:

[0352] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[0353] Step 9:

[0354] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0355] Step 10:

[0356] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[0357] Step 11:

[0358] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[0359] Step 12:

[0360] The server passes the user's facial image contained in the image file to the image emotion analysis means, which analyzes the emotion.

[0361] Step 13:

[0362] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[0363] Step 14:

[0364] The server comprehensively analyzes the acquired voice emotion data and image emotion data and passes it to the natural language processing module.

[0365] Step 15:

[0366] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0367] Step 16:

[0368] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0369] Step 17:

[0370] Based on the analysis results and emotional data obtained by the server from the natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0371] Step 18:

[0372] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0373] Step 19:

[0374] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0375] Step 20:

[0376] The server uses a personalized learning module to generate feedback optimized for a particular reviewer by referencing past feedback data.

[0377] Step 21:

[0378] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[0379] Step 22:

[0380] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[0381] Step 23:

[0382] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[0383] Example 2

[0384] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] Conventional presentation analysis systems provide limited feedback based on the analysis of image files and audio data, and are insufficient in individual optimization that takes into account the user's emotions. Furthermore, improvements are needed in the accuracy of the analysis results and the specific content of the feedback. Therefore, a system is needed that provides detailed and specific feedback to help users deliver effective presentations and generates feedback that corresponds to the user's emotional state.

[0386] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload an image file of a presentation material, a means for a user to upload presentation audio data, a means for a terminal to transmit a file designated by the user to the server, a means for the server to save the received file in a temporary storage directory, a means for the server to read the image file from the temporary storage directory and pass it to an image analysis module, a means for the image analysis module to extract text from the image file and return it to the server, a means for the server to format the acquired text data and save it in a data store for the natural language processing module, a means for the server to read an audio file from the temporary storage directory and pass it to the audio analysis module, a means for the audio analysis module to convert audio data into text and return it to the server, and a means for the server to convert the acquired audio text data into text. The system includes a means for formatting the data and analysis results and saving them in a data store, a means for the server to pass text data obtained from the image and audio analyses to a natural language processing module, a means for the natural language processing module to analyze the text data and detect redundancy, unclear expressions, and logical inconsistencies, a means for the server to generate feedback including specific improvements and suggestions based on the analysis results, a means for the server to pass audio data to a voice emotion analysis means and analyze the user's emotions, a means for the voice emotion analysis means to extract emotions from the audio data and return this to the server, a means for the server to pass facial images in image files to an image emotion analysis means and analyze them, a means for the image emotion analysis means to extract emotions from the facial images and return this to the server, and a means for the server to comprehensively analyze the audio emotion data and image emotion data and reflect this in feedback. This makes it possible to provide individually optimized, detailed feedback that takes into account the emotional state of the user based on the analysis results of the user's presentation materials and presentation audio data.

[0387] "User" refers to a person who uploads image files of presentation materials and presentation audio data through the system interface.

[0388] "Terminal" refers to a device that reads a file specified by a user and sends it to a server.

[0389] A "server" refers to a computer system that stores files received from users in a temporary storage directory and performs various processes such as image analysis and audio analysis.

[0390] The "temporary storage directory" refers to a directory for temporarily storing files received by the server.

[0391] "Image analysis module" refers to software or hardware for extracting text from image files.

[0392] "Speech Analysis Module" refers to software or hardware for converting speech data into text and extracting the text data and speech characteristics.

[0393] A "natural language processing module" refers to software that analyzes text data and detects redundancies, unclear expressions, and logical inconsistencies.

[0394] "Voice emotion analysis means" refers to software or hardware for extracting a user's emotion from voice data.

[0395] "Image emotion analysis means" refers to software or hardware for extracting a user's emotion from a facial image.

[0396] "Natural language processing technology" refers to all technologies for analyzing text data and understanding its meaning and structure.

[0397] "Data store" refers to a database or storage system for storing text data and voice characteristic data obtained through analysis.

[0398] "Weighting algorithm" refers to a computational method for adjusting analysis results based on specific review destinations or presentation content.

[0399] "Personalized Learning Module" refers to software that references past feedback data and optimizes feedback for specific reviewers.

[0400] This invention relates to a system that analyzes image files and presentation audio data for presentation materials and provides feedback. It also has a function to generate feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions. The main components of this system include a data receiving means, an image analysis means, an audio analysis means, a natural language processing means, a feedback generation means, a weighting means, an individual learning means, an emotion engine (a voice emotion analysis means and an image emotion analysis means), and a feedback display means.

[0401] The user uploads image files of presentation materials and audio data for the presentation through a dedicated interface. The device reads the files specified by the user and sends them to the server. The server saves the received files in a temporary storage directory. At this time, a timestamp and unique ID are added to the file name to ensure the uniqueness and security of the file.

[0402] The server reads the image files from the temporary storage directory and passes them to the image analysis module, which extracts the text from the image files and returns the text data to the server, which formats the text data and stores it in a data store for the natural language processing module. This extracts the text information from the presentation slides and prepares them for the next processing step.

[0403] For speech analysis, the server reads the audio file from the temporary storage directory and passes it to the speech analysis module. The speech analysis module converts the audio data into text and returns the text data and speech characteristics to the server. The server formats the acquired speech text data and analysis results and saves them in a data store for the natural language processing module. This allows the presentation's speaking style and content to be extracted as text data.

[0404] Next, in the natural language processing and feedback generation step, the server passes the text data obtained from image and audio analysis to a natural language processing module. The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc. Based on the analysis results obtained from the natural language processing module, the server generates written feedback including specific points for improvement and suggestions.

[0405] Furthermore, in emotion engine processing, the server passes voice data to voice emotion analysis means, which analyzes the user's emotion. The voice emotion analysis means extracts emotion from the voice data and returns this emotion data to the server. The server also passes the user's facial image contained in the image file to image emotion analysis means, which analyzes the emotion. The image emotion analysis means extracts emotion from the facial image and returns this emotion data to the server. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects the results in feedback.

[0406] In the weighting and personalized optimization step, the user inputs the specific reviewer (e.g., president, manager, client) and presentation content (proposal, apology, etc.). The server applies a weighting algorithm based on the received information and adjusts the analysis results. The server then uses a personalized learning module to generate feedback optimized for the specific reviewer by referencing past feedback data.

[0407] Below is an example prompt:

[0408] Proposal presentation example

[0409] I have uploaded the following data in order to make a presentation proposing a new product.

[0410] Slide image: New product proposal slide

[0411] Audio data: Presentation audio

[0412] We ask for feedback on how to make the proposal clear and how to reflect any nervousness you may have.

[0413] Apology Presentation Example

[0414] The following data has been uploaded in order to make an apology presentation for the project delay.

[0415] Slide image: Apology slide

[0416] Audio data: Presentation audio

[0417] Point out inappropriate language or confusing parts and ask for feedback that includes encouraging words that reflect your disappointment.

[0418] This system provides specific and detailed feedback to help users give effective presentations, and enables support according to the user's emotional state.

[0419] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0420] Step 1:

[0421] The user uses a dedicated interface to select image files of presentation materials and presentation audio data.

[0422] Input: User-selected image and audio files

[0423] Output: Path of the selected file

[0424] Specific operation: The user opens a browser or a dedicated app, clicks the "Select files" button, selects the image and audio files to upload, and clicks the "Upload" button.

[0425] Step 2:

[0426] The terminal reads the image file and audio file specified by the user and transmits them to the server.

[0427] Input: Paths to image and audio files

[0428] Output: Image and audio files sent to the server

[0429] Specific operation: The device loads the selected file into memory and sends the file data to the server using an HTTP POST request. After sending, the device notifies the user that the file upload is complete.

[0430] Step 3:

[0431] The server saves the received file in a temporary directory.

[0432] Input: Image and audio files sent from the device

[0433] Output: File saved in temporary directory

[0434] Specific operation: The server saves the received file in a temporary directory. When saving, a timestamp and a unique ID are added to the file name to ensure the uniqueness and security of the file.

[0435] Step 4:

[0436] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0437] Input: Image file saved in temporary directory

[0438] Output: Image data passed to the image analysis module

[0439] Specific operation: The server reads the target image file from the temporary storage directory into memory and passes the file data to the API endpoint of the image analysis module.

[0440] Step 5:

[0441] The image analysis module extracts the text within the image file and returns the text data to the server.

[0442] Input: Image data passed to the image analysis module

[0443] Output: Text data extracted from the image

[0444] Specific operation: The image analysis module uses OCR technology (e.g., Tesseract) to extract text from the image and returns the extracted text data to the server in JSON format.

[0445] Step 6:

[0446] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[0447] Input: Text data returned from the image analysis module

[0448] Output: Formatted text data

[0449] Specific operation: The server converts the text data it receives into a specific format (e.g., separates each sentence) and saves the formatted data in a data store (e.g., MongoDB).

[0450] Step 7:

[0451] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0452] Input: Audio file saved in temporary directory

[0453] Output: Audio data passed to the audio analysis module

[0454] Specific operation: The server reads the target audio file from the temporary storage directory into memory and passes the file data to the API endpoint of the audio analysis module.

[0455] Step 8:

[0456] The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[0457] Input: Audio data passed to the audio analysis module

[0458] Output: Text data and speech characteristics extracted from the audio

[0459] Specific operation: The speech analysis module converts speech into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and returns the text data and speech characteristics (e.g., voice tone, speed, volume) to the server in JSON format.

[0460] Step 9:

[0461] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0462] Input: Speech text data and speech characteristics data returned from the speech analysis module

[0463] Output: Formatted speech text data and speech characteristics data

[0464] Specific operation: The server converts the acquired voice text data into a specific format (e.g., separates it into sentences) and saves the data and voice characteristic data in a data store.

[0465] Step 10:

[0466] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0467] Input: Formatted image and audio text data

[0468] Output: Text data passed to the natural language processing module

[0469] Specific operation: The server reads the text data from the image analysis and voice analysis from the data store and sends the data to the API endpoint of the natural language processing module.

[0470] Step 11:

[0471] A natural language processing module analyzes text data to detect redundancies, unclear expressions, and logical inconsistencies.

[0472] Input: Text data passed to the natural language processing module

[0473] Output: Feedback data for the parsed text data

[0474] Specific operation: The natural language processing module uses a generative AI model (e.g., GPT-3) to analyze the text data, detecting redundancies, unclear expressions, logical inconsistencies, etc., and returns the analysis results to the server in JSON format.

[0475] Step 12:

[0476] Based on the analysis results, the server generates feedback including specific areas for improvement and suggestions.

[0477] Input: Analysis results returned from the natural language processing module

[0478] Output: Feedback document with specific improvements and suggestions

[0479] Specific actions: The server compiles the analysis results and generates a feedback document containing specific improvements and suggestions.

[0480] Step 13:

[0481] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[0482] Input: Audio file saved in temporary directory

[0483] Output: Emotion data based on audio data

[0484] Specific operation: The server passes the voice data to the API of the voice emotion analysis module, determines the emotion based on the voice characteristics (e.g., tone, intensity, speed) and returns the emotion data.

[0485] Step 14:

[0486] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[0487] Input: Audio data passed to the audio emotion analysis module

[0488] Output: Extracted emotion data

[0489] Specific operation: The voice emotion analysis module analyzes the voice data, extracts emotions (e.g., tension, anxiety, joy), and returns the results to the server in JSON format.

[0490] Step 15:

[0491] The server passes the facial image in the image file to the image emotion analysis means, and analyzes the emotion.

[0492] Input: Face image saved in temporary directory

[0493] Output: Emotion data based on facial images

[0494] Specific operation: The server passes the user's facial image in the image file to the API of the image emotion analysis module, and determines the emotion based on facial expression characteristics (e.g., smile, eyebrow movement).

[0495] Step 16:

[0496] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[0497] Input: A face image passed to the image emotion analysis module

[0498] Output: Extracted emotion data

[0499] Specific operation: The image emotion analysis module analyzes the facial image, extracts emotions (e.g., happiness, surprise, sadness), and returns the results to the server in JSON format.

[0500] Step 17:

[0501] The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[0502] Input: Extracted speech and image emotion data

[0503] Output: Feedback text reflecting the sentiment data

[0504] Specific operation: The server integrates emotional data obtained from audio and images and adds feedback according to the user's emotional state (e.g., how to relax if tense, words of encouragement if depressed).

[0505] As described above, the program of this system includes a process for multifaceted analysis of presentation materials and audio data uploaded by the user and generating detailed feedback that takes the user's emotions into consideration.

[0506] (Application example 2)

[0507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0508] Conventional presentation feedback systems simply analyze text and voice, and do not provide detailed feedback tailored to the user's emotional state or the intended recipient. As a result, they may not receive optimal instruction to improve their presentation skills. In the training of logistics center staff, improving presentation skills contributes to improved work efficiency, so a system that provides more effective and individually optimized feedback is needed.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for performing emotion analysis and generating feedback according to the user's emotional state, and means for generating information for training logistics center staff and improving their presentation skills. This enables logistics center staff to improve their presentation skills and receive optimized feedback according to their emotional state.

[0510] "Presentation materials" refers to image files such as slides and documents used in a presentation.

[0511] "Presentation audio data" refers to digital data of the audio spoken by the presenter during a presentation.

[0512] "Image analysis" refers to the technique of extracting text and important visual information from image files.

[0513] "Voice analysis" refers to the technology of converting voice data into text and analyzing speaking style, emphasis, etc.

[0514] "Feedback" refers to specific advice or suggestions regarding the quality of the presentation and areas for improvement.

[0515] "Emotion analysis" refers to the technology of detecting and analyzing a user's emotional state from their voice and visual data.

[0516] A "logistics center" refers to a facility that stores, sorts, and ships goods and materials.

[0517] "Education" refers to the guidance and training of logistics center staff aimed at improving work efficiency and presentation skills.

[0518] "Natural language processing technology" refers to artificial intelligence technology for analyzing text data, understanding its meaning and structure, and processing it.

[0519] "Personalized learning" refers to a technology that uses past data to learn in order to provide optimal feedback for specific users and situations.

[0520] "Weighting" refers to the practice of prioritizing and adjusting analysis results based on specific review destinations or presentation content.

[0521] "Server" refers to a computer system that receives, stores, analyzes data, and generates feedback.

[0522] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[0523] The system for implementing this invention receives image files of presentation materials and presentation audio data, analyzes them, and generates and displays feedback. The system program and its processing flow are described below.

[0524] System Overview

[0525] The system consists of the following main components:

[0526] A means of receiving image files of presentation materials

[0527] A means of receiving presentation audio data

[0528] means for analyzing the image file and the audio data, respectively;

[0529] means for generating feedback based on the analysis results;

[0530] A means of performing emotion analysis and generating feedback based on the user's emotional state

[0531] A means of providing information to train logistics center staff and improve their presentation skills

[0532] Hardware and Software Configuration

[0533] Hardware:

[0534] Smartphone (iOS / Android)

[0535] Server (AWS, Google Cloud GCP, etc.)

[0536] Head-mounted displays (Microsoft HoloLens, Google Glass)

[0537] software:

[0538] Image processing library (OpenCV)

[0539] Speech analysis library (Google Cloud Speech-to-Text API)

[0540] Natural Language Processing (spaCy, NLTK)

[0541] Sentiment analysis library (TensorFlow, Keras)

[0542] Processing flow explanation

[0543] 1. Data Receipt and Storage:

[0544] The user uploads image files and audio data of presentation materials via a smartphone application.

[0545] The device reads the data and sends it to the server.

[0546] The server receives these files and stores them in a secure temporary storage directory.

[0547] 2. Image Analysis:

[0548] The server loads the image file from the temporary storage directory.

[0549] Image analysis module (OpenCV) extracts text and important visual information from image files.

[0550] 3. Audio analysis:

[0551] The server passes the audio file to a speech analysis module (Google Cloud Speech-to-Text API) and converts it into text.

[0552] The speech analysis module analyzes speaking style and emphasis, and generates text data.

[0553] 4. Natural Language Processing and Feedback Generation:

[0554] The server passes the text data obtained from image analysis and voice analysis to a natural language processing module (spaCy).

[0555] The natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[0556] 5. Emotion analysis:

[0557] The server passes the audio data and image files to an emotion analysis module (TensorFlow, Keras) to analyze the user's emotional state.

[0558] Obtain emotional data and reflect it in feedback.

[0559] Specific examples

[0560] Presentation support with smart trainers

[0561] Users upload slides about new logistics solutions and audio data for presentations based on them.

[0562] The server analyzes the images and audio and generates text data about the content of the slides.

[0563] The natural language processing module detects redundant expressions and logical inconsistencies and generates feedback suggesting specific improvements.

[0564] The sentiment analysis module detects nervousness or lack of confidence during a presentation and adds encouraging words or advice as feedback.

[0565] Prompt Sentence Examples

[0566] "Please advise the user on improvements based on the slide presentation and audio data."

[0567] "Perform emotional analysis during presentations and offer advice on how to relax if people are nervous."

[0568] As described above, this system helps logistics center staff improve their presentation skills by analyzing presentation materials and audio data, and provides optimal feedback according to their emotional state.

[0569] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0570] Specific processing steps of the program

[0571] Step 1:

[0572] Data Receipt and Storage

[0573] Input: Image files of presentation materials and audio data of presentations

[0574] How it works: A user uploads image files and audio data for a presentation using a smartphone application.

[0575] Output: Image files and audio data are sent to the server and stored in a secure temporary directory.

[0576] Step 2:

[0577] Loading and analyzing image files

[0578] Input: Image file saved in temporary directory

[0579] Operation: The server passes the image file to the image analysis module (OpenCV).

[0580] Data processing / calculation: The image analysis module extracts text and important visual information from the image files and returns it to the server.

[0581] Output: Extracted text data

[0582] Step 3:

[0583] Loading and analyzing audio files

[0584] Input: Audio file saved in temporary directory

[0585] How it works: The server passes the audio file to the speech analysis module (Google Cloud Speech-to-Text API).

[0586] Data processing / calculation: The speech analysis module converts the speech data into text, analyzes the speech characteristics (emphasis, speaking style, etc.), and returns the text to the server.

[0587] Output: Converted text data and voice characteristics data

[0588] Step 4:

[0589] Natural Language Processing and Feedback Generation

[0590] Input: Extracted text data and speech characteristics data

[0591] How it works: The server passes this data to the natural language processing module (spaCy).

[0592] Data processing / calculation: A natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[0593] Output: Feedback text with improvements

[0594] Step 5:

[0595] Emotion analysis

[0596] Input: Audio data and image files

[0597] How it works: The server passes the audio data and image files to the emotion analysis module (TensorFlow, Keras).

[0598] Data processing / calculation: The emotion analysis module analyzes the user's emotional state (e.g., tension, relief, excitement, etc.) based on their voice and image, and returns the result to the server.

[0599] Output: Emotional state data

[0600] Step 6:

[0601] Consolidating and providing final feedback

[0602] Input: Feedback text and emotional state data

[0603] Operation: The server integrates the emotional state data with the feedback text to generate the final feedback.

[0604] Output: Consolidated feedback text

[0605] What it does: The server displays the consolidated feedback to the user.

[0606] The above are the specific processing steps of this system. Through the data processing and calculations performed at each step, the user can receive detailed feedback based on the presentation materials and audio data, and that is tailored to the user's emotional state.

[0607] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0608] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0609] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0610] [Second embodiment]

[0611] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0612] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0613] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0614] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0615] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0616] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0617] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0618] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0619] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0620] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0621] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0622] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0623] This invention relates to a system that analyzes image files and audio data of presentation materials and provides feedback. The program processing of this system is explained in detail below in natural language.

[0624] System Overview

[0625] Users upload image files to be used in their presentations and audio data recording the presentation to the system. The server receives these files and performs image and audio analysis. The analyzed text data is further analyzed using natural language processing technology to generate specific feedback based on the presentation's content and speaking style. The feedback is weighted based on the specific reviewer and presentation content and provided to the user.

[0626] Program processing

[0627] Receiving and storing data

[0628] 1. After the user logs in, they upload the image files of the presentation materials and the presentation audio data through a dedicated interface.

[0629] 2. The terminal reads the file specified by the user and sends it to the server.

[0630] 3. The server receives these files and stores them in a secure temporary storage directory.

[0631] Image analysis

[0632] 1. The server loads the image file from the temporary storage directory.

[0633] 2. The server starts the image analysis module and performs processing to convert the image file into text data.

[0634] 3. The image analysis module extracts the text in the image file and returns the data to the server.

[0635] 4. The server formats the retrieved text data and prepares it for passing to the natural language processing module.

[0636] Audio analysis

[0637] 1. The server loads the audio file from the temporary storage directory.

[0638] 2. The server launches the speech analysis module and processes the speech data to convert it into text.

[0639] 3. The speech analysis module converts the speech into text data and returns the data to the server.

[0640] 4. The server formats the acquired speech-to-text data and prepares it for passing to the natural language processing module.

[0641] Natural Language Processing and Feedback Generation

[0642] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[0643] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[0644] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0645] Weighting and Individual Optimization

[0646] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[0647] 2. The server applies a weighting algorithm based on the received information to adjust the analysis results.

[0648] 3. The server uses a personalized learning module to look at past feedback data and generate feedback optimized for the specific reviewer.

[0649] Specific examples

[0650] Example 1: Proposal presentation

[0651] The user uploads slide images and audio data for new product proposals.

[0652] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[0653] The server generates specific feedback based on the analysis, such as identifying redundancies in the proposal or adding specific use cases to attract the client's interest.

[0654] Improve your presentation materials and delivery based on the feedback provided by users.

[0655] Example 2: Apology presentation

[0656] A user uploads slide images and audio data apologizing for the project delay.

[0657] The server performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[0658] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[0659] Improve your apology presentation based on user-provided feedback.

[0660] This concludes the description of the "Mode for Carrying Out the Invention" of the present invention. This system provides specific and detailed feedback to help users give effective presentations.

[0661] The processing flow will be explained below.

[0662] Step 1:

[0663] Users upload image files of presentation materials and presentation audio data through a dedicated interface.

[0664] Step 2:

[0665] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[0666] Step 3:

[0667] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[0668] Step 4:

[0669] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0670] Step 5:

[0671] The image analysis module extracts the text within the image file and returns the text data to the server.

[0672] Step 6:

[0673] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[0674] Step 7:

[0675] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0676] Step 8:

[0677] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[0678] Step 9:

[0679] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0680] Step 10:

[0681] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0682] Step 11:

[0683] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0684] Step 12:

[0685] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[0686] Step 13:

[0687] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0688] Step 14:

[0689] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0690] Step 15:

[0691] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[0692] Step 16:

[0693] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[0694] Step 17:

[0695] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[0696] Step 18:

[0697] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[0698] Example 1

[0699] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] Conventional presentation feedback systems mainly analyze image files and audio data individually, which has the drawback of insufficient comprehensive analysis and specific feedback to users. Furthermore, since it is not possible to weight or optimize based on specific reviewers or presentation content, it is difficult to obtain the specific and accurate feedback that users need.

[0701] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0702] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for analyzing the analyzed text data using natural language processing technology, means for generating feedback based on the analysis results, means for displaying the feedback, means for the user to input specific review recipients and presentation content, means for weighting the feedback based on the input information, and means for optimizing the feedback using past feedback data. This makes it possible to comprehensively analyze the presentation materials and audio data and provide specific and accurate feedback based on the specific review recipients and presentation content.

[0703] "Presentation materials" are a collection of images and slides that visually represent the contents of a presentation.

[0704] "Image files" refer to still image data that has been digitized from presentation materials.

[0705] "Presentation audio data" refers to audio data recorded from a presentation.

[0706] "Server" refers to the central computer system used for analysis, data processing, and feedback generation.

[0707] A "terminal" refers to a computer or device operated by a user, and is responsible for exchanging data with a server.

[0708] "User" refers to a person who uses the system to upload presentation data and receive feedback.

[0709] "Analysis means" refers to the function of converting image files and audio data into text data and analyzing it using natural language processing technology.

[0710] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[0711] "Feedback" refers to information, including suggestions for improvement and recommendations, provided to users based on analyzed data.

[0712] "Weighting" refers to the process of adjusting the analysis results based on the specific review destination or presentation content.

[0713] "Optimization" refers to using past feedback data to make adjustments to improve the accuracy and usefulness of feedback.

[0714] The present invention relates to a system for analyzing image files and audio data of presentation materials and providing feedback to users. An embodiment of this system will be described in detail below.

[0715] System Overview

[0716] The system of the present invention allows users to upload image files used as presentation materials and audio data for the presentation, and then analyzes the materials and audio and provides feedback. Specifically, the server receives and analyzes these files, and uses natural language processing technology to generate feedback appropriate for the user.

[0717] Hardware and software used

[0718] Server: A central computer system that analyzes and processes data and generates feedback. The server is equipped with image analysis modules, audio analysis modules, and natural language processing modules.

[0719] Device: A computer or device operated by a user to upload files or display feedback.

[0720] Image Analysis Module: Analyzes image files using image analysis libraries such as Python, OpenCV, and Tesseract OCR.

[0721] Speech analysis module: Analyzes speech data using speech analysis APIs such as Google Speech-to-Text and IBM Watson Speech to Text.

[0722] Natural Language Processing module: Analyzes and processes text data using the NLTK library and generative AI models.

[0723] Data processing and calculation

[0724] When a user uploads image files and audio data for a presentation, the device sends these files to the server. The server receives these files and stores them in a secure temporary directory. The server then launches image and audio analysis modules to convert the image files into text data and the audio data into text.

[0725] The server passes the acquired text data to a natural language processing module to detect redundancies, unclear expressions, and logical inconsistencies. The server generates feedback based on the analysis results and provides it to the user.

[0726] Additionally, when a user inputs a specific review destination or presentation content (e.g., proposal, apology), the server applies a weighting algorithm and references past feedback data to optimize the feedback.

[0727] Specific examples

[0728] Proposal presentation example

[0729] User: Uploads slide images and audio data for a new product proposal presentation to the system.

[0730] Server: Performs image and audio analysis to analyze the content of slides and speaking style.

[0731] Server: Generates specific feedback based on the analysis, such as redundancies in the proposal or adding specific use cases to capture the client's interest.

[0732] Users: Improve their presentation materials and delivery based on the feedback provided.

[0733] Example of an apology presentation

[0734] User: Upload slide images and audio data to the system apologizing for the project delay.

[0735] Server: Performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[0736] Server: Based on the analysis results, provide feedback on specific improvement measures and how to choose the right words to use in your apology.

[0737] User: Improve your apology presentation based on the feedback provided.

[0738] This is the embodiment of the present invention. This system allows users to receive specific and detailed feedback on how to give effective presentations.

[0739] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0740] Step 1:

[0741] A user logs in.

[0742] Input: User ID and password.

[0743] How it works: A user accesses the system's login page and enters their user ID and password.

[0744] Output: If authentication is successful, the user can access the main interface of the system.

[0745] Step 2:

[0746] The user uploads image files of the presentation materials and audio data of the presentation.

[0747] Input: Image files and audio data files.

[0748] How it works: The user selects image and audio files in a dedicated interface and clicks the upload button.

[0749] Output: The selected file is prepared by the terminal.

[0750] Step 3:

[0751] The terminal reads the file specified by the user and sends it to the server.

[0752] Input: Image files and audio data files.

[0753] How it works: The device gets the path of the selected file and sends it to the server using an HTTP POST request.

[0754] Output: The file data sent to the server.

[0755] Step 4:

[0756] The server receives the file and stores it in a secure temporary storage directory.

[0757] Input: Submitted image and audio data files.

[0758] How it works: The server saves the received file in a temporary directory, such as the / tmp directory.

[0759] Output: File saved in temporary directory.

[0760] Step 5:

[0761] The server loads the image file from the temporary storage directory.

[0762] Input: The path to the saved image file.

[0763] What it does: The server loads the image file from the temporary storage directory into memory.

[0764] Output: The loaded image data.

[0765] Step 6:

[0766] The server starts the image analysis module and converts the image file into text data.

[0767] Input: The loaded image data.

[0768] How it works: The server uses image analysis libraries such as OpenCV and Tesseract OCR to extract text from images.

[0769] Output: The extracted text data.

[0770] Step 7:

[0771] The server reads the audio file from the temporary storage directory.

[0772] Input: The path to the saved audio file.

[0773] How it works: The server loads the audio file from the temporary storage directory into memory.

[0774] Output: The loaded audio data.

[0775] Step 8:

[0776] The server starts a voice analysis module and converts the voice data into text data.

[0777] Input: The loaded audio data.

[0778] How it works: The server uses a speech analysis API such as Google Speech-to-Text or IBM Watson Speech to Text to convert the audio data into text.

[0779] Output: The converted speech-to-text data.

[0780] Step 9:

[0781] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0782] Input: Extracted image-text data and audio-text data.

[0783] How it works: The server sends text data to the natural language processing module via an API.

[0784] Output: Text data sent to the natural language processing module.

[0785] Step 10:

[0786] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0787] Input: The text data sent.

[0788] How it works: The natural language processing module uses the NLTK library and generative AI models to analyze the quality of text.

[0789] Output: Analysis result data.

[0790] Step 11:

[0791] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[0792] Input: Analysis result data.

[0793] How it works: The server uses templates based on the analysis results to summarize specific improvements and suggestions in natural language.

[0794] Output: The generated feedback sentence.

[0795] Step 12:

[0796] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0797] Input: Specific reviewer and presentation content.

[0798] How it works: The user selects and inputs the review recipient and presentation purpose in the interface.

[0799] Output: The input review destination and presentation content data.

[0800] Step 13:

[0801] Based on the information received, the server applies a weighting algorithm to adjust the analysis results.

[0802] Input: Review destination and presentation content data, analysis result data.

[0803] How it works: The server applies weighting parameters appropriate to the reviewer and presentation content to adjust the feedback.

[0804] Output: Weighted analysis result data.

[0805] Step 14:

[0806] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[0807] Input: Weighted analysis result data, past feedback data.

[0808] How it works: The server queries a database of past feedback and optimizes the feedback based on feedback performance in similar situations.

[0809] Output: Optimized feedback text.

[0810] The above is the specific flow of program processing in this system.

[0811] (Application example 1)

[0812] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0813] Conventional presentation training systems simply require users to upload presentation materials and audio data, which the system then analyzes and provides feedback. This makes it difficult to provide feedback optimized for specific environments (e.g., in a factory) or situations (e.g., safety measures presentations). They also lack the ability to detect redundant sentences or unclear expressions in real time and immediately suggest specific areas for improvement. This leaves users facing specific challenges in conducting effective presentation training.

[0814] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0815] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for displaying the feedback, and means for coordinating with a robot that performs presentation training in a factory. This allows users to receive feedback in real time that is optimized for specific environments and situations in the factory, enabling more effective presentation training.

[0816] An "image file of presentation materials" is a file in which information such as slides and charts used in a presentation is saved in image format.

[0817] "Presentation audio data" refers to data recorded in audio format of what was said during a presentation.

[0818] "Means for analysis" refers to the technology or device that converts received image files or audio data into text information and performs analysis.

[0819] "Means for generating feedback" refers to technology or devices that generate suggestions for improvement or comments to be provided to users based on the content of the presentation and the results of data analysis.

[0820] "Means for displaying feedback" refers to techniques or devices for visually or audibly presenting the generated feedback to the user.

[0821] A "robot for in-factory presentation training" is an autonomous or remotely controlled mechanical device that supports presentation training in a factory and provides feedback.

[0822] A "weighting means" is a technique or device that adjusts the evaluation or results based on the analyzed text data according to specific criteria.

[0823] "Means for optimizing feedback through individual learning" refers to technology or devices that generate feedback optimized for each user based on past analysis results and feedback data.

[0824] "Means for converting voice data to text" refers to technology or devices that use voice recognition technology to convert the spoken content in an audio file into text form.

[0825] "Means for analyzing using natural language processing technology" refers to technology or devices that perform semantic analysis and error checking of text data converted using natural language processing technology.

[0826] "Means for detecting redundant sentences and unclear expressions" refers to technology or devices that identify unimportant long sentences and difficult-to-understand expressions from the analyzed text data and point them out to the user.

[0827] This invention relates to a system that analyzes image files of presentation materials and audio data of presentations and provides feedback. This system is intended to support presentation training, particularly in factories, and to provide users with optimized feedback.

[0828] System configuration and program processing

[0829] Receiving and storing data

[0830] The server receives and temporarily stores image files and audio data for presentation materials uploaded by users via the interface of the robots in the factory. The server stores the received files in a secure temporary storage directory and prepares them for analysis.

[0831] Image analysis

[0832] The server uses PIL (Python Imaging Library) and pytesseract libraries to convert image files to text, extracting the text data within the image files, which is then formatted by the server and prepared for natural language processing.

[0833] Audio analysis

[0834] The server uses the speech_recognition library to convert the audio file to text, which is then formatted and prepared for natural language processing as well.

[0835] Natural Language Processing and Feedback Generation

[0836] The server analyzes the text data obtained from image and audio analysis using natural language processing technology. This process uses the spacy library. During the analysis, redundant sentences and unclear expressions are identified in the text data, and specific feedback on improvements is generated.

[0837] Weighting and Individual Optimization

[0838] The server weights specific review destinations and presentation content based on the analyzed text data, and also provides optimized feedback for each user by referencing past feedback data through personalized learning.

[0839] Specific examples

[0840] Example 1: Proposal presentation

[0841] When a user gives a presentation to propose a new product in a factory, they upload slide images and audio data. The server analyzes these files and generates specific feedback such as advice like, "The text on this slide is redundant. Please explain it more succinctly."

[0842] Example 2: Safety Presentation

[0843] When giving a presentation on safety measures in a factory, the user uploads the materials and audio data. The server analyzes the data and provides feedback such as, "The explanation of safety precautions tends to be redundant. In particular, make the section on 'checking safety devices' more concise."

[0844] Example prompt sentence:

[0845] "The text on this slide is redundant. Please explain it more succinctly."

[0846] "Safety precautions can be redundant. Please be more concise, especially the 'check safety equipment' section."

[0847] This invention allows users to receive real-time feedback optimized for specific environments and situations within a factory, enabling more effective presentation training. This system is an important tool for improving the effectiveness of presentations in specific situations.

[0848] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0849] Step 1:

[0850] A user logs in and uploads image files and audio data for presentation materials. The user uses the robot interface in the factory to select these files and send them to the server. The input is the image files and audio files, and the output is the server receiving the files.

[0851] Step 2:

[0852] After the server receives the uploaded files, it stores them in a secure temporary directory. The input is the image and audio files uploaded by the user, and the output is that these files are stored in the temporary directory.

[0853] Step 3:

[0854] The server reads the saved image file and converts it to text data using image analysis modules (PIL and pytesseract). The input is the image file saved in the temporary storage directory, and the output is the converted text data. This text data is returned to the server.

[0855] Step 4:

[0856] The server reads the saved audio file and converts it into text using a speech analysis module (speech_recognition library). The input is the audio file saved in the temporary storage directory, and the output is the converted text. This text data is also returned to the server.

[0857] Step 5:

[0858] The server passes the text data obtained from image and audio analysis to a natural language processing module (the spacy library) for analysis. The input is the converted text data, and the output is the analysis results. Based on these analysis results, redundant sentences and unclear expressions are detected.

[0859] Step 6:

[0860] The server generates specific feedback based on the analysis results of the natural language processing. The input is the analysis results from the natural language processing module, and the output is the generated feedback. This feedback includes specific advice on redundant sentences and areas for improvement.

[0861] Step 7:

[0862] The server weights the generated feedback based on the specific reviewer and presentation content. The input is feedback and reviewer information, and the output is weighted feedback.

[0863] Step 8:

[0864] The server uses an individual learning module to generate optimized feedback by referring to past feedback data, with the input being the past feedback data and the current feedback, and the output being the optimized feedback.

[0865] Step 9:

[0866] The server visually displays the generated feedback to the user: the input is the optimized feedback, and the output is the user seeing the feedback on the screen.

[0867] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0868] overview

[0869] This invention relates to a system that analyzes image files and audio data from presentation materials and provides feedback. It also has the function of generating feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions.

[0870] System configuration

[0871] The system consists of the following main components:

[0872] 1. Data Receiving Method

[0873] 2. Image analysis methods

[0874] 3. Audio analysis methods

[0875] 4. Natural Language Processing Methods

[0876] 5. Feedback Generation Methods

[0877] 6. Weighting Methods

[0878] 7. Personalized learning tools

[0879] 8. Emotion Engine (Voice Emotion Analysis and Image Emotion Analysis)

[0880] 9. Feedback Display Methods

[0881] Program processing

[0882] The program processing of this system will be specifically explained below.

[0883] Receiving and storing data

[0884] 1. The user uploads the image files of the presentation materials and the presentation audio data through a dedicated interface.

[0885] 2. The terminal reads the file specified by the user and sends it to the server.

[0886] 3. The server receives the file and stores it in a secure temporary storage directory.

[0887] Image analysis

[0888] 1. The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0889] 2. The image analysis module extracts the text from the image file and returns the text data to the server.

[0890] 3. The server formats the text data it receives and stores it in a data store for the natural language processing module.

[0891] Audio analysis

[0892] 1. The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0893] 2. The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[0894] 3. The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0895] Natural Language Processing and Feedback Generation

[0896] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[0897] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[0898] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0899] Emotion engine processing

[0900] 1. The server passes the voice data to the voice emotion analysis means and analyzes the user's emotions.

[0901] 2. The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[0902] 3. The server passes the user's facial image contained in the image file to the image emotion analysis means, and analyzes the emotion.

[0903] 4. The image emotion analysis means extracts emotions from the facial image and returns the emotion data to the server.

[0904] 5. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[0905] Weighting and Individual Optimization

[0906] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[0907] 2. Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0908] 3. The server uses a personalized learning module to generate feedback optimized for the specific reviewer, referencing past feedback data.

[0909] Specific examples

[0910] Example 1: Proposal presentation

[0911] The user uploads slide images and audio data for new product proposals.

[0912] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[0913] The server uses an emotion engine to analyze the user's emotions and provides feedback based on that. For example, if the user is nervous, the server adds advice to ease the tension.

[0914] Based on the analysis, the server generates specific feedback, such as redundancy in the proposal or adding use cases to attract the client's interest.

[0915] Improve your presentation materials and delivery based on the feedback provided by users.

[0916] Example 2: Apology presentation

[0917] A user uploads slide images and audio data apologizing for the project delay.

[0918] The server performs image and audio analysis to point out inappropriate comments and unclear improvement suggestions.

[0919] The server uses an emotion engine to analyze the user's emotions and provide feedback based on that. For example, if the user is feeling down, it might add words of encouragement or an appropriate apology.

[0920] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[0921] Improve your apology presentation based on user-provided feedback.

[0922] This concludes the "Mode for Carrying Out the Invention" of the present invention. This system not only provides specific and detailed feedback to help users give effective presentations, but also enables support based on the user's emotional state.

[0923] The processing flow will be explained below.

[0924] Step 1:

[0925] The user uploads image files of presentation materials and presentation audio data from a dedicated interface.

[0926] Step 2:

[0927] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[0928] Step 3:

[0929] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[0930] Step 4:

[0931] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[0932] Step 5:

[0933] The image analysis module extracts the text within the image file and returns the text data to the server.

[0934] Step 6:

[0935] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[0936] Step 7:

[0937] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[0938] Step 8:

[0939] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[0940] Step 9:

[0941] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[0942] Step 10:

[0943] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[0944] Step 11:

[0945] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[0946] Step 12:

[0947] The server passes the user's facial image contained in the image file to the image emotion analysis means, which analyzes the emotion.

[0948] Step 13:

[0949] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[0950] Step 14:

[0951] The server comprehensively analyzes the acquired voice emotion data and image emotion data and passes it to the natural language processing module.

[0952] Step 15:

[0953] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[0954] Step 16:

[0955] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[0956] Step 17:

[0957] Based on the analysis results and emotional data obtained by the server from the natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[0958] Step 18:

[0959] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[0960] Step 19:

[0961] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[0962] Step 20:

[0963] The server uses a personalized learning module to generate feedback optimized for a particular reviewer by referencing past feedback data.

[0964] Step 21:

[0965] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[0966] Step 22:

[0967] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[0968] Step 23:

[0969] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[0970] Example 2

[0971] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0972] Conventional presentation analysis systems provide limited feedback based on the analysis of image files and audio data, and are insufficient in individual optimization that takes into account the user's emotions. Furthermore, improvements are needed in the accuracy of the analysis results and the specific content of the feedback. Therefore, a system is needed that provides detailed and specific feedback to help users deliver effective presentations and generates feedback that corresponds to the user's emotional state.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload an image file of a presentation material, a means for a user to upload presentation audio data, a means for a terminal to transmit a file designated by the user to the server, a means for the server to save the received file in a temporary storage directory, a means for the server to read the image file from the temporary storage directory and pass it to an image analysis module, a means for the image analysis module to extract text from the image file and return it to the server, a means for the server to format the acquired text data and save it in a data store for the natural language processing module, a means for the server to read an audio file from the temporary storage directory and pass it to the audio analysis module, a means for the audio analysis module to convert audio data into text and return it to the server, and a means for the server to convert the acquired audio text data into text. The system includes a means for formatting the data and analysis results and saving them in a data store, a means for the server to pass text data obtained from the image and audio analyses to a natural language processing module, a means for the natural language processing module to analyze the text data and detect redundancy, unclear expressions, and logical inconsistencies, a means for the server to generate feedback including specific improvements and suggestions based on the analysis results, a means for the server to pass audio data to a voice emotion analysis means and analyze the user's emotions, a means for the voice emotion analysis means to extract emotions from the audio data and return this to the server, a means for the server to pass facial images in image files to an image emotion analysis means and analyze them, a means for the image emotion analysis means to extract emotions from the facial images and return this to the server, and a means for the server to comprehensively analyze the audio emotion data and image emotion data and reflect this in feedback. This makes it possible to provide individually optimized, detailed feedback that takes into account the emotional state of the user based on the analysis results of the user's presentation materials and presentation audio data.

[0974] "User" refers to a person who uploads image files of presentation materials and presentation audio data through the system interface.

[0975] "Terminal" refers to a device that reads a file specified by a user and sends it to a server.

[0976] A "server" refers to a computer system that stores files received from users in a temporary storage directory and performs various processes such as image analysis and audio analysis.

[0977] The "temporary storage directory" refers to a directory for temporarily storing files received by the server.

[0978] "Image analysis module" refers to software or hardware for extracting text from image files.

[0979] "Speech Analysis Module" refers to software or hardware for converting speech data into text and extracting the text data and speech characteristics.

[0980] A "natural language processing module" refers to software that analyzes text data and detects redundancies, unclear expressions, and logical inconsistencies.

[0981] "Voice emotion analysis means" refers to software or hardware for extracting a user's emotion from voice data.

[0982] "Image emotion analysis means" refers to software or hardware for extracting a user's emotion from a facial image.

[0983] "Natural language processing technology" refers to all technologies for analyzing text data and understanding its meaning and structure.

[0984] "Data store" refers to a database or storage system for storing text data and voice characteristic data obtained through analysis.

[0985] "Weighting algorithm" refers to a computational method for adjusting analysis results based on specific review destinations or presentation content.

[0986] "Personalized Learning Module" refers to software that references past feedback data and optimizes feedback for specific reviewers.

[0987] This invention relates to a system that analyzes image files and presentation audio data for presentation materials and provides feedback. It also has a function to generate feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions. The main components of this system include a data receiving means, an image analysis means, an audio analysis means, a natural language processing means, a feedback generation means, a weighting means, an individual learning means, an emotion engine (a voice emotion analysis means and an image emotion analysis means), and a feedback display means.

[0988] The user uploads image files of presentation materials and audio data for the presentation through a dedicated interface. The device reads the files specified by the user and sends them to the server. The server saves the received files in a temporary storage directory. At this time, a timestamp and unique ID are added to the file name to ensure the uniqueness and security of the file.

[0989] The server reads the image files from the temporary storage directory and passes them to the image analysis module, which extracts the text from the image files and returns the text data to the server, which formats the text data and stores it in a data store for the natural language processing module. This extracts the text information from the presentation slides and prepares them for the next processing step.

[0990] For speech analysis, the server reads the audio file from the temporary storage directory and passes it to the speech analysis module. The speech analysis module converts the audio data into text and returns the text data and speech characteristics to the server. The server formats the acquired speech text data and analysis results and saves them in a data store for the natural language processing module. This allows the presentation's speaking style and content to be extracted as text data.

[0991] Next, in the natural language processing and feedback generation step, the server passes the text data obtained from image and audio analysis to a natural language processing module. The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc. Based on the analysis results obtained from the natural language processing module, the server generates written feedback including specific points for improvement and suggestions.

[0992] Furthermore, in emotion engine processing, the server passes voice data to voice emotion analysis means, which analyzes the user's emotion. The voice emotion analysis means extracts emotion from the voice data and returns this emotion data to the server. The server also passes the user's facial image contained in the image file to image emotion analysis means, which analyzes the emotion. The image emotion analysis means extracts emotion from the facial image and returns this emotion data to the server. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects the results in feedback.

[0993] In the weighting and personalized optimization step, the user inputs the specific reviewer (e.g., president, manager, client) and presentation content (proposal, apology, etc.). The server applies a weighting algorithm based on the received information and adjusts the analysis results. The server then uses a personalized learning module to generate feedback optimized for the specific reviewer by referencing past feedback data.

[0994] Below is an example prompt:

[0995] Proposal presentation example

[0996] I have uploaded the following data in order to make a presentation proposing a new product.

[0997] Slide image: New product proposal slide

[0998] Audio data: Presentation audio

[0999] We ask for feedback on how to make the proposal clear and how to reflect any nervousness you may have.

[1000] Apology Presentation Example

[1001] The following data has been uploaded in order to make an apology presentation for the project delay.

[1002] Slide image: Apology slide

[1003] Audio data: Presentation audio

[1004] Point out inappropriate language or confusing parts and ask for feedback that includes encouraging words that reflect your disappointment.

[1005] This system provides specific and detailed feedback to help users give effective presentations, and enables support according to the user's emotional state.

[1006] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1007] Step 1:

[1008] The user uses a dedicated interface to select image files of presentation materials and presentation audio data.

[1009] Input: User-selected image and audio files

[1010] Output: Path of the selected file

[1011] Specific operation: The user opens a browser or a dedicated app, clicks the "Select files" button, selects the image and audio files to upload, and clicks the "Upload" button.

[1012] Step 2:

[1013] The terminal reads the image file and audio file specified by the user and transmits them to the server.

[1014] Input: Paths to image and audio files

[1015] Output: Image and audio files sent to the server

[1016] Specific operation: The device loads the selected file into memory and sends the file data to the server using an HTTP POST request. After sending, the device notifies the user that the file upload is complete.

[1017] Step 3:

[1018] The server saves the received file in a temporary directory.

[1019] Input: Image and audio files sent from the device

[1020] Output: File saved in temporary directory

[1021] Specific operation: The server saves the received file in a temporary directory. When saving, a timestamp and a unique ID are added to the file name to ensure the uniqueness and security of the file.

[1022] Step 4:

[1023] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1024] Input: Image file saved in temporary directory

[1025] Output: Image data passed to the image analysis module

[1026] Specific operation: The server reads the target image file from the temporary storage directory into memory and passes the file data to the API endpoint of the image analysis module.

[1027] Step 5:

[1028] The image analysis module extracts the text within the image file and returns the text data to the server.

[1029] Input: Image data passed to the image analysis module

[1030] Output: Text data extracted from the image

[1031] Specific operation: The image analysis module uses OCR technology (e.g., Tesseract) to extract text from the image and returns the extracted text data to the server in JSON format.

[1032] Step 6:

[1033] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[1034] Input: Text data returned from the image analysis module

[1035] Output: Formatted text data

[1036] Specific operation: The server converts the text data it receives into a specific format (e.g., separates each sentence) and saves the formatted data in a data store (e.g., MongoDB).

[1037] Step 7:

[1038] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1039] Input: Audio file saved in temporary directory

[1040] Output: Audio data passed to the audio analysis module

[1041] Specific operation: The server reads the target audio file from the temporary storage directory into memory and passes the file data to the API endpoint of the audio analysis module.

[1042] Step 8:

[1043] The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[1044] Input: Audio data passed to the audio analysis module

[1045] Output: Text data and speech characteristics extracted from the audio

[1046] Specific operation: The speech analysis module converts speech into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and returns the text data and speech characteristics (e.g., voice tone, speed, volume) to the server in JSON format.

[1047] Step 9:

[1048] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1049] Input: Speech text data and speech characteristics data returned from the speech analysis module

[1050] Output: Formatted speech text data and speech characteristics data

[1051] Specific operation: The server converts the acquired voice text data into a specific format (e.g., separates it into sentences) and saves the data and voice characteristic data in a data store.

[1052] Step 10:

[1053] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1054] Input: Formatted image and audio text data

[1055] Output: Text data passed to the natural language processing module

[1056] Specific operation: The server reads the text data from the image analysis and voice analysis from the data store and sends the data to the API endpoint of the natural language processing module.

[1057] Step 11:

[1058] A natural language processing module analyzes text data to detect redundancies, unclear expressions, and logical inconsistencies.

[1059] Input: Text data passed to the natural language processing module

[1060] Output: Feedback data for the parsed text data

[1061] Specific operation: The natural language processing module uses a generative AI model (e.g., GPT-3) to analyze the text data, detecting redundancies, unclear expressions, logical inconsistencies, etc., and returns the analysis results to the server in JSON format.

[1062] Step 12:

[1063] Based on the analysis results, the server generates feedback including specific areas for improvement and suggestions.

[1064] Input: Analysis results returned from the natural language processing module

[1065] Output: Feedback document with specific improvements and suggestions

[1066] Specific actions: The server compiles the analysis results and generates a feedback document containing specific improvements and suggestions.

[1067] Step 13:

[1068] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[1069] Input: Audio file saved in temporary directory

[1070] Output: Emotion data based on audio data

[1071] Specific operation: The server passes the voice data to the API of the voice emotion analysis module, determines the emotion based on the voice characteristics (e.g., tone, intensity, speed) and returns the emotion data.

[1072] Step 14:

[1073] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[1074] Input: Audio data passed to the audio emotion analysis module

[1075] Output: Extracted emotion data

[1076] Specific operation: The voice emotion analysis module analyzes the voice data, extracts emotions (e.g., tension, anxiety, joy), and returns the results to the server in JSON format.

[1077] Step 15:

[1078] The server passes the facial image in the image file to the image emotion analysis means, and analyzes the emotion.

[1079] Input: Face image saved in temporary directory

[1080] Output: Emotion data based on facial images

[1081] Specific operation: The server passes the user's facial image in the image file to the API of the image emotion analysis module, and determines the emotion based on facial expression characteristics (e.g., smile, eyebrow movement).

[1082] Step 16:

[1083] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[1084] Input: A face image passed to the image emotion analysis module

[1085] Output: Extracted emotion data

[1086] Specific operation: The image emotion analysis module analyzes the facial image, extracts emotions (e.g., happiness, surprise, sadness), and returns the results to the server in JSON format.

[1087] Step 17:

[1088] The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[1089] Input: Extracted speech and image emotion data

[1090] Output: Feedback text reflecting the sentiment data

[1091] Specific operation: The server integrates emotional data obtained from audio and images and adds feedback according to the user's emotional state (e.g., how to relax if tense, words of encouragement if depressed).

[1092] As described above, the program of this system includes a process for multifaceted analysis of presentation materials and audio data uploaded by the user and generating detailed feedback that takes the user's emotions into consideration.

[1093] (Application example 2)

[1094] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1095] Conventional presentation feedback systems simply analyze text and voice, and do not provide detailed feedback tailored to the user's emotional state or the intended recipient. As a result, they may not receive optimal instruction to improve their presentation skills. In the training of logistics center staff, improving presentation skills contributes to improved work efficiency, so a system that provides more effective and individually optimized feedback is needed.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for performing emotion analysis and generating feedback according to the user's emotional state, and means for generating information for training logistics center staff and improving their presentation skills. This enables logistics center staff to improve their presentation skills and receive optimized feedback according to their emotional state.

[1097] "Presentation materials" refers to image files such as slides and documents used in a presentation.

[1098] "Presentation audio data" refers to digital data of the audio spoken by the presenter during a presentation.

[1099] "Image analysis" refers to the technique of extracting text and important visual information from image files.

[1100] "Voice analysis" refers to the technology of converting voice data into text and analyzing speaking style, emphasis, etc.

[1101] "Feedback" refers to specific advice or suggestions regarding the quality of the presentation and areas for improvement.

[1102] "Emotion analysis" refers to the technology of detecting and analyzing a user's emotional state from their voice and visual data.

[1103] A "logistics center" refers to a facility that stores, sorts, and ships goods and materials.

[1104] "Education" refers to the guidance and training of logistics center staff aimed at improving work efficiency and presentation skills.

[1105] "Natural language processing technology" refers to artificial intelligence technology for analyzing text data, understanding its meaning and structure, and processing it.

[1106] "Personalized learning" refers to a technology that uses past data to learn in order to provide optimal feedback for specific users and situations.

[1107] "Weighting" refers to the practice of prioritizing and adjusting analysis results based on specific review destinations or presentation content.

[1108] "Server" refers to a computer system that receives, stores, analyzes data, and generates feedback.

[1109] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[1110] The system for implementing this invention receives image files of presentation materials and presentation audio data, analyzes them, and generates and displays feedback. The system program and its processing flow are described below.

[1111] System Overview

[1112] The system consists of the following main components:

[1113] A means of receiving image files of presentation materials

[1114] A means of receiving presentation audio data

[1115] means for analyzing the image file and the audio data, respectively;

[1116] means for generating feedback based on the analysis results;

[1117] A means of performing emotion analysis and generating feedback based on the user's emotional state

[1118] A means of providing information to train logistics center staff and improve their presentation skills

[1119] Hardware and Software Configuration

[1120] Hardware:

[1121] Smartphone (iOS / Android)

[1122] Server (AWS, Google Cloud GCP, etc.)

[1123] Head-mounted displays (Microsoft HoloLens, Google Glass)

[1124] software:

[1125] Image processing library (OpenCV)

[1126] Speech analysis library (Google Cloud Speech-to-Text API)

[1127] Natural Language Processing (spaCy, NLTK)

[1128] Sentiment analysis library (TensorFlow, Keras)

[1129] Processing flow explanation

[1130] 1. Data Receipt and Storage:

[1131] The user uploads image files and audio data of presentation materials via a smartphone application.

[1132] The device reads the data and sends it to the server.

[1133] The server receives these files and stores them in a secure temporary storage directory.

[1134] 2. Image Analysis:

[1135] The server loads the image file from the temporary storage directory.

[1136] Image analysis module (OpenCV) extracts text and important visual information from image files.

[1137] 3. Audio analysis:

[1138] The server passes the audio file to a speech analysis module (Google Cloud Speech-to-Text API) and converts it into text.

[1139] The speech analysis module analyzes speaking style and emphasis, and generates text data.

[1140] 4. Natural Language Processing and Feedback Generation:

[1141] The server passes the text data obtained from image analysis and voice analysis to a natural language processing module (spaCy).

[1142] The natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[1143] 5. Emotion analysis:

[1144] The server passes the audio data and image files to an emotion analysis module (TensorFlow, Keras) to analyze the user's emotional state.

[1145] Obtain emotional data and reflect it in feedback.

[1146] Specific examples

[1147] Presentation support with smart trainers

[1148] Users upload slides about new logistics solutions and audio data for presentations based on them.

[1149] The server analyzes the images and audio and generates text data about the content of the slides.

[1150] The natural language processing module detects redundant expressions and logical inconsistencies and generates feedback suggesting specific improvements.

[1151] The sentiment analysis module detects nervousness or lack of confidence during a presentation and adds encouraging words or advice as feedback.

[1152] Prompt Sentence Examples

[1153] "Please advise the user on improvements based on the slide presentation and audio data."

[1154] "Perform emotional analysis during presentations and offer advice on how to relax if people are nervous."

[1155] As described above, this system helps logistics center staff improve their presentation skills by analyzing presentation materials and audio data, and provides optimal feedback according to their emotional state.

[1156] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1157] Specific processing steps of the program

[1158] Step 1:

[1159] Data Receipt and Storage

[1160] Input: Image files of presentation materials and audio data of presentations

[1161] How it works: A user uploads image files and audio data for a presentation using a smartphone application.

[1162] Output: Image files and audio data are sent to the server and stored in a secure temporary directory.

[1163] Step 2:

[1164] Loading and analyzing image files

[1165] Input: Image file saved in temporary directory

[1166] Operation: The server passes the image file to the image analysis module (OpenCV).

[1167] Data processing / calculation: The image analysis module extracts text and important visual information from the image files and returns it to the server.

[1168] Output: Extracted text data

[1169] Step 3:

[1170] Loading and analyzing audio files

[1171] Input: Audio file saved in temporary directory

[1172] How it works: The server passes the audio file to the speech analysis module (Google Cloud Speech-to-Text API).

[1173] Data processing / calculation: The speech analysis module converts the speech data into text, analyzes the speech characteristics (emphasis, speaking style, etc.), and returns the text to the server.

[1174] Output: Converted text data and voice characteristics data

[1175] Step 4:

[1176] Natural Language Processing and Feedback Generation

[1177] Input: Extracted text data and speech characteristics data

[1178] How it works: The server passes this data to the natural language processing module (spaCy).

[1179] Data processing / calculation: A natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[1180] Output: Feedback text with improvements

[1181] Step 5:

[1182] Emotion analysis

[1183] Input: Audio data and image files

[1184] How it works: The server passes the audio data and image files to the emotion analysis module (TensorFlow, Keras).

[1185] Data processing / calculation: The emotion analysis module analyzes the user's emotional state (e.g., tension, relief, excitement, etc.) based on their voice and image, and returns the result to the server.

[1186] Output: Emotional state data

[1187] Step 6:

[1188] Consolidating and providing final feedback

[1189] Input: Feedback text and emotional state data

[1190] Operation: The server integrates the emotional state data with the feedback text to generate the final feedback.

[1191] Output: Consolidated feedback text

[1192] What it does: The server displays the consolidated feedback to the user.

[1193] The above are the specific processing steps of this system. Through the data processing and calculations performed at each step, the user can receive detailed feedback based on the presentation materials and audio data, and that is tailored to the user's emotional state.

[1194] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1195] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1196] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1197] [Third embodiment]

[1198] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1199] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1200] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1201] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1202] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1203] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1204] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1205] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1206] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1207] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1208] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1209] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1210] This invention relates to a system that analyzes image files and audio data of presentation materials and provides feedback. The program processing of this system is explained in detail below in natural language.

[1211] System Overview

[1212] Users upload image files to be used in their presentations and audio data recording the presentation to the system. The server receives these files and performs image and audio analysis. The analyzed text data is further analyzed using natural language processing technology to generate specific feedback based on the presentation's content and speaking style. The feedback is weighted based on the specific reviewer and presentation content and provided to the user.

[1213] Program processing

[1214] Receiving and storing data

[1215] 1. After the user logs in, they upload the image files of the presentation materials and the presentation audio data through a dedicated interface.

[1216] 2. The terminal reads the file specified by the user and sends it to the server.

[1217] 3. The server receives these files and stores them in a secure temporary storage directory.

[1218] Image analysis

[1219] 1. The server loads the image file from the temporary storage directory.

[1220] 2. The server starts the image analysis module and performs processing to convert the image file into text data.

[1221] 3. The image analysis module extracts the text in the image file and returns the data to the server.

[1222] 4. The server formats the retrieved text data and prepares it for passing to the natural language processing module.

[1223] Audio analysis

[1224] 1. The server loads the audio file from the temporary storage directory.

[1225] 2. The server launches the speech analysis module and processes the speech data to convert it into text.

[1226] 3. The speech analysis module converts the speech into text data and returns the data to the server.

[1227] 4. The server formats the acquired speech-to-text data and prepares it for passing to the natural language processing module.

[1228] Natural Language Processing and Feedback Generation

[1229] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[1230] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[1231] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[1232] Weighting and Individual Optimization

[1233] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[1234] 2. The server applies a weighting algorithm based on the received information to adjust the analysis results.

[1235] 3. The server uses a personalized learning module to look at past feedback data and generate feedback optimized for the specific reviewer.

[1236] Specific examples

[1237] Example 1: Proposal presentation

[1238] The user uploads slide images and audio data for new product proposals.

[1239] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[1240] The server generates specific feedback based on the analysis, such as identifying redundancies in the proposal or adding specific use cases to attract the client's interest.

[1241] Improve your presentation materials and delivery based on the feedback provided by users.

[1242] Example 2: Apology presentation

[1243] A user uploads slide images and audio data apologizing for the project delay.

[1244] The server performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[1245] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[1246] Improve your apology presentation based on user-provided feedback.

[1247] This concludes the description of the "Mode for Carrying Out the Invention" of the present invention. This system provides specific and detailed feedback to help users give effective presentations.

[1248] The processing flow will be explained below.

[1249] Step 1:

[1250] Users upload image files of presentation materials and presentation audio data through a dedicated interface.

[1251] Step 2:

[1252] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[1253] Step 3:

[1254] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[1255] Step 4:

[1256] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1257] Step 5:

[1258] The image analysis module extracts the text within the image file and returns the text data to the server.

[1259] Step 6:

[1260] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[1261] Step 7:

[1262] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1263] Step 8:

[1264] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[1265] Step 9:

[1266] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1267] Step 10:

[1268] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1269] Step 11:

[1270] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[1271] Step 12:

[1272] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[1273] Step 13:

[1274] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[1275] Step 14:

[1276] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[1277] Step 15:

[1278] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[1279] Step 16:

[1280] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[1281] Step 17:

[1282] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[1283] Step 18:

[1284] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[1285] Example 1

[1286] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1287] Conventional presentation feedback systems mainly analyze image files and audio data individually, which has the drawback of insufficient comprehensive analysis and specific feedback to users. Furthermore, since it is not possible to weight or optimize based on specific reviewers or presentation content, it is difficult to obtain the specific and accurate feedback that users need.

[1288] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1289] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for analyzing the analyzed text data using natural language processing technology, means for generating feedback based on the analysis results, means for displaying the feedback, means for the user to input specific review recipients and presentation content, means for weighting the feedback based on the input information, and means for optimizing the feedback using past feedback data. This makes it possible to comprehensively analyze the presentation materials and audio data and provide specific and accurate feedback based on the specific review recipients and presentation content.

[1290] "Presentation materials" are a collection of images and slides that visually represent the contents of a presentation.

[1291] "Image files" refer to still image data that has been digitized from presentation materials.

[1292] "Presentation audio data" refers to audio data recorded from a presentation.

[1293] "Server" refers to the central computer system used for analysis, data processing, and feedback generation.

[1294] A "terminal" refers to a computer or device operated by a user, and is responsible for exchanging data with a server.

[1295] "User" refers to a person who uses the system to upload presentation data and receive feedback.

[1296] "Analysis means" refers to the function of converting image files and audio data into text data and analyzing it using natural language processing technology.

[1297] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[1298] "Feedback" refers to information, including suggestions for improvement and recommendations, provided to users based on analyzed data.

[1299] "Weighting" refers to the process of adjusting the analysis results based on the specific review destination or presentation content.

[1300] "Optimization" refers to using past feedback data to make adjustments to improve the accuracy and usefulness of feedback.

[1301] The present invention relates to a system for analyzing image files and audio data of presentation materials and providing feedback to users. An embodiment of this system will be described in detail below.

[1302] System Overview

[1303] The system of the present invention allows users to upload image files used as presentation materials and audio data for the presentation, and then analyzes the materials and audio and provides feedback. Specifically, the server receives and analyzes these files, and uses natural language processing technology to generate feedback appropriate for the user.

[1304] Hardware and software used

[1305] Server: A central computer system that analyzes and processes data and generates feedback. The server is equipped with image analysis modules, audio analysis modules, and natural language processing modules.

[1306] Device: A computer or device operated by a user to upload files or display feedback.

[1307] Image Analysis Module: Analyzes image files using image analysis libraries such as Python, OpenCV, and Tesseract OCR.

[1308] Speech analysis module: Analyzes speech data using speech analysis APIs such as Google Speech-to-Text and IBM Watson Speech to Text.

[1309] Natural Language Processing module: Analyzes and processes text data using the NLTK library and generative AI models.

[1310] Data processing and calculation

[1311] When a user uploads image files and audio data for a presentation, the device sends these files to the server. The server receives these files and stores them in a secure temporary directory. The server then launches image and audio analysis modules to convert the image files into text data and the audio data into text.

[1312] The server passes the acquired text data to a natural language processing module to detect redundancies, unclear expressions, and logical inconsistencies. The server generates feedback based on the analysis results and provides it to the user.

[1313] Additionally, when a user inputs a specific review destination or presentation content (e.g., proposal, apology), the server applies a weighting algorithm and references past feedback data to optimize the feedback.

[1314] Specific examples

[1315] Proposal presentation example

[1316] User: Uploads slide images and audio data for a new product proposal presentation to the system.

[1317] Server: Performs image and audio analysis to analyze the content of slides and speaking style.

[1318] Server: Generates specific feedback based on the analysis, such as redundancies in the proposal or adding specific use cases to capture the client's interest.

[1319] Users: Improve their presentation materials and delivery based on the feedback provided.

[1320] Example of an apology presentation

[1321] User: Upload slide images and audio data to the system apologizing for the project delay.

[1322] Server: Performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[1323] Server: Based on the analysis results, provide feedback on specific improvement measures and how to choose the right words to use in your apology.

[1324] User: Improve your apology presentation based on the feedback provided.

[1325] This is the embodiment of the present invention. This system allows users to receive specific and detailed feedback on how to give effective presentations.

[1326] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1327] Step 1:

[1328] A user logs in.

[1329] Input: User ID and password.

[1330] How it works: A user accesses the system's login page and enters their user ID and password.

[1331] Output: If authentication is successful, the user can access the main interface of the system.

[1332] Step 2:

[1333] The user uploads image files of the presentation materials and audio data of the presentation.

[1334] Input: Image files and audio data files.

[1335] How it works: The user selects image and audio files in a dedicated interface and clicks the upload button.

[1336] Output: The selected file is prepared by the terminal.

[1337] Step 3:

[1338] The terminal reads the file specified by the user and sends it to the server.

[1339] Input: Image files and audio data files.

[1340] How it works: The device gets the path of the selected file and sends it to the server using an HTTP POST request.

[1341] Output: The file data sent to the server.

[1342] Step 4:

[1343] The server receives the file and stores it in a secure temporary storage directory.

[1344] Input: Submitted image and audio data files.

[1345] How it works: The server saves the received file in a temporary directory, such as the / tmp directory.

[1346] Output: File saved in temporary directory.

[1347] Step 5:

[1348] The server loads the image file from the temporary storage directory.

[1349] Input: The path to the saved image file.

[1350] What it does: The server loads the image file from the temporary storage directory into memory.

[1351] Output: The loaded image data.

[1352] Step 6:

[1353] The server starts the image analysis module and converts the image file into text data.

[1354] Input: The loaded image data.

[1355] How it works: The server uses image analysis libraries such as OpenCV and Tesseract OCR to extract text from images.

[1356] Output: The extracted text data.

[1357] Step 7:

[1358] The server reads the audio file from the temporary storage directory.

[1359] Input: The path to the saved audio file.

[1360] How it works: The server loads the audio file from the temporary storage directory into memory.

[1361] Output: The loaded audio data.

[1362] Step 8:

[1363] The server starts a voice analysis module and converts the voice data into text data.

[1364] Input: The loaded audio data.

[1365] How it works: The server uses a speech analysis API such as Google Speech-to-Text or IBM Watson Speech to Text to convert the audio data into text.

[1366] Output: The converted speech-to-text data.

[1367] Step 9:

[1368] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1369] Input: Extracted image-text data and audio-text data.

[1370] How it works: The server sends text data to the natural language processing module via an API.

[1371] Output: Text data sent to the natural language processing module.

[1372] Step 10:

[1373] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[1374] Input: The text data sent.

[1375] How it works: The natural language processing module uses the NLTK library and generative AI models to analyze the quality of text.

[1376] Output: Analysis result data.

[1377] Step 11:

[1378] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[1379] Input: Analysis result data.

[1380] How it works: The server uses templates based on the analysis results to summarize specific improvements and suggestions in natural language.

[1381] Output: The generated feedback sentence.

[1382] Step 12:

[1383] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[1384] Input: Specific reviewer and presentation content.

[1385] How it works: The user selects and inputs the review recipient and presentation purpose in the interface.

[1386] Output: The input review destination and presentation content data.

[1387] Step 13:

[1388] Based on the information received, the server applies a weighting algorithm to adjust the analysis results.

[1389] Input: Review destination and presentation content data, analysis result data.

[1390] How it works: The server applies weighting parameters appropriate to the reviewer and presentation content to adjust the feedback.

[1391] Output: Weighted analysis result data.

[1392] Step 14:

[1393] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[1394] Input: Weighted analysis result data, past feedback data.

[1395] How it works: The server queries a database of past feedback and optimizes the feedback based on feedback performance in similar situations.

[1396] Output: Optimized feedback text.

[1397] The above is the specific flow of program processing in this system.

[1398] (Application example 1)

[1399] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1400] Conventional presentation training systems simply require users to upload presentation materials and audio data, which the system then analyzes and provides feedback. This makes it difficult to provide feedback optimized for specific environments (e.g., in a factory) or situations (e.g., safety measures presentations). They also lack the ability to detect redundant sentences or unclear expressions in real time and immediately suggest specific areas for improvement. This leaves users facing specific challenges in conducting effective presentation training.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1402] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for displaying the feedback, and means for coordinating with a robot that performs presentation training in a factory. This allows users to receive feedback in real time that is optimized for specific environments and situations in the factory, enabling more effective presentation training.

[1403] An "image file of presentation materials" is a file in which information such as slides and charts used in a presentation is saved in image format.

[1404] "Presentation audio data" refers to data recorded in audio format of what was said during a presentation.

[1405] "Means for analysis" refers to the technology or device that converts received image files or audio data into text information and performs analysis.

[1406] "Means for generating feedback" refers to technology or devices that generate suggestions for improvement or comments to be provided to users based on the content of the presentation and the results of data analysis.

[1407] "Means for displaying feedback" refers to techniques or devices for visually or audibly presenting the generated feedback to the user.

[1408] A "robot for in-factory presentation training" is an autonomous or remotely controlled mechanical device that supports presentation training in a factory and provides feedback.

[1409] A "weighting means" is a technique or device that adjusts the evaluation or results based on the analyzed text data according to specific criteria.

[1410] "Means for optimizing feedback through individual learning" refers to technology or devices that generate feedback optimized for each user based on past analysis results and feedback data.

[1411] "Means for converting voice data to text" refers to technology or devices that use voice recognition technology to convert the spoken content in an audio file into text form.

[1412] "Means for analyzing using natural language processing technology" refers to technology or devices that perform semantic analysis and error checking of text data converted using natural language processing technology.

[1413] "Means for detecting redundant sentences and unclear expressions" refers to technology or devices that identify unimportant long sentences and difficult-to-understand expressions from the analyzed text data and point them out to the user.

[1414] This invention relates to a system that analyzes image files of presentation materials and audio data of presentations and provides feedback. This system is intended to support presentation training, particularly in factories, and to provide users with optimized feedback.

[1415] System configuration and program processing

[1416] Receiving and storing data

[1417] The server receives and temporarily stores image files and audio data for presentation materials uploaded by users via the interface of the robots in the factory. The server stores the received files in a secure temporary storage directory and prepares them for analysis.

[1418] Image analysis

[1419] The server uses PIL (Python Imaging Library) and pytesseract libraries to convert image files to text, extracting the text data within the image files, which is then formatted by the server and prepared for natural language processing.

[1420] Audio analysis

[1421] The server uses the speech_recognition library to convert the audio file to text, which is then formatted and prepared for natural language processing as well.

[1422] Natural Language Processing and Feedback Generation

[1423] The server analyzes the text data obtained from image and audio analysis using natural language processing technology. This process uses the spacy library. During the analysis, redundant sentences and unclear expressions are identified in the text data, and specific feedback on improvements is generated.

[1424] Weighting and Individual Optimization

[1425] The server weights specific review destinations and presentation content based on the analyzed text data, and also provides optimized feedback for each user by referencing past feedback data through personalized learning.

[1426] Specific examples

[1427] Example 1: Proposal presentation

[1428] When a user gives a presentation to propose a new product in a factory, they upload slide images and audio data. The server analyzes these files and generates specific feedback such as advice like, "The text on this slide is redundant. Please explain it more succinctly."

[1429] Example 2: Safety Presentation

[1430] When giving a presentation on safety measures in a factory, the user uploads the materials and audio data. The server analyzes the data and provides feedback such as, "The explanation of safety precautions tends to be redundant. In particular, make the section on 'checking safety devices' more concise."

[1431] Example prompt sentence:

[1432] "The text on this slide is redundant. Please explain it more succinctly."

[1433] "Safety precautions can be redundant. Please be more concise, especially the 'check safety equipment' section."

[1434] This invention allows users to receive real-time feedback optimized for specific environments and situations within a factory, enabling more effective presentation training. This system is an important tool for improving the effectiveness of presentations in specific situations.

[1435] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1436] Step 1:

[1437] A user logs in and uploads image files and audio data for presentation materials. The user uses the robot interface in the factory to select these files and send them to the server. The input is the image files and audio files, and the output is the server receiving the files.

[1438] Step 2:

[1439] After the server receives the uploaded files, it stores them in a secure temporary directory. The input is the image and audio files uploaded by the user, and the output is that these files are stored in the temporary directory.

[1440] Step 3:

[1441] The server reads the saved image file and converts it to text data using image analysis modules (PIL and pytesseract). The input is the image file saved in the temporary storage directory, and the output is the converted text data. This text data is returned to the server.

[1442] Step 4:

[1443] The server reads the saved audio file and converts it into text using a speech analysis module (speech_recognition library). The input is the audio file saved in the temporary storage directory, and the output is the converted text. This text data is also returned to the server.

[1444] Step 5:

[1445] The server passes the text data obtained from image and audio analysis to a natural language processing module (the spacy library) for analysis. The input is the converted text data, and the output is the analysis results. Based on these analysis results, redundant sentences and unclear expressions are detected.

[1446] Step 6:

[1447] The server generates specific feedback based on the analysis results of the natural language processing. The input is the analysis results from the natural language processing module, and the output is the generated feedback. This feedback includes specific advice on redundant sentences and areas for improvement.

[1448] Step 7:

[1449] The server weights the generated feedback based on the specific reviewer and presentation content. The input is feedback and reviewer information, and the output is weighted feedback.

[1450] Step 8:

[1451] The server uses an individual learning module to generate optimized feedback by referring to past feedback data, with the input being the past feedback data and the current feedback, and the output being the optimized feedback.

[1452] Step 9:

[1453] The server visually displays the generated feedback to the user: the input is the optimized feedback, and the output is the user seeing the feedback on the screen.

[1454] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1455] overview

[1456] This invention relates to a system that analyzes image files and audio data from presentation materials and provides feedback. It also has the function of generating feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions.

[1457] System configuration

[1458] The system consists of the following main components:

[1459] 1. Data Receiving Method

[1460] 2. Image analysis methods

[1461] 3. Audio analysis methods

[1462] 4. Natural Language Processing Methods

[1463] 5. Feedback Generation Methods

[1464] 6. Weighting Methods

[1465] 7. Personalized learning tools

[1466] 8. Emotion Engine (Voice Emotion Analysis and Image Emotion Analysis)

[1467] 9. Feedback Display Methods

[1468] Program processing

[1469] The program processing of this system will be specifically explained below.

[1470] Receiving and storing data

[1471] 1. The user uploads the image files of the presentation materials and the presentation audio data through a dedicated interface.

[1472] 2. The terminal reads the file specified by the user and sends it to the server.

[1473] 3. The server receives the file and stores it in a secure temporary storage directory.

[1474] Image analysis

[1475] 1. The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1476] 2. The image analysis module extracts the text from the image file and returns the text data to the server.

[1477] 3. The server formats the text data it receives and stores it in a data store for the natural language processing module.

[1478] Audio analysis

[1479] 1. The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1480] 2. The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[1481] 3. The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1482] Natural Language Processing and Feedback Generation

[1483] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[1484] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[1485] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[1486] Emotion engine processing

[1487] 1. The server passes the voice data to the voice emotion analysis means and analyzes the user's emotions.

[1488] 2. The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[1489] 3. The server passes the user's facial image contained in the image file to the image emotion analysis means, and analyzes the emotion.

[1490] 4. The image emotion analysis means extracts emotions from the facial image and returns the emotion data to the server.

[1491] 5. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[1492] Weighting and Individual Optimization

[1493] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[1494] 2. Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[1495] 3. The server uses a personalized learning module to generate feedback optimized for the specific reviewer, referencing past feedback data.

[1496] Specific examples

[1497] Example 1: Proposal presentation

[1498] The user uploads slide images and audio data for new product proposals.

[1499] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[1500] The server uses an emotion engine to analyze the user's emotions and provides feedback based on that. For example, if the user is nervous, the server adds advice to ease the tension.

[1501] Based on the analysis, the server generates specific feedback, such as redundancy in the proposal or adding use cases to attract the client's interest.

[1502] Improve your presentation materials and delivery based on the feedback provided by users.

[1503] Example 2: Apology presentation

[1504] A user uploads slide images and audio data apologizing for the project delay.

[1505] The server performs image and audio analysis to point out inappropriate comments and unclear improvement suggestions.

[1506] The server uses an emotion engine to analyze the user's emotions and provide feedback based on that. For example, if the user is feeling down, it might add words of encouragement or an appropriate apology.

[1507] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[1508] Improve your apology presentation based on user-provided feedback.

[1509] This concludes the "Mode for Carrying Out the Invention" of the present invention. This system not only provides specific and detailed feedback to help users give effective presentations, but also enables support based on the user's emotional state.

[1510] The processing flow will be explained below.

[1511] Step 1:

[1512] The user uploads image files of presentation materials and presentation audio data from a dedicated interface.

[1513] Step 2:

[1514] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[1515] Step 3:

[1516] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[1517] Step 4:

[1518] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1519] Step 5:

[1520] The image analysis module extracts the text within the image file and returns the text data to the server.

[1521] Step 6:

[1522] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[1523] Step 7:

[1524] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1525] Step 8:

[1526] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[1527] Step 9:

[1528] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1529] Step 10:

[1530] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[1531] Step 11:

[1532] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[1533] Step 12:

[1534] The server passes the user's facial image contained in the image file to the image emotion analysis means, which analyzes the emotion.

[1535] Step 13:

[1536] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[1537] Step 14:

[1538] The server comprehensively analyzes the acquired voice emotion data and image emotion data and passes it to the natural language processing module.

[1539] Step 15:

[1540] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1541] Step 16:

[1542] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[1543] Step 17:

[1544] Based on the analysis results and emotional data obtained by the server from the natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[1545] Step 18:

[1546] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[1547] Step 19:

[1548] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[1549] Step 20:

[1550] The server uses a personalized learning module to generate feedback optimized for a particular reviewer by referencing past feedback data.

[1551] Step 21:

[1552] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[1553] Step 22:

[1554] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[1555] Step 23:

[1556] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[1557] Example 2

[1558] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1559] Conventional presentation analysis systems provide limited feedback based on the analysis of image files and audio data, and are insufficient in individual optimization that takes into account the user's emotions. Furthermore, improvements are needed in the accuracy of the analysis results and the specific content of the feedback. Therefore, a system is needed that provides detailed and specific feedback to help users deliver effective presentations and generates feedback that corresponds to the user's emotional state.

[1560] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload an image file of a presentation material, a means for a user to upload presentation audio data, a means for a terminal to transmit a file designated by the user to the server, a means for the server to save the received file in a temporary storage directory, a means for the server to read the image file from the temporary storage directory and pass it to an image analysis module, a means for the image analysis module to extract text from the image file and return it to the server, a means for the server to format the acquired text data and save it in a data store for the natural language processing module, a means for the server to read an audio file from the temporary storage directory and pass it to the audio analysis module, a means for the audio analysis module to convert audio data into text and return it to the server, and a means for the server to convert the acquired audio text data into text. The system includes a means for formatting the data and analysis results and saving them in a data store, a means for the server to pass text data obtained from the image and audio analyses to a natural language processing module, a means for the natural language processing module to analyze the text data and detect redundancy, unclear expressions, and logical inconsistencies, a means for the server to generate feedback including specific improvements and suggestions based on the analysis results, a means for the server to pass audio data to a voice emotion analysis means and analyze the user's emotions, a means for the voice emotion analysis means to extract emotions from the audio data and return this to the server, a means for the server to pass facial images in image files to an image emotion analysis means and analyze them, a means for the image emotion analysis means to extract emotions from the facial images and return this to the server, and a means for the server to comprehensively analyze the audio emotion data and image emotion data and reflect this in feedback. This makes it possible to provide individually optimized, detailed feedback that takes into account the emotional state of the user based on the analysis results of the user's presentation materials and presentation audio data.

[1561] "User" refers to a person who uploads image files of presentation materials and presentation audio data through the system interface.

[1562] "Terminal" refers to a device that reads a file specified by a user and sends it to a server.

[1563] A "server" refers to a computer system that stores files received from users in a temporary storage directory and performs various processes such as image analysis and audio analysis.

[1564] The "temporary storage directory" refers to a directory for temporarily storing files received by the server.

[1565] "Image analysis module" refers to software or hardware for extracting text from image files.

[1566] "Speech Analysis Module" refers to software or hardware for converting speech data into text and extracting the text data and speech characteristics.

[1567] A "natural language processing module" refers to software that analyzes text data and detects redundancies, unclear expressions, and logical inconsistencies.

[1568] "Voice emotion analysis means" refers to software or hardware for extracting a user's emotion from voice data.

[1569] "Image emotion analysis means" refers to software or hardware for extracting a user's emotion from a facial image.

[1570] "Natural language processing technology" refers to all technologies for analyzing text data and understanding its meaning and structure.

[1571] "Data store" refers to a database or storage system for storing text data and voice characteristic data obtained through analysis.

[1572] "Weighting algorithm" refers to a computational method for adjusting analysis results based on specific review destinations or presentation content.

[1573] "Personalized Learning Module" refers to software that references past feedback data and optimizes feedback for specific reviewers.

[1574] This invention relates to a system that analyzes image files and presentation audio data for presentation materials and provides feedback. It also has a function to generate feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions. The main components of this system include a data receiving means, an image analysis means, an audio analysis means, a natural language processing means, a feedback generation means, a weighting means, an individual learning means, an emotion engine (a voice emotion analysis means and an image emotion analysis means), and a feedback display means.

[1575] The user uploads image files of presentation materials and audio data for the presentation through a dedicated interface. The device reads the files specified by the user and sends them to the server. The server saves the received files in a temporary storage directory. At this time, a timestamp and unique ID are added to the file name to ensure the uniqueness and security of the file.

[1576] The server reads the image files from the temporary storage directory and passes them to the image analysis module, which extracts the text from the image files and returns the text data to the server, which formats the text data and stores it in a data store for the natural language processing module. This extracts the text information from the presentation slides and prepares them for the next processing step.

[1577] For speech analysis, the server reads the audio file from the temporary storage directory and passes it to the speech analysis module. The speech analysis module converts the audio data into text and returns the text data and speech characteristics to the server. The server formats the acquired speech text data and analysis results and saves them in a data store for the natural language processing module. This allows the presentation's speaking style and content to be extracted as text data.

[1578] Next, in the natural language processing and feedback generation step, the server passes the text data obtained from image and audio analysis to a natural language processing module. The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc. Based on the analysis results obtained from the natural language processing module, the server generates written feedback including specific points for improvement and suggestions.

[1579] Furthermore, in emotion engine processing, the server passes voice data to voice emotion analysis means, which analyzes the user's emotion. The voice emotion analysis means extracts emotion from the voice data and returns this emotion data to the server. The server also passes the user's facial image contained in the image file to image emotion analysis means, which analyzes the emotion. The image emotion analysis means extracts emotion from the facial image and returns this emotion data to the server. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects the results in feedback.

[1580] In the weighting and personalized optimization step, the user inputs the specific reviewer (e.g., president, manager, client) and presentation content (proposal, apology, etc.). The server applies a weighting algorithm based on the received information and adjusts the analysis results. The server then uses a personalized learning module to generate feedback optimized for the specific reviewer by referencing past feedback data.

[1581] Below is an example prompt:

[1582] Proposal presentation example

[1583] I have uploaded the following data in order to make a presentation proposing a new product.

[1584] Slide image: New product proposal slide

[1585] Audio data: Presentation audio

[1586] We ask for feedback on how to make the proposal clear and how to reflect any nervousness you may have.

[1587] Apology Presentation Example

[1588] The following data has been uploaded in order to make an apology presentation for the project delay.

[1589] Slide image: Apology slide

[1590] Audio data: Presentation audio

[1591] Point out inappropriate language or confusing parts and ask for feedback that includes encouraging words that reflect your disappointment.

[1592] This system provides specific and detailed feedback to help users give effective presentations, and enables support according to the user's emotional state.

[1593] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1594] Step 1:

[1595] The user uses a dedicated interface to select image files of presentation materials and presentation audio data.

[1596] Input: User-selected image and audio files

[1597] Output: Path of the selected file

[1598] Specific operation: The user opens a browser or a dedicated app, clicks the "Select files" button, selects the image and audio files to upload, and clicks the "Upload" button.

[1599] Step 2:

[1600] The terminal reads the image file and audio file specified by the user and transmits them to the server.

[1601] Input: Paths to image and audio files

[1602] Output: Image and audio files sent to the server

[1603] Specific operation: The device loads the selected file into memory and sends the file data to the server using an HTTP POST request. After sending, the device notifies the user that the file upload is complete.

[1604] Step 3:

[1605] The server saves the received file in a temporary directory.

[1606] Input: Image and audio files sent from the device

[1607] Output: File saved in temporary directory

[1608] Specific operation: The server saves the received file in a temporary directory. When saving, a timestamp and a unique ID are added to the file name to ensure the uniqueness and security of the file.

[1609] Step 4:

[1610] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1611] Input: Image file saved in temporary directory

[1612] Output: Image data passed to the image analysis module

[1613] Specific operation: The server reads the target image file from the temporary storage directory into memory and passes the file data to the API endpoint of the image analysis module.

[1614] Step 5:

[1615] The image analysis module extracts the text within the image file and returns the text data to the server.

[1616] Input: Image data passed to the image analysis module

[1617] Output: Text data extracted from the image

[1618] Specific operation: The image analysis module uses OCR technology (e.g., Tesseract) to extract text from the image and returns the extracted text data to the server in JSON format.

[1619] Step 6:

[1620] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[1621] Input: Text data returned from the image analysis module

[1622] Output: Formatted text data

[1623] Specific operation: The server converts the text data it receives into a specific format (e.g., separates each sentence) and saves the formatted data in a data store (e.g., MongoDB).

[1624] Step 7:

[1625] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1626] Input: Audio file saved in temporary directory

[1627] Output: Audio data passed to the audio analysis module

[1628] Specific operation: The server reads the target audio file from the temporary storage directory into memory and passes the file data to the API endpoint of the audio analysis module.

[1629] Step 8:

[1630] The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[1631] Input: Audio data passed to the audio analysis module

[1632] Output: Text data and speech characteristics extracted from the audio

[1633] Specific operation: The speech analysis module converts speech into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and returns the text data and speech characteristics (e.g., voice tone, speed, volume) to the server in JSON format.

[1634] Step 9:

[1635] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1636] Input: Speech text data and speech characteristics data returned from the speech analysis module

[1637] Output: Formatted speech text data and speech characteristics data

[1638] Specific operation: The server converts the acquired voice text data into a specific format (e.g., separates it into sentences) and saves the data and voice characteristic data in a data store.

[1639] Step 10:

[1640] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1641] Input: Formatted image and audio text data

[1642] Output: Text data passed to the natural language processing module

[1643] Specific operation: The server reads the text data from the image analysis and voice analysis from the data store and sends the data to the API endpoint of the natural language processing module.

[1644] Step 11:

[1645] A natural language processing module analyzes text data to detect redundancies, unclear expressions, and logical inconsistencies.

[1646] Input: Text data passed to the natural language processing module

[1647] Output: Feedback data for the parsed text data

[1648] Specific operation: The natural language processing module uses a generative AI model (e.g., GPT-3) to analyze the text data, detecting redundancies, unclear expressions, logical inconsistencies, etc., and returns the analysis results to the server in JSON format.

[1649] Step 12:

[1650] Based on the analysis results, the server generates feedback including specific areas for improvement and suggestions.

[1651] Input: Analysis results returned from the natural language processing module

[1652] Output: Feedback document with specific improvements and suggestions

[1653] Specific actions: The server compiles the analysis results and generates a feedback document containing specific improvements and suggestions.

[1654] Step 13:

[1655] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[1656] Input: Audio file saved in temporary directory

[1657] Output: Emotion data based on audio data

[1658] Specific operation: The server passes the voice data to the API of the voice emotion analysis module, determines the emotion based on the voice characteristics (e.g., tone, intensity, speed) and returns the emotion data.

[1659] Step 14:

[1660] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[1661] Input: Audio data passed to the audio emotion analysis module

[1662] Output: Extracted emotion data

[1663] Specific operation: The voice emotion analysis module analyzes the voice data, extracts emotions (e.g., tension, anxiety, joy), and returns the results to the server in JSON format.

[1664] Step 15:

[1665] The server passes the facial image in the image file to the image emotion analysis means, and analyzes the emotion.

[1666] Input: Face image saved in temporary directory

[1667] Output: Emotion data based on facial images

[1668] Specific operation: The server passes the user's facial image in the image file to the API of the image emotion analysis module, and determines the emotion based on facial expression characteristics (e.g., smile, eyebrow movement).

[1669] Step 16:

[1670] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[1671] Input: A face image passed to the image emotion analysis module

[1672] Output: Extracted emotion data

[1673] Specific operation: The image emotion analysis module analyzes the facial image, extracts emotions (e.g., happiness, surprise, sadness), and returns the results to the server in JSON format.

[1674] Step 17:

[1675] The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[1676] Input: Extracted speech and image emotion data

[1677] Output: Feedback text reflecting the sentiment data

[1678] Specific operation: The server integrates emotional data obtained from audio and images and adds feedback according to the user's emotional state (e.g., how to relax if tense, words of encouragement if depressed).

[1679] As described above, the program of this system includes a process for multifaceted analysis of presentation materials and audio data uploaded by the user and generating detailed feedback that takes the user's emotions into consideration.

[1680] (Application example 2)

[1681] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1682] Conventional presentation feedback systems simply analyze text and voice, and do not provide detailed feedback tailored to the user's emotional state or the intended recipient. As a result, they may not receive optimal instruction to improve their presentation skills. In the training of logistics center staff, improving presentation skills contributes to improved work efficiency, so a system that provides more effective and individually optimized feedback is needed.

[1683] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for performing emotion analysis and generating feedback according to the user's emotional state, and means for generating information for training logistics center staff and improving their presentation skills. This enables logistics center staff to improve their presentation skills and receive optimized feedback according to their emotional state.

[1684] "Presentation materials" refers to image files such as slides and documents used in a presentation.

[1685] "Presentation audio data" refers to digital data of the audio spoken by the presenter during a presentation.

[1686] "Image analysis" refers to the technique of extracting text and important visual information from image files.

[1687] "Voice analysis" refers to the technology of converting voice data into text and analyzing speaking style, emphasis, etc.

[1688] "Feedback" refers to specific advice or suggestions regarding the quality of the presentation and areas for improvement.

[1689] "Emotion analysis" refers to the technology of detecting and analyzing a user's emotional state from their voice and visual data.

[1690] A "logistics center" refers to a facility that stores, sorts, and ships goods and materials.

[1691] "Education" refers to the guidance and training of logistics center staff aimed at improving work efficiency and presentation skills.

[1692] "Natural language processing technology" refers to artificial intelligence technology for analyzing text data, understanding its meaning and structure, and processing it.

[1693] "Personalized learning" refers to a technology that uses past data to learn in order to provide optimal feedback for specific users and situations.

[1694] "Weighting" refers to the practice of prioritizing and adjusting analysis results based on specific review destinations or presentation content.

[1695] "Server" refers to a computer system that receives, stores, analyzes data, and generates feedback.

[1696] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[1697] The system for implementing this invention receives image files of presentation materials and presentation audio data, analyzes them, and generates and displays feedback. The system program and its processing flow are described below.

[1698] System Overview

[1699] The system consists of the following main components:

[1700] A means of receiving image files of presentation materials

[1701] A means of receiving presentation audio data

[1702] means for analyzing the image file and the audio data, respectively;

[1703] means for generating feedback based on the analysis results;

[1704] A means of performing emotion analysis and generating feedback based on the user's emotional state

[1705] A means of providing information to train logistics center staff and improve their presentation skills

[1706] Hardware and Software Configuration

[1707] Hardware:

[1708] Smartphone (iOS / Android)

[1709] Server (AWS, Google Cloud GCP, etc.)

[1710] Head-mounted displays (Microsoft HoloLens, Google Glass)

[1711] software:

[1712] Image processing library (OpenCV)

[1713] Speech analysis library (Google Cloud Speech-to-Text API)

[1714] Natural Language Processing (spaCy, NLTK)

[1715] Sentiment analysis library (TensorFlow, Keras)

[1716] Processing flow explanation

[1717] 1. Data Receipt and Storage:

[1718] The user uploads image files and audio data of presentation materials via a smartphone application.

[1719] The device reads the data and sends it to the server.

[1720] The server receives these files and stores them in a secure temporary storage directory.

[1721] 2. Image Analysis:

[1722] The server loads the image file from the temporary storage directory.

[1723] Image analysis module (OpenCV) extracts text and important visual information from image files.

[1724] 3. Audio analysis:

[1725] The server passes the audio file to a speech analysis module (Google Cloud Speech-to-Text API) and converts it into text.

[1726] The speech analysis module analyzes speaking style and emphasis, and generates text data.

[1727] 4. Natural Language Processing and Feedback Generation:

[1728] The server passes the text data obtained from image analysis and voice analysis to a natural language processing module (spaCy).

[1729] The natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[1730] 5. Emotion analysis:

[1731] The server passes the audio data and image files to an emotion analysis module (TensorFlow, Keras) to analyze the user's emotional state.

[1732] Obtain emotional data and reflect it in feedback.

[1733] Specific examples

[1734] Presentation support with smart trainers

[1735] Users upload slides about new logistics solutions and audio data for presentations based on them.

[1736] The server analyzes the images and audio and generates text data about the content of the slides.

[1737] The natural language processing module detects redundant expressions and logical inconsistencies and generates feedback suggesting specific improvements.

[1738] The sentiment analysis module detects nervousness or lack of confidence during a presentation and adds encouraging words or advice as feedback.

[1739] Prompt Sentence Examples

[1740] "Please advise the user on improvements based on the slide presentation and audio data."

[1741] "Perform emotional analysis during presentations and offer advice on how to relax if people are nervous."

[1742] As described above, this system helps logistics center staff improve their presentation skills by analyzing presentation materials and audio data, and provides optimal feedback according to their emotional state.

[1743] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1744] Specific processing steps of the program

[1745] Step 1:

[1746] Data Receipt and Storage

[1747] Input: Image files of presentation materials and audio data of presentations

[1748] How it works: A user uploads image files and audio data for a presentation using a smartphone application.

[1749] Output: Image files and audio data are sent to the server and stored in a secure temporary directory.

[1750] Step 2:

[1751] Loading and analyzing image files

[1752] Input: Image file saved in temporary directory

[1753] Operation: The server passes the image file to the image analysis module (OpenCV).

[1754] Data processing / calculation: The image analysis module extracts text and important visual information from the image files and returns it to the server.

[1755] Output: Extracted text data

[1756] Step 3:

[1757] Loading and analyzing audio files

[1758] Input: Audio file saved in temporary directory

[1759] How it works: The server passes the audio file to the speech analysis module (Google Cloud Speech-to-Text API).

[1760] Data processing / calculation: The speech analysis module converts the speech data into text, analyzes the speech characteristics (emphasis, speaking style, etc.), and returns the text to the server.

[1761] Output: Converted text data and voice characteristics data

[1762] Step 4:

[1763] Natural Language Processing and Feedback Generation

[1764] Input: Extracted text data and speech characteristics data

[1765] How it works: The server passes this data to the natural language processing module (spaCy).

[1766] Data processing / calculation: A natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[1767] Output: Feedback text with improvements

[1768] Step 5:

[1769] Emotion analysis

[1770] Input: Audio data and image files

[1771] How it works: The server passes the audio data and image files to the emotion analysis module (TensorFlow, Keras).

[1772] Data processing / calculation: The emotion analysis module analyzes the user's emotional state (e.g., tension, relief, excitement, etc.) based on their voice and image, and returns the result to the server.

[1773] Output: Emotional state data

[1774] Step 6:

[1775] Consolidating and providing final feedback

[1776] Input: Feedback text and emotional state data

[1777] Operation: The server integrates the emotional state data with the feedback text to generate the final feedback.

[1778] Output: Consolidated feedback text

[1779] What it does: The server displays the consolidated feedback to the user.

[1780] The above are the specific processing steps of this system. Through the data processing and calculations performed at each step, the user can receive detailed feedback based on the presentation materials and audio data, and that is tailored to the user's emotional state.

[1781] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1782] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1783] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1784] [Fourth embodiment]

[1785] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1786] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1787] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1788] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1789] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1790] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1791] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1792] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1793] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1794] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1795] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1796] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1797] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1798] This invention relates to a system that analyzes image files and audio data of presentation materials and provides feedback. The program processing of this system is explained in detail below in natural language.

[1799] System Overview

[1800] Users upload image files to be used in their presentations and audio data recording the presentation to the system. The server receives these files and performs image and audio analysis. The analyzed text data is further analyzed using natural language processing technology to generate specific feedback based on the presentation's content and speaking style. The feedback is weighted based on the specific reviewer and presentation content and provided to the user.

[1801] Program processing

[1802] Receiving and storing data

[1803] 1. After the user logs in, they upload the image files of the presentation materials and the presentation audio data through a dedicated interface.

[1804] 2. The terminal reads the file specified by the user and sends it to the server.

[1805] 3. The server receives these files and stores them in a secure temporary storage directory.

[1806] Image analysis

[1807] 1. The server loads the image file from the temporary storage directory.

[1808] 2. The server starts the image analysis module and performs processing to convert the image file into text data.

[1809] 3. The image analysis module extracts the text in the image file and returns the data to the server.

[1810] 4. The server formats the retrieved text data and prepares it for passing to the natural language processing module.

[1811] Audio analysis

[1812] 1. The server loads the audio file from the temporary storage directory.

[1813] 2. The server launches the speech analysis module and processes the speech data to convert it into text.

[1814] 3. The speech analysis module converts the speech into text data and returns the data to the server.

[1815] 4. The server formats the acquired speech-to-text data and prepares it for passing to the natural language processing module.

[1816] Natural Language Processing and Feedback Generation

[1817] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[1818] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[1819] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[1820] Weighting and Individual Optimization

[1821] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[1822] 2. The server applies a weighting algorithm based on the received information to adjust the analysis results.

[1823] 3. The server uses a personalized learning module to look at past feedback data and generate feedback optimized for the specific reviewer.

[1824] Specific examples

[1825] Example 1: Proposal presentation

[1826] The user uploads slide images and audio data for new product proposals.

[1827] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[1828] The server generates specific feedback based on the analysis, such as identifying redundancies in the proposal or adding specific use cases to attract the client's interest.

[1829] Improve your presentation materials and delivery based on the feedback provided by users.

[1830] Example 2: Apology presentation

[1831] A user uploads slide images and audio data apologizing for the project delay.

[1832] The server performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[1833] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[1834] Improve your apology presentation based on user-provided feedback.

[1835] This concludes the description of the "Mode for Carrying Out the Invention" of the present invention. This system provides specific and detailed feedback to help users give effective presentations.

[1836] The processing flow will be explained below.

[1837] Step 1:

[1838] Users upload image files of presentation materials and presentation audio data through a dedicated interface.

[1839] Step 2:

[1840] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[1841] Step 3:

[1842] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[1843] Step 4:

[1844] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[1845] Step 5:

[1846] The image analysis module extracts the text within the image file and returns the text data to the server.

[1847] Step 6:

[1848] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[1849] Step 7:

[1850] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[1851] Step 8:

[1852] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[1853] Step 9:

[1854] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[1855] Step 10:

[1856] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1857] Step 11:

[1858] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[1859] Step 12:

[1860] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[1861] Step 13:

[1862] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[1863] Step 14:

[1864] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[1865] Step 15:

[1866] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[1867] Step 16:

[1868] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[1869] Step 17:

[1870] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[1871] Step 18:

[1872] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[1873] Example 1

[1874] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1875] Conventional presentation feedback systems mainly analyze image files and audio data individually, which has the drawback of insufficient comprehensive analysis and specific feedback to users. Furthermore, since it is not possible to weight or optimize based on specific reviewers or presentation content, it is difficult to obtain the specific and accurate feedback that users need.

[1876] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1877] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for analyzing the analyzed text data using natural language processing technology, means for generating feedback based on the analysis results, means for displaying the feedback, means for the user to input specific review recipients and presentation content, means for weighting the feedback based on the input information, and means for optimizing the feedback using past feedback data. This makes it possible to comprehensively analyze the presentation materials and audio data and provide specific and accurate feedback based on the specific review recipients and presentation content.

[1878] "Presentation materials" are a collection of images and slides that visually represent the contents of a presentation.

[1879] "Image files" refer to still image data that has been digitized from presentation materials.

[1880] "Presentation audio data" refers to audio data recorded from a presentation.

[1881] "Server" refers to the central computer system used for analysis, data processing, and feedback generation.

[1882] A "terminal" refers to a computer or device operated by a user, and is responsible for exchanging data with a server.

[1883] "User" refers to a person who uses the system to upload presentation data and receive feedback.

[1884] "Analysis means" refers to the function of converting image files and audio data into text data and analyzing it using natural language processing technology.

[1885] "Natural language processing technology" refers to technology that enables computers to understand, analyze, and generate human language.

[1886] "Feedback" refers to information, including suggestions for improvement and recommendations, provided to users based on analyzed data.

[1887] "Weighting" refers to the process of adjusting the analysis results based on the specific review destination or presentation content.

[1888] "Optimization" refers to using past feedback data to make adjustments to improve the accuracy and usefulness of feedback.

[1889] The present invention relates to a system for analyzing image files and audio data of presentation materials and providing feedback to users. An embodiment of this system will be described in detail below.

[1890] System Overview

[1891] The system of the present invention allows users to upload image files used as presentation materials and audio data for the presentation, and then analyzes the materials and audio and provides feedback. Specifically, the server receives and analyzes these files, and uses natural language processing technology to generate feedback appropriate for the user.

[1892] Hardware and software used

[1893] Server: A central computer system that analyzes and processes data and generates feedback. The server is equipped with image analysis modules, audio analysis modules, and natural language processing modules.

[1894] Device: A computer or device operated by a user to upload files or display feedback.

[1895] Image Analysis Module: Analyzes image files using image analysis libraries such as Python, OpenCV, and Tesseract OCR.

[1896] Speech analysis module: Analyzes speech data using speech analysis APIs such as Google Speech-to-Text and IBM Watson Speech to Text.

[1897] Natural Language Processing module: Analyzes and processes text data using the NLTK library and generative AI models.

[1898] Data processing and calculation

[1899] When a user uploads image files and audio data for a presentation, the device sends these files to the server. The server receives these files and stores them in a secure temporary directory. The server then launches image and audio analysis modules to convert the image files into text data and the audio data into text.

[1900] The server passes the acquired text data to a natural language processing module to detect redundancies, unclear expressions, and logical inconsistencies. The server generates feedback based on the analysis results and provides it to the user.

[1901] Additionally, when a user inputs a specific review destination or presentation content (e.g., proposal, apology), the server applies a weighting algorithm and references past feedback data to optimize the feedback.

[1902] Specific examples

[1903] Proposal presentation example

[1904] User: Uploads slide images and audio data for a new product proposal presentation to the system.

[1905] Server: Performs image and audio analysis to analyze the content of slides and speaking style.

[1906] Server: Generates specific feedback based on the analysis, such as redundancies in the proposal or adding specific use cases to capture the client's interest.

[1907] Users: Improve their presentation materials and delivery based on the feedback provided.

[1908] Example of an apology presentation

[1909] User: Upload slide images and audio data to the system apologizing for the project delay.

[1910] Server: Performs image and audio analysis to point out inappropriate comments and unclear areas for improvement.

[1911] Server: Based on the analysis results, provide feedback on specific improvement measures and how to choose the right words to use in your apology.

[1912] User: Improve your apology presentation based on the feedback provided.

[1913] This is the embodiment of the present invention. This system allows users to receive specific and detailed feedback on how to give effective presentations.

[1914] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1915] Step 1:

[1916] A user logs in.

[1917] Input: User ID and password.

[1918] How it works: A user accesses the system's login page and enters their user ID and password.

[1919] Output: If authentication is successful, the user can access the main interface of the system.

[1920] Step 2:

[1921] The user uploads image files of the presentation materials and audio data of the presentation.

[1922] Input: Image files and audio data files.

[1923] How it works: The user selects image and audio files in a dedicated interface and clicks the upload button.

[1924] Output: The selected file is prepared by the terminal.

[1925] Step 3:

[1926] The terminal reads the file specified by the user and sends it to the server.

[1927] Input: Image files and audio data files.

[1928] How it works: The device gets the path of the selected file and sends it to the server using an HTTP POST request.

[1929] Output: The file data sent to the server.

[1930] Step 4:

[1931] The server receives the file and stores it in a secure temporary storage directory.

[1932] Input: Submitted image and audio data files.

[1933] How it works: The server saves the received file in a temporary directory, such as the / tmp directory.

[1934] Output: File saved in temporary directory.

[1935] Step 5:

[1936] The server loads the image file from the temporary storage directory.

[1937] Input: The path to the saved image file.

[1938] What it does: The server loads the image file from the temporary storage directory into memory.

[1939] Output: The loaded image data.

[1940] Step 6:

[1941] The server starts the image analysis module and converts the image file into text data.

[1942] Input: The loaded image data.

[1943] How it works: The server uses image analysis libraries such as OpenCV and Tesseract OCR to extract text from images.

[1944] Output: The extracted text data.

[1945] Step 7:

[1946] The server reads the audio file from the temporary storage directory.

[1947] Input: The path to the saved audio file.

[1948] How it works: The server loads the audio file from the temporary storage directory into memory.

[1949] Output: The loaded audio data.

[1950] Step 8:

[1951] The server starts a voice analysis module and converts the voice data into text data.

[1952] Input: The loaded audio data.

[1953] How it works: The server uses a speech analysis API such as Google Speech-to-Text or IBM Watson Speech to Text to convert the audio data into text.

[1954] Output: The converted speech-to-text data.

[1955] Step 9:

[1956] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[1957] Input: Extracted image-text data and audio-text data.

[1958] How it works: The server sends text data to the natural language processing module via an API.

[1959] Output: Text data sent to the natural language processing module.

[1960] Step 10:

[1961] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[1962] Input: The text data sent.

[1963] How it works: The natural language processing module uses the NLTK library and generative AI models to analyze the quality of text.

[1964] Output: Analysis result data.

[1965] Step 11:

[1966] Based on the analysis results obtained by the server from the natural language processing module, it generates written feedback including specific areas for improvement and suggestions.

[1967] Input: Analysis result data.

[1968] How it works: The server uses templates based on the analysis results to summarize specific improvements and suggestions in natural language.

[1969] Output: The generated feedback sentence.

[1970] Step 12:

[1971] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[1972] Input: Specific reviewer and presentation content.

[1973] How it works: The user selects and inputs the review recipient and presentation purpose in the interface.

[1974] Output: The input review destination and presentation content data.

[1975] Step 13:

[1976] Based on the information received, the server applies a weighting algorithm to adjust the analysis results.

[1977] Input: Review destination and presentation content data, analysis result data.

[1978] How it works: The server applies weighting parameters appropriate to the reviewer and presentation content to adjust the feedback.

[1979] Output: Weighted analysis result data.

[1980] Step 14:

[1981] The server uses a personalized learning module to reference past feedback data and generate feedback optimized for a particular reviewer.

[1982] Input: Weighted analysis result data, past feedback data.

[1983] How it works: The server queries a database of past feedback and optimizes the feedback based on feedback performance in similar situations.

[1984] Output: Optimized feedback text.

[1985] The above is the specific flow of program processing in this system.

[1986] (Application example 1)

[1987] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1988] Conventional presentation training systems simply require users to upload presentation materials and audio data, which the system then analyzes and provides feedback. This makes it difficult to provide feedback optimized for specific environments (e.g., in a factory) or situations (e.g., safety measures presentations). They also lack the ability to detect redundant sentences or unclear expressions in real time and immediately suggest specific areas for improvement. This leaves users facing specific challenges in conducting effective presentation training.

[1989] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1990] In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for displaying the feedback, and means for coordinating with a robot that performs presentation training in a factory. This allows users to receive feedback in real time that is optimized for specific environments and situations in the factory, enabling more effective presentation training.

[1991] An "image file of presentation materials" is a file in which information such as slides and charts used in a presentation is saved in image format.

[1992] "Presentation audio data" refers to data recorded in audio format of what was said during a presentation.

[1993] "Means for analysis" refers to the technology or device that converts received image files or audio data into text information and performs analysis.

[1994] "Means for generating feedback" refers to technology or devices that generate suggestions for improvement or comments to be provided to users based on the content of the presentation and the results of data analysis.

[1995] "Means for displaying feedback" refers to techniques or devices for visually or audibly presenting the generated feedback to the user.

[1996] A "robot for in-factory presentation training" is an autonomous or remotely controlled mechanical device that supports presentation training in a factory and provides feedback.

[1997] A "weighting means" is a technique or device that adjusts the evaluation or results based on the analyzed text data according to specific criteria.

[1998] "Means for optimizing feedback through individual learning" refers to technology or devices that generate feedback optimized for each user based on past analysis results and feedback data.

[1999] "Means for converting voice data to text" refers to technology or devices that use voice recognition technology to convert the spoken content in an audio file into text form.

[2000] "Means for analyzing using natural language processing technology" refers to technology or devices that perform semantic analysis and error checking of text data converted using natural language processing technology.

[2001] "Means for detecting redundant sentences and unclear expressions" refers to technology or devices that identify unimportant long sentences and difficult-to-understand expressions from the analyzed text data and point them out to the user.

[2002] This invention relates to a system that analyzes image files of presentation materials and audio data of presentations and provides feedback. This system is intended to support presentation training, particularly in factories, and to provide users with optimized feedback.

[2003] System configuration and program processing

[2004] Receiving and storing data

[2005] The server receives and temporarily stores image files and audio data for presentation materials uploaded by users via the interface of the robots in the factory. The server stores the received files in a secure temporary storage directory and prepares them for analysis.

[2006] Image analysis

[2007] The server uses PIL (Python Imaging Library) and pytesseract libraries to convert image files to text, extracting the text data within the image files, which is then formatted by the server and prepared for natural language processing.

[2008] Audio analysis

[2009] The server uses the speech_recognition library to convert the audio file to text, which is then formatted and prepared for natural language processing as well.

[2010] Natural Language Processing and Feedback Generation

[2011] The server analyzes the text data obtained from image and audio analysis using natural language processing technology. This process uses the spacy library. During the analysis, redundant sentences and unclear expressions are identified in the text data, and specific feedback on improvements is generated.

[2012] Weighting and Individual Optimization

[2013] The server weights specific review destinations and presentation content based on the analyzed text data, and also provides optimized feedback for each user by referencing past feedback data through personalized learning.

[2014] Specific examples

[2015] Example 1: Proposal presentation

[2016] When a user gives a presentation to propose a new product in a factory, they upload slide images and audio data. The server analyzes these files and generates specific feedback such as advice like, "The text on this slide is redundant. Please explain it more succinctly."

[2017] Example 2: Safety Presentation

[2018] When giving a presentation on safety measures in a factory, the user uploads the materials and audio data. The server analyzes the data and provides feedback such as, "The explanation of safety precautions tends to be redundant. In particular, make the section on 'checking safety devices' more concise."

[2019] Example prompt sentence:

[2020] "The text on this slide is redundant. Please explain it more succinctly."

[2021] "Safety precautions can be redundant. Please be more concise, especially the 'check safety equipment' section."

[2022] This invention allows users to receive real-time feedback optimized for specific environments and situations within a factory, enabling more effective presentation training. This system is an important tool for improving the effectiveness of presentations in specific situations.

[2023] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2024] Step 1:

[2025] A user logs in and uploads image files and audio data for presentation materials. The user uses the robot interface in the factory to select these files and send them to the server. The input is the image files and audio files, and the output is the server receiving the files.

[2026] Step 2:

[2027] After the server receives the uploaded files, it stores them in a secure temporary directory. The input is the image and audio files uploaded by the user, and the output is that these files are stored in the temporary directory.

[2028] Step 3:

[2029] The server reads the saved image file and converts it to text data using image analysis modules (PIL and pytesseract). The input is the image file saved in the temporary storage directory, and the output is the converted text data. This text data is returned to the server.

[2030] Step 4:

[2031] The server reads the saved audio file and converts it into text using a speech analysis module (speech_recognition library). The input is the audio file saved in the temporary storage directory, and the output is the converted text. This text data is also returned to the server.

[2032] Step 5:

[2033] The server passes the text data obtained from image and audio analysis to a natural language processing module (the spacy library) for analysis. The input is the converted text data, and the output is the analysis results. Based on these analysis results, redundant sentences and unclear expressions are detected.

[2034] Step 6:

[2035] The server generates specific feedback based on the analysis results of the natural language processing. The input is the analysis results from the natural language processing module, and the output is the generated feedback. This feedback includes specific advice on redundant sentences and areas for improvement.

[2036] Step 7:

[2037] The server weights the generated feedback based on the specific reviewer and presentation content. The input is feedback and reviewer information, and the output is weighted feedback.

[2038] Step 8:

[2039] The server uses an individual learning module to generate optimized feedback by referring to past feedback data, with the input being the past feedback data and the current feedback, and the output being the optimized feedback.

[2040] Step 9:

[2041] The server visually displays the generated feedback to the user: the input is the optimized feedback, and the output is the user seeing the feedback on the screen.

[2042] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2043] overview

[2044] This invention relates to a system that analyzes image files and audio data from presentation materials and provides feedback. It also has the function of generating feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions.

[2045] System configuration

[2046] The system consists of the following main components:

[2047] 1. Data Receiving Method

[2048] 2. Image analysis methods

[2049] 3. Audio analysis methods

[2050] 4. Natural Language Processing Methods

[2051] 5. Feedback Generation Methods

[2052] 6. Weighting Methods

[2053] 7. Personalized learning tools

[2054] 8. Emotion Engine (Voice Emotion Analysis and Image Emotion Analysis)

[2055] 9. Feedback Display Methods

[2056] Program processing

[2057] The program processing of this system will be specifically explained below.

[2058] Receiving and storing data

[2059] 1. The user uploads the image files of the presentation materials and the presentation audio data through a dedicated interface.

[2060] 2. The terminal reads the file specified by the user and sends it to the server.

[2061] 3. The server receives the file and stores it in a secure temporary storage directory.

[2062] Image analysis

[2063] 1. The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[2064] 2. The image analysis module extracts the text from the image file and returns the text data to the server.

[2065] 3. The server formats the text data it receives and stores it in a data store for the natural language processing module.

[2066] Audio analysis

[2067] 1. The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[2068] 2. The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[2069] 3. The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[2070] Natural Language Processing and Feedback Generation

[2071] 1. The server passes the text data obtained from image analysis and audio analysis to the natural language processing module.

[2072] 2. The natural language processing module analyzes the text data to detect redundancies, unclear expressions, logical inconsistencies, etc.

[2073] 3. Based on the analysis results obtained by the server's natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[2074] Emotion engine processing

[2075] 1. The server passes the voice data to the voice emotion analysis means and analyzes the user's emotions.

[2076] 2. The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[2077] 3. The server passes the user's facial image contained in the image file to the image emotion analysis means, and analyzes the emotion.

[2078] 4. The image emotion analysis means extracts emotions from the facial image and returns the emotion data to the server.

[2079] 5. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[2080] Weighting and Individual Optimization

[2081] 1. The user enters a specific reviewer (e.g., president, department manager, client) and presentation content (proposal, apology, etc.).

[2082] 2. Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[2083] 3. The server uses a personalized learning module to generate feedback optimized for the specific reviewer, referencing past feedback data.

[2084] Specific examples

[2085] Example 1: Proposal presentation

[2086] The user uploads slide images and audio data for new product proposals.

[2087] The server performs image and audio analysis to analyze the content of the slides and the speaking style.

[2088] The server uses an emotion engine to analyze the user's emotions and provides feedback based on that. For example, if the user is nervous, the server adds advice to ease the tension.

[2089] Based on the analysis, the server generates specific feedback, such as redundancy in the proposal or adding use cases to attract the client's interest.

[2090] Improve your presentation materials and delivery based on the feedback provided by users.

[2091] Example 2: Apology presentation

[2092] A user uploads slide images and audio data apologizing for the project delay.

[2093] The server performs image and audio analysis to point out inappropriate comments and unclear improvement suggestions.

[2094] The server uses an emotion engine to analyze the user's emotions and provide feedback based on that. For example, if the user is feeling down, it might add words of encouragement or an appropriate apology.

[2095] Based on the analysis results, the server provides feedback on specific improvement measures and how to choose the right words to use in an apology.

[2096] Improve your apology presentation based on user-provided feedback.

[2097] This concludes the "Mode for Carrying Out the Invention" of the present invention. This system not only provides specific and detailed feedback to help users give effective presentations, but also enables support based on the user's emotional state.

[2098] The processing flow will be explained below.

[2099] Step 1:

[2100] The user uploads image files of presentation materials and presentation audio data from a dedicated interface.

[2101] Step 2:

[2102] The terminal reads the file specified by the user from the local storage and transmits it to the server.

[2103] Step 3:

[2104] The server receives the uploaded file and saves it in a temporary directory. The server returns a message to the terminal indicating that the file was successfully received.

[2105] Step 4:

[2106] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[2107] Step 5:

[2108] The image analysis module extracts the text within the image file and returns the text data to the server.

[2109] Step 6:

[2110] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[2111] Step 7:

[2112] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[2113] Step 8:

[2114] The speech analysis module converts the speech data into text and returns the text data and speech characteristics (speech rate, intonation, pauses, etc.) to the server.

[2115] Step 9:

[2116] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[2117] Step 10:

[2118] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[2119] Step 11:

[2120] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[2121] Step 12:

[2122] The server passes the user's facial image contained in the image file to the image emotion analysis means, which analyzes the emotion.

[2123] Step 13:

[2124] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[2125] Step 14:

[2126] The server comprehensively analyzes the acquired voice emotion data and image emotion data and passes it to the natural language processing module.

[2127] Step 15:

[2128] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[2129] Step 16:

[2130] The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc.

[2131] Step 17:

[2132] Based on the analysis results and emotional data obtained by the server from the natural language processing module, the server generates written feedback including specific areas for improvement and suggestions.

[2133] Step 18:

[2134] The user inputs a specific reviewer (e.g., president, department manager, client) and the presentation content (proposal, apology, etc.).

[2135] Step 19:

[2136] Based on the information received by the server, a weighting algorithm is applied to adjust the analysis results.

[2137] Step 20:

[2138] The server uses a personalized learning module to generate feedback optimized for a particular reviewer by referencing past feedback data.

[2139] Step 21:

[2140] The server generates the final feedback in JSON or HTML format and sends it to the user's device.

[2141] Step 22:

[2142] The terminal displays the received feedback to the user in real time, and the user confirms the feedback content.

[2143] Step 23:

[2144] Based on the feedback provided by the user, the presentation materials and delivery method are revised and improved.

[2145] Example 2

[2146] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2147] Conventional presentation analysis systems provide limited feedback based on the analysis of image files and audio data, and are insufficient in individual optimization that takes into account the user's emotions. Furthermore, improvements are needed in the accuracy of the analysis results and the specific content of the feedback. Therefore, a system is needed that provides detailed and specific feedback to help users deliver effective presentations and generates feedback that corresponds to the user's emotional state.

[2148] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to upload an image file of a presentation material, a means for a user to upload presentation audio data, a means for a terminal to transmit a file designated by the user to the server, a means for the server to save the received file in a temporary storage directory, a means for the server to read the image file from the temporary storage directory and pass it to an image analysis module, a means for the image analysis module to extract text from the image file and return it to the server, a means for the server to format the acquired text data and save it in a data store for the natural language processing module, a means for the server to read an audio file from the temporary storage directory and pass it to the audio analysis module, a means for the audio analysis module to convert audio data into text and return it to the server, and a means for the server to convert the acquired audio text data into text. The system includes a means for formatting the data and analysis results and saving them in a data store, a means for the server to pass text data obtained from the image and audio analyses to a natural language processing module, a means for the natural language processing module to analyze the text data and detect redundancy, unclear expressions, and logical inconsistencies, a means for the server to generate feedback including specific improvements and suggestions based on the analysis results, a means for the server to pass audio data to a voice emotion analysis means and analyze the user's emotions, a means for the voice emotion analysis means to extract emotions from the audio data and return this to the server, a means for the server to pass facial images in image files to an image emotion analysis means and analyze them, a means for the image emotion analysis means to extract emotions from the facial images and return this to the server, and a means for the server to comprehensively analyze the audio emotion data and image emotion data and reflect this in feedback. This makes it possible to provide individually optimized, detailed feedback that takes into account the emotional state of the user based on the analysis results of the user's presentation materials and presentation audio data.

[2149] "User" refers to a person who uploads image files of presentation materials and presentation audio data through the system interface.

[2150] "Terminal" refers to a device that reads a file specified by a user and sends it to a server.

[2151] A "server" refers to a computer system that stores files received from users in a temporary storage directory and performs various processes such as image analysis and audio analysis.

[2152] The "temporary storage directory" refers to a directory for temporarily storing files received by the server.

[2153] "Image analysis module" refers to software or hardware for extracting text from image files.

[2154] "Speech Analysis Module" refers to software or hardware for converting speech data into text and extracting the text data and speech characteristics.

[2155] A "natural language processing module" refers to software that analyzes text data and detects redundancies, unclear expressions, and logical inconsistencies.

[2156] "Voice emotion analysis means" refers to software or hardware for extracting a user's emotion from voice data.

[2157] "Image emotion analysis means" refers to software or hardware for extracting a user's emotion from a facial image.

[2158] "Natural language processing technology" refers to all technologies for analyzing text data and understanding its meaning and structure.

[2159] "Data store" refers to a database or storage system for storing text data and voice characteristic data obtained through analysis.

[2160] "Weighting algorithm" refers to a computational method for adjusting analysis results based on specific review destinations or presentation content.

[2161] "Personalized Learning Module" refers to software that references past feedback data and optimizes feedback for specific reviewers.

[2162] This invention relates to a system that analyzes image files and presentation audio data for presentation materials and provides feedback. It also has a function to generate feedback according to the user's emotional state by combining it with an emotion engine that recognizes the user's emotions. The main components of this system include a data receiving means, an image analysis means, an audio analysis means, a natural language processing means, a feedback generation means, a weighting means, an individual learning means, an emotion engine (a voice emotion analysis means and an image emotion analysis means), and a feedback display means.

[2163] The user uploads image files of presentation materials and audio data for the presentation through a dedicated interface. The device reads the files specified by the user and sends them to the server. The server saves the received files in a temporary storage directory. At this time, a timestamp and unique ID are added to the file name to ensure the uniqueness and security of the file.

[2164] The server reads the image files from the temporary storage directory and passes them to the image analysis module, which extracts the text from the image files and returns the text data to the server, which formats the text data and stores it in a data store for the natural language processing module. This extracts the text information from the presentation slides and prepares them for the next processing step.

[2165] For speech analysis, the server reads the audio file from the temporary storage directory and passes it to the speech analysis module. The speech analysis module converts the audio data into text and returns the text data and speech characteristics to the server. The server formats the acquired speech text data and analysis results and saves them in a data store for the natural language processing module. This allows the presentation's speaking style and content to be extracted as text data.

[2166] Next, in the natural language processing and feedback generation step, the server passes the text data obtained from image and audio analysis to a natural language processing module. The natural language processing module analyzes the text data and detects redundancies, unclear expressions, logical inconsistencies, etc. Based on the analysis results obtained from the natural language processing module, the server generates written feedback including specific points for improvement and suggestions.

[2167] Furthermore, in emotion engine processing, the server passes voice data to voice emotion analysis means, which analyzes the user's emotion. The voice emotion analysis means extracts emotion from the voice data and returns this emotion data to the server. The server also passes the user's facial image contained in the image file to image emotion analysis means, which analyzes the emotion. The image emotion analysis means extracts emotion from the facial image and returns this emotion data to the server. The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects the results in feedback.

[2168] In the weighting and personalized optimization step, the user inputs the specific reviewer (e.g., president, manager, client) and presentation content (proposal, apology, etc.). The server applies a weighting algorithm based on the received information and adjusts the analysis results. The server then uses a personalized learning module to generate feedback optimized for the specific reviewer by referencing past feedback data.

[2169] Below is an example prompt:

[2170] Proposal presentation example

[2171] I have uploaded the following data in order to make a presentation proposing a new product.

[2172] Slide image: New product proposal slide

[2173] Audio data: Presentation audio

[2174] We ask for feedback on how to make the proposal clear and how to reflect any nervousness you may have.

[2175] Apology Presentation Example

[2176] The following data has been uploaded in order to make an apology presentation for the project delay.

[2177] Slide image: Apology slide

[2178] Audio data: Presentation audio

[2179] Point out inappropriate language or confusing parts and ask for feedback that includes encouraging words that reflect your disappointment.

[2180] This system provides specific and detailed feedback to help users give effective presentations, and enables support according to the user's emotional state.

[2181] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2182] Step 1:

[2183] The user uses a dedicated interface to select image files of presentation materials and presentation audio data.

[2184] Input: User-selected image and audio files

[2185] Output: Path of the selected file

[2186] Specific operation: The user opens a browser or a dedicated app, clicks the "Select files" button, selects the image and audio files to upload, and clicks the "Upload" button.

[2187] Step 2:

[2188] The terminal reads the image file and audio file specified by the user and transmits them to the server.

[2189] Input: Paths to image and audio files

[2190] Output: Image and audio files sent to the server

[2191] Specific operation: The device loads the selected file into memory and sends the file data to the server using an HTTP POST request. After sending, the device notifies the user that the file upload is complete.

[2192] Step 3:

[2193] The server saves the received file in a temporary directory.

[2194] Input: Image and audio files sent from the device

[2195] Output: File saved in temporary directory

[2196] Specific operation: The server saves the received file in a temporary directory. When saving, a timestamp and a unique ID are added to the file name to ensure the uniqueness and security of the file.

[2197] Step 4:

[2198] The server reads the image file from the temporary storage directory and passes it to the image analysis module.

[2199] Input: Image file saved in temporary directory

[2200] Output: Image data passed to the image analysis module

[2201] Specific operation: The server reads the target image file from the temporary storage directory into memory and passes the file data to the API endpoint of the image analysis module.

[2202] Step 5:

[2203] The image analysis module extracts the text within the image file and returns the text data to the server.

[2204] Input: Image data passed to the image analysis module

[2205] Output: Text data extracted from the image

[2206] Specific operation: The image analysis module uses OCR technology (e.g., Tesseract) to extract text from the image and returns the extracted text data to the server in JSON format.

[2207] Step 6:

[2208] The server formats the acquired text data and stores it in a data store for the natural language processing module.

[2209] Input: Text data returned from the image analysis module

[2210] Output: Formatted text data

[2211] Specific operation: The server converts the text data it receives into a specific format (e.g., separates each sentence) and saves the formatted data in a data store (e.g., MongoDB).

[2212] Step 7:

[2213] The server reads the audio file from the temporary storage directory and passes it to the audio analysis module.

[2214] Input: Audio file saved in temporary directory

[2215] Output: Audio data passed to the audio analysis module

[2216] Specific operation: The server reads the target audio file from the temporary storage directory into memory and passes the file data to the API endpoint of the audio analysis module.

[2217] Step 8:

[2218] The speech analysis module converts the speech data into text and returns the text data and speech characteristics to the server.

[2219] Input: Audio data passed to the audio analysis module

[2220] Output: Text data and speech characteristics extracted from the audio

[2221] Specific operation: The speech analysis module converts speech into text using speech recognition technology (e.g., Google Cloud Speech-to-Text) and returns the text data and speech characteristics (e.g., voice tone, speed, volume) to the server in JSON format.

[2222] Step 9:

[2223] The server formats the acquired speech-to-text data and analysis results and stores them in a data store for the natural language processing module.

[2224] Input: Speech text data and speech characteristics data returned from the speech analysis module

[2225] Output: Formatted speech text data and speech characteristics data

[2226] Specific operation: The server converts the acquired voice text data into a specific format (e.g., separates it into sentences) and saves the data and voice characteristic data in a data store.

[2227] Step 10:

[2228] The server passes the text data obtained from image analysis and voice analysis to the natural language processing module.

[2229] Input: Formatted image and audio text data

[2230] Output: Text data passed to the natural language processing module

[2231] Specific operation: The server reads the text data from the image analysis and voice analysis from the data store and sends the data to the API endpoint of the natural language processing module.

[2232] Step 11:

[2233] A natural language processing module analyzes text data to detect redundancies, unclear expressions, and logical inconsistencies.

[2234] Input: Text data passed to the natural language processing module

[2235] Output: Feedback data for the parsed text data

[2236] Specific operation: The natural language processing module uses a generative AI model (e.g., GPT-3) to analyze the text data, detecting redundancies, unclear expressions, logical inconsistencies, etc., and returns the analysis results to the server in JSON format.

[2237] Step 12:

[2238] Based on the analysis results, the server generates feedback including specific areas for improvement and suggestions.

[2239] Input: Analysis results returned from the natural language processing module

[2240] Output: Feedback document with specific improvements and suggestions

[2241] Specific actions: The server compiles the analysis results and generates a feedback document containing specific improvements and suggestions.

[2242] Step 13:

[2243] The server passes the voice data to a voice emotion analysis means, which analyzes the user's emotion.

[2244] Input: Audio file saved in temporary directory

[2245] Output: Emotion data based on audio data

[2246] Specific operation: The server passes the voice data to the API of the voice emotion analysis module, determines the emotion based on the voice characteristics (e.g., tone, intensity, speed) and returns the emotion data.

[2247] Step 14:

[2248] The voice emotion analysis means extracts emotions from the voice data and returns the emotion data to the server.

[2249] Input: Audio data passed to the audio emotion analysis module

[2250] Output: Extracted emotion data

[2251] Specific operation: The voice emotion analysis module analyzes the voice data, extracts emotions (e.g., tension, anxiety, joy), and returns the results to the server in JSON format.

[2252] Step 15:

[2253] The server passes the facial image in the image file to the image emotion analysis means, and analyzes the emotion.

[2254] Input: Face image saved in temporary directory

[2255] Output: Emotion data based on facial images

[2256] Specific operation: The server passes the user's facial image in the image file to the API of the image emotion analysis module, and determines the emotion based on facial expression characteristics (e.g., smile, eyebrow movement).

[2257] Step 16:

[2258] The image emotion analysis means extracts emotions from the facial images and returns the emotion data to the server.

[2259] Input: A face image passed to the image emotion analysis module

[2260] Output: Extracted emotion data

[2261] Specific operation: The image emotion analysis module analyzes the facial image, extracts emotions (e.g., happiness, surprise, sadness), and returns the results to the server in JSON format.

[2262] Step 17:

[2263] The server comprehensively analyzes the acquired voice emotion data and image emotion data and reflects it in feedback.

[2264] Input: Extracted speech and image emotion data

[2265] Output: Feedback text reflecting the sentiment data

[2266] Specific operation: The server integrates emotional data obtained from audio and images and adds feedback according to the user's emotional state (e.g., how to relax if tense, words of encouragement if depressed).

[2267] As described above, the program of this system includes a process for multifaceted analysis of presentation materials and audio data uploaded by the user and generating detailed feedback that takes the user's emotions into consideration.

[2268] (Application example 2)

[2269] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2270] Conventional presentation feedback systems simply analyze text and voice, and do not provide detailed feedback tailored to the user's emotional state or the intended recipient. As a result, they may not receive optimal instruction to improve their presentation skills. In the training of logistics center staff, improving presentation skills contributes to improved work efficiency, so a system that provides more effective and individually optimized feedback is needed.

[2271] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving image files of presentation materials, means for receiving presentation audio data, means for analyzing the image files and the audio data, means for generating feedback based on the analysis results, means for performing emotion analysis and generating feedback according to the user's emotional state, and means for generating information for training logistics center staff and improving their presentation skills. This enables logistics center staff to improve their presentation skills and receive optimized feedback according to their emotional state.

[2272] "Presentation materials" refers to image files such as slides and documents used in a presentation.

[2273] "Presentation audio data" refers to digital data of the audio spoken by the presenter during a presentation.

[2274] "Image analysis" refers to the technique of extracting text and important visual information from image files.

[2275] "Voice analysis" refers to the technology of converting voice data into text and analyzing speaking style, emphasis, etc.

[2276] "Feedback" refers to specific advice or suggestions regarding the quality of the presentation and areas for improvement.

[2277] "Emotion analysis" refers to the technology of detecting and analyzing a user's emotional state from their voice and visual data.

[2278] A "logistics center" refers to a facility that stores, sorts, and ships goods and materials.

[2279] "Education" refers to the guidance and training of logistics center staff aimed at improving work efficiency and presentation skills.

[2280] "Natural language processing technology" refers to artificial intelligence technology for analyzing text data, understanding its meaning and structure, and processing it.

[2281] "Personalized learning" refers to a technology that uses past data to learn in order to provide optimal feedback for specific users and situations.

[2282] "Weighting" refers to the practice of prioritizing and adjusting analysis results based on specific review destinations or presentation content.

[2283] "Server" refers to a computer system that receives, stores, analyzes data, and generates feedback.

[2284] "User" refers to a person who uses the system to upload presentation materials and audio data and receive feedback.

[2285] The system for implementing this invention receives image files of presentation materials and presentation audio data, analyzes them, and generates and displays feedback. The system program and its processing flow are described below.

[2286] System Overview

[2287] The system consists of the following main components:

[2288] A means of receiving image files of presentation materials

[2289] A means of receiving presentation audio data

[2290] means for analyzing the image file and the audio data, respectively;

[2291] means for generating feedback based on the analysis results;

[2292] A means of performing emotion analysis and generating feedback based on the user's emotional state

[2293] A means of providing information to train logistics center staff and improve their presentation skills

[2294] Hardware and Software Configuration

[2295] Hardware:

[2296] Smartphone (iOS / Android)

[2297] Server (AWS, Google Cloud GCP, etc.)

[2298] Head-mounted displays (Microsoft HoloLens, Google Glass)

[2299] software:

[2300] Image processing library (OpenCV)

[2301] Speech analysis library (Google Cloud Speech-to-Text API)

[2302] Natural Language Processing (spaCy, NLTK)

[2303] Sentiment analysis library (TensorFlow, Keras)

[2304] Processing flow explanation

[2305] 1. Data Receipt and Storage:

[2306] The user uploads image files and audio data of presentation materials via a smartphone application.

[2307] The device reads the data and sends it to the server.

[2308] The server receives these files and stores them in a secure temporary storage directory.

[2309] 2. Image Analysis:

[2310] The server loads the image file from the temporary storage directory.

[2311] Image analysis module (OpenCV) extracts text and important visual information from image files.

[2312] 3. Audio analysis:

[2313] The server passes the audio file to a speech analysis module (Google Cloud Speech-to-Text API) and converts it into text.

[2314] The speech analysis module analyzes speaking style and emphasis, and generates text data.

[2315] 4. Natural Language Processing and Feedback Generation:

[2316] The server passes the text data obtained from image analysis and voice analysis to a natural language processing module (spaCy).

[2317] The natural language processing module analyzes the text, detects redundancies and logical inconsistencies, and generates specific suggestions for improvement.

[2318] 5. Emotion analysis:

[2319] The server passes the audio data and image files to an emotion analysis module (TensorFlow, Keras) to analyze the user's emotional state. 【2...

Claims

1. means for receiving image files of presentation materials; means for receiving presentation audio data; means for analyzing the image file and the audio data; means for generating feedback based on the analysis results; The system includes means for displaying said feedback.

2. A means for weighting specific review destinations and presentation contents based on the analyzed image files and audio data; The system of claim 1 , further comprising means for optimizing feedback through personalized learning.

3. The analyzing means converts the image file into text data; a means for converting the audio data into text; The system of claim 1 further comprising means for analyzing the converted text data using natural language processing techniques.

Images