System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting speech data and converting it to text in real-time; means for analyzing the converted text data and evaluating the communication; means for capturing and collecting visual support material and evaluating its effectiveness; means for performing sentiment analysis based on the speech data and evaluating the understanding and satisfaction of the participants; means for providing feedback in real-time based on these evaluation results; and means for generating a detailed feedback report after the meeting ends.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Corporate meetings are held frequently, posing challenges in their efficiency and productivity. In particular, a lack of feedback makes it difficult to improve meeting quality. Productivity is affected by a variety of factors, including the appropriateness and persuasiveness of explanations given during meetings, clear communication, the effectiveness of visual support, and participants' low levels of understanding and persuasion. To address these challenges, this invention provides a system that provides real-time evaluation and feedback of meetings. [Means for solving the problem]

[0005] This invention provides a system including means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for evaluating participants' levels of understanding and satisfaction by performing sentiment analysis based on the voice data, means for providing real-time feedback based on the results of these evaluations, and means for generating a detailed feedback report after the meeting.The system further includes means for generating a follow-up plan for specific participants based on the evaluation results, and the real-time feedback means notifies the user of the feedback content on their device, allowing them to implement improvement measures on the spot, thereby improving the quality of the meeting.

[0006] "Voice data" refers to audio information generated during meetings, conversations, etc. that is recorded and transmitted in digital format.

[0007] A "means for converting to text" is a device or method that receives voice data and converts the content into text using voice recognition technology.

[0008] "Converted character string data" refers to data obtained by converting voice data into text format using voice recognition technology.

[0009] "Communication evaluation" is the process of evaluating the clarity, logic, appropriateness, etc. of what is said.

[0010] "Visual support materials" refers to information that is presented visually, such as presentation materials, whiteboards, and slides.

[0011] "Means of capturing or collecting" means a method or device that captures visual support materials in digital form using a camera, screen capture device, or other device.

[0012] "Means of evaluation" refers to a method or device for analyzing and evaluating the content and effects of collected data.

[0013] "Sentiment analysis" is the process of analyzing the tone of voice, pitch, rhythm of speech, etc. extracted from audio data to evaluate the speaker's emotions and psychological state.

[0014] The "means for providing feedback in real time" refers to a method or device for providing evaluation results obtained during a meeting as immediate feedback and notifying users of areas for improvement, etc.

[0015] A "detailed feedback report" is a report provided after the meeting that summarizes the detailed analysis results of each evaluation item and areas for improvement.

[0016] A "follow-up plan" is a plan to provide additional support or advice to specific participants based on the evaluation results.

[0017] "User's terminal" refers to devices used by users, such as PCs, smartphones, and tablets. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[0040] System Configuration

[0041] 1. User operations

[0042] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[0043] 2. Audio data collection and conversion

[0044] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[0045] 3. Communication Evaluation

[0046] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[0047] 4. Evaluating visual support

[0048] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[0049] 5. Sentiment analysis

[0050] The server performs sentiment analysis based on the audio data, specifically analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, level of understanding, and degree of agreement.

[0051] 6. Real-time feedback

[0052] The server generates real-time feedback based on the evaluation results and sends it to the device, where users can check the feedback and immediately implement improvements during the meeting.

[0053] 7. Generate detailed feedback reports

[0054] After the meeting, the server generates a detailed feedback report based on all the evaluation data and provides it to the user, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0055] Specific examples

[0056] For presentations

[0057] 1. Setup and Start

[0058] The user sets up a "presentation" on the device and starts the meeting, the server loads the template, and the setup is complete.

[0059] 2. Collection of audio data

[0060] When a user gives a presentation, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0061] 3. Communication evaluation and sentiment analysis

[0062] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0063] 4. Evaluating visual support

[0064] When a user is using a slide, the device collects images of the slide and sends them to the server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[0065] 5. Real-time feedback

[0066] The server generates feedback and sends it to the device in real time, allowing users to review the feedback and implement improvements during the meeting.

[0067] 6. Generate detailed feedback reports

[0068] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user, including the adequacy of the explanation, clarity of communication, visual support, and sentiment analysis results.

[0069] In this way, the present invention can greatly improve the efficiency and productivity of meetings.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[0073] Step 2:

[0074] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[0075] Step 3:

[0076] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[0077] Step 4:

[0078] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[0079] Step 5:

[0080] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[0081] Step 6:

[0082] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[0083] Step 7:

[0084] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[0085] Step 8:

[0086] The server performs sentiment analysis based on the audio data, analyzing the tone of voice, pitch, rhythm of speech, etc. to assess participants' emotions, level of understanding, and degree of agreement.

[0087] Step 9:

[0088] The server generates real-time feedback based on these evaluation results, and the generated feedback is sent to the device.

[0089] Step 10:

[0090] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[0091] Step 11:

[0092] After the meeting, the server generates a detailed feedback report based on all the evaluation data, and provides the report to the user.

[0093] Step 12:

[0094] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Many modern corporate meetings require increased efficiency and productivity. However, in traditional meetings, it is difficult to evaluate the quality of communication between participants, their level of understanding, and the appropriateness of visual support in real time and provide immediate feedback. Furthermore, creating detailed evaluation reports after meetings and creating follow-up plans for specific participants can be time-consuming and laborious, leading to delays in implementation. A system that can effectively resolve these issues and improve the quality of meetings is needed.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, means for setting and recording the type of meeting, and means for using natural language processing technology and image recognition technology to evaluate the voice and visual data. This makes it possible to provide effective feedback in real time during meetings and quickly create a detailed evaluation report after the meeting ends, thereby improving the overall quality and productivity of meetings.

[0100] "Means for collecting audio data" refers to a function in which the device uses a microphone to record audio during a meeting in real time and acquires this audio as data.

[0101] The "means for converting into text in real time" is a function that uses voice recognition technology to instantly convert collected voice data into text format.

[0102] The "means for analyzing the converted character string data" is a function that uses natural language processing technology to analyze text data and evaluate the quality and effectiveness of communication.

[0103] "Means for capturing and collecting visual support materials" refers to the ability to record and collect presentation materials and visual support using a camera or screen capture function.

[0104] The "means for evaluating the effectiveness" is a function that uses image recognition technology to analyze collected visual support materials and evaluate their visual effectiveness and readability.

[0105] "Means for performing emotion analysis based on audio data" is a function that analyzes the tone, pitch, rhythm, etc. of the voice from collected audio data to evaluate the emotions and level of understanding of participants.

[0106] "Means for assessing participants' level of understanding and satisfaction" is a function that evaluates how well each participant understood and agreed with the meeting content based on the results of sentiment analysis.

[0107] "Means for providing feedback in real time" refers to a function that instantly generates feedback based on the analyzed evaluation results and notifies the user's device.

[0108] The "means for generating a detailed feedback report after the meeting" is a function for generating a report including detailed evaluation results after the meeting based on all evaluation data during the meeting.

[0109] The "means for setting and recording the type of meeting" is a function that allows the user to select and set the type of meeting and record that information in the database.

[0110] "Natural language processing technology" is a technology for analyzing text data and understanding meaning and context, and for evaluating communication based on the analysis results.

[0111] "Image recognition technology" is a technology for analyzing image data and evaluating the characteristics and effects of visual information.

[0112] The system of the present invention is designed to improve the efficiency and productivity of corporate meetings by providing voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan suggestions.

[0113] 1. User operations

[0114] The user first selects the type of meeting on their device and registers the information with the server. For example, if the user selects "presentation," the server loads the corresponding template and settings. Once the settings are complete, the user can start the meeting as usual.

[0115] 2. Audio data collection and conversion

[0116] As the meeting progresses, the device uses its microphone to collect real-time audio data and transmits it to a server, which then converts the collected audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, which stores the converted text data for analysis.

[0117] 3. Communication Evaluation

[0118] The server analyzes the stored text data using natural language processing (NLP) technology, such as SpaCy or Google NLP. This analysis evaluates the appropriateness of the explanation, the appropriateness of the language, and the clarity of the communication. For example, it can generate feedback such as "There are not enough specific examples."

[0119] 4. Evaluating visual support

[0120] When a user uses presentation materials, the device captures the materials using the camera or screen capture function. The captured image data is sent to a server, which then evaluates it using image recognition technology such as Google Cloud Vision API. For example, the server generates feedback such as, "The font on the slides is small and difficult to read."

[0121] 5. Sentiment analysis

[0122] The server analyzes the tone, pitch, rhythm, and other aspects of the voice data and performs emotional analysis, which evaluates participants' emotions and intelligibility and can generate alerts such as "a specific participant may have a question."

[0123] 6. Real-time feedback

[0124] The server generates real-time feedback based on the analysis results and sends it to the device. Users can check this feedback during the meeting and take immediate action. For example, feedback such as "Please make the font larger on the slides" may be displayed on the device.

[0125] 7. Generate detailed feedback reports

[0126] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants. Users can receive the report in PDF format or other formats.

[0127] Specific examples

[0128] 1. Setup and Start

[0129] The user sets up a "presentation" on the device and starts a meeting, and the server loads the relevant template and completes the setup.

[0130] 2. Collection of audio data

[0131] When a user gives a presentation, the device collects the audio in real time and sends it to the server, which converts the audio data into text and stores it.

[0132] 3. Communication evaluation and sentiment analysis

[0133] The server analyzes the stored text data using NLP technology and generates feedback such as "You should use more concrete examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0134] 4. Evaluating visual support

[0135] When a user is using a slide deck, the device collects images of the slide deck and sends them to the server, which evaluates the font size, color usage, etc. and generates feedback such as "The font on the slide deck is small and difficult to read."

[0136] 5. Real-time feedback

[0137] The feedback generated by the server is sent to the terminal in real time, and the user can check the content and implement improvements during the meeting.

[0138] 6. Generate detailed feedback reports

[0139] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user.

[0140] This system can significantly improve the efficiency and productivity of meetings.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The user sets the meeting type on the device. Specifically, the user launches the device's application, selects the appropriate meeting type from "Presentation," "1-on-1," or "Regular Meeting," and sends the information to the server. At this time, the device receives the meeting type and other related setting information (e.g., scheduled time, participant list, etc.) as input data and sends it to the server. Based on the received data, the server loads the templates and tools required for the meeting.

[0144] Step 2:

[0145] During meetings, the device uses a microphone to collect voice data in real time. Specifically, the device's microphone records what is being said and sends the voice data sequentially to the server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API or similar. In this process, speech recognition technology is applied to the input voice data to generate text data. The converted text data is stored on the server for later analysis.

[0146] Step 3:

[0147] The server analyzes the stored text data using natural language processing (NLP) technology. Specifically, it receives the text data as input and evaluates the appropriateness of the explanation, the appropriateness of the words, the clarity of the communication, and so on. For example, based on identified keywords and phrases, it generates feedback such as "There are not enough specific examples" or "There is too much technical terminology." This feedback information is output as the analysis result.

[0148] Step 4:

[0149] When a user uses presentation materials, the device collects the materials using the device's camera or screen capture function. Specifically, the device's camera takes a picture of the slides or printed materials, or the screen capture function captures the digital materials. The image data of these visual support materials is sent as input to a server. The server then uses image recognition technology such as Google Cloud Vision API to analyze the image data and evaluate the visual support. For example, specific feedback such as "The font on the slides is small and difficult to read" is output.

[0150] Step 5:

[0151] The server performs sentiment analysis on the collected voice data. Specifically, it receives the voice data as input and analyzes the tone, pitch, rhythm, etc. of the voice. Using sentiment analysis technology such as IBM Watson's Tone Analyzer, it evaluates the participants' emotions and level of understanding, and outputs a judgment such as "a particular participant may have a question."

[0152] Step 6:

[0153] The server generates real-time feedback based on the analysis results and sends it to the device. Specifically, it integrates the results of the communication evaluation, the visual support evaluation, and the sentiment analysis, and immediately notifies the user of areas for improvement. The device displays this to the user in real time, and the user can review the feedback and implement improvements during the meeting. Examples of feedback include "Please make the font larger on the slides" and "Please use more specific examples."

[0154] Step 7:

[0155] After the meeting, the server generates a detailed feedback report based on all the evaluation data. Specifically, it integrates the results of the voice-to-text conversion, communication evaluation, visual support evaluation, and sentiment analysis to create the report. The report is generated in a format such as PDF and sent to the user via email or cloud storage service. The report also includes a follow-up plan for specific participants.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Conventional meeting systems make it difficult to provide real-time feedback on participants' communication skills, the appropriateness of visual support, their emotions, and their level of understanding. Real-time feedback also lacks the ability to provide immediate improvement measures, preventing participants from making improvements on the spot. Furthermore, the lack of a function to generate detailed feedback reports and follow-up plans for specific participants prevents meetings from being efficiently and productively implemented.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, and means for displaying the feedback and evaluation results on the smart glasses in real time, thereby enabling improved communication during meetings, appropriate use of visual support, understanding of participants' emotions and levels of understanding, providing immediate feedback, and generating a detailed feedback report and follow-up plan.

[0161] "Audio data" is information in which audio signals are recorded in digital format, and is basic data for using collected audio for analysis and processing.

[0162] A "character string" is data that is generated by analyzing voice data using voice recognition technology and expressing it in a text format that can be understood as language.

[0163] "Communication evaluation" is the process of analyzing the text data converted from audio data to evaluate the content of participants' statements, the validity of their explanations, the appropriateness of their words, and whether or not they communicated clearly.

[0164] "Visual support materials" are visual content such as slides, charts, and images used to support a presentation or explanation.

[0165] "Sentiment analysis" is the process of analyzing voice tone, pitch, and speaking rhythm based on audio data to assess participants' emotions, understanding, and agreement.

[0166] "Real-time feedback" refers to feedback information provided to participants immediately during a meeting based on the evaluation results, allowing participants to implement improvement measures on the spot.

[0167] A "feedback report" is a detailed report generated after a meeting that includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and a follow-up plan.

[0168] "Smart glasses" are glasses-type devices that have the ability to overlay digital information onto virtual and real space, allowing users to check the information in real time.

[0169] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[0170] System configuration

[0171] The system uses smart glasses to collect and convert voice data, evaluate communication, analyze sentiment, provide real-time feedback, generate detailed feedback reports, evaluate visual support, and display feedback.

[0172] 1. User operations

[0173] The user uses a device connected to the smart glasses to set the type of meeting (presentation, one-on-one, regular meeting, etc.) and register it on the server. Once the meeting starts, the user only needs to conduct regular conversations and presentations.

[0174] 2. Audio data collection and conversion

[0175] The smart glasses' microphone collects voice data in real time and sends it to a cloud server, which then converts the data into text using a voice recognition system.

[0176] 3. Communication Evaluation

[0177] The server analyzes the text data and uses natural language processing (NLP) techniques to evaluate the communication, including whether the explanation is clear, the words used are appropriate, and the delivery style is easy for participants to understand.

[0178] 4. Sentiment analysis

[0179] The server performs sentiment analysis on the audio data, analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, understanding, and satisfaction, and generates an alert if a specific participant may have a question.

[0180] 5. Evaluating visual support

[0181] When a user uses presentation materials, the smart glasses use their built-in camera and screen capture function to collect the materials and send them to a server, which then uses image recognition technology to evaluate the materials' readability and visual effect.

[0182] 6. Real-time feedback

[0183] The server generates real-time feedback based on the evaluation results, and users can check the feedback displayed on their smart glasses and immediately implement improvements during the meeting.

[0184] 7. Generate detailed feedback reports

[0185] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0186] Specific examples

[0187] For example, when presenting emergency response measures at a security meeting, the following steps are taken:

[0188] 1. Setup and Start

[0189] The user puts on the smart glasses, sets up an "emergency response presentation" on the device, and starts the meeting.

[0190] 2. Collection of audio data

[0191] When a user gives a presentation, the microphone in the smart glasses collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0192] 3. Communication evaluation and sentiment analysis

[0193] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0194] 4. Evaluating visual support

[0195] When a user is using a slide, the camera in the smart glasses collects images of the slide and sends them to a server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[0196] 5. Real-time feedback

[0197] The feedback generated by the server is sent to the smart glasses in real time, allowing users to review the feedback and take immediate action during the meeting.

[0198] 6. Generate detailed feedback reports

[0199] Once the presentation is over, the server generates and provides a detailed feedback report to the user, including the adequacy of the explanation, clarity of communication, visual support, sentiment analysis results, and a follow-up plan for specific participants.

[0200] Prompt Sentence Examples

[0201] By using prompts like the following, you can get specific feedback from the generative AI model based on its analysis results:

[0202] Example prompt sentence:

[0203] "Please rate the following meeting:

[0204] 1. The explanation was easy to understand.

[0205] 2. The points were clearly communicated.

[0206] 3. Participants responded actively.

[0207] Please take these factors into consideration and provide a specific assessment and suggestions for improvement."

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] Audio data collection

[0211] The user puts on the smart glasses and starts the meeting. The microphone in the smart glasses connected to the device collects audio data in real time and transmits it to the server.

[0212] <input>: what the user says

[0213] <Output>: Audio data

[0214] <Specific operation>: The microphone built into the smart glasses captures audio signals and transmits them to the device via wireless communication.

[0215] Step 2:

[0216] Converting audio data to a string

[0217] The server converts the voice data collected in step 1 into a string of text in real time using a speech recognition system (e.g., Google Speech-to-Text API).

[0218] <Input>: Audio data

[0219] <Output>: Converted string data

[0220] <Specific operation>: The server calls the speech recognition API and converts the speech data into text data. The converted text is stored in a database.

[0221] Step 3:

[0222] Communication Evaluation

[0223] The server analyzes the string data converted in step 2 using natural language processing (NLP) technology (e.g., the Transformers library) and evaluates the communication.

[0224] <Input>: String data

[0225] <Output>: Communication evaluation results

[0226] <Specific operation>: The server uses the NLP model to check the grammar of the text data, extract keywords, and evaluate the clarity and appropriateness of the explanations, and generates the results as feedback.

[0227] Step 4:

[0228] sentiment analysis

[0229] The server performs sentiment analysis based on the voice data from step 1 and the text data from step 2. It analyzes the tone, pitch, and rhythm of the voice to evaluate the participants' emotions, level of understanding, and degree of agreement.

[0230] <Input>: Audio data and text data

[0231] <Output>:Sentiment analysis results

[0232] <Specific operation>: The server uses a voice analysis algorithm to analyze the emotional tone and pitch fluctuations of the voice and identify the emotional state based on that.

[0233] Step 5:

[0234] Evaluating visual support

[0235] When a user uses presentation materials, the smart glasses' camera and screen capture function are used to collect the materials and send them to the server, which then uses image recognition technology (e.g., OpenCV) to evaluate the readability and visual effect of the materials.

[0236] <Input>: Image data of presentation materials

[0237] <Output>: Visual support evaluation results

[0238] <Specific operation>: The camera in the smart glasses takes a photo of the document and sends the image to the server, which uses an image processing algorithm to evaluate the font size and color of the text.

[0239] Step 6:

[0240] Generate real-time feedback

[0241] The server generates real-time feedback based on the evaluation results of steps 3, 4, and 5 and notifies the smart glasses, allowing the user to check the feedback on the spot and take corrective measures.

[0242] <Input>: Communication evaluation results, emotion analysis results, visual support evaluation results

[0243] <Output>: Real-time feedback

[0244] <Specific operation>: The server consolidates the evaluation results, generates feedback messages containing important points and improvement measures, and displays them on the smart glasses display.

[0245] Step 7:

[0246] Generate detailed feedback reports

[0247] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0248] <input>: All evaluation data

[0249] <output>:Detailed feedback report

[0250] <Specific operation>: The server aggregates all evaluation results and generates a detailed feedback report based on the report template. The report is provided to the user in a format such as PDF.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, emotion analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Furthermore, this system is combined with an emotion engine that recognizes the user's emotions, allowing the emotion data to be reflected in the feedback in real time. Specific embodiments will be described below.

[0253] System Configuration

[0254] 1. User operations

[0255] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[0256] 2. Audio data collection and conversion

[0257] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[0258] 3. Communication Evaluation

[0259] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[0260] 4. Evaluating visual support

[0261] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[0262] 5. Sentiment analysis using an emotion engine

[0263] The server includes an emotion engine for analyzing emotions based on voice data. Specifically, the emotion engine analyzes the tone of voice, pitch, rhythm of speech, etc., to evaluate participants' emotions, level of understanding, and degree of agreement. The emotion engine also analyzes image data for more accurate emotion recognition.

[0264] 6. Real-time feedback

[0265] The server generates real-time feedback based on the evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback and immediately implement improvements during the meeting.

[0266] 7. Generate detailed feedback reports and follow-up plans

[0267] After the meeting, the server generates a detailed feedback report based on all the evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants, allowing users to obtain specific improvement suggestions for the next meeting.

[0268] Specific examples

[0269] For 1-on-1 meetings

[0270] 1. Setup and Start

[0271] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[0272] 2. Collection of audio data

[0273] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0274] 3. Communication evaluation and sentiment analysis

[0275] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[0276] 4. Evaluating visual support

[0277] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[0278] 5. Real-time feedback

[0279] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[0280] 6. Generate detailed feedback reports

[0281] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for any anxious employees.

[0282] In this way, the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[0283] The processing flow will be explained below.

[0284] Step 1:

[0285] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[0286] Step 2:

[0287] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[0288] Step 3:

[0289] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[0290] Step 4:

[0291] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[0292] Step 5:

[0293] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[0294] Step 6:

[0295] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[0296] Step 7:

[0297] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[0298] Step 8:

[0299] The server uses an emotion engine to analyze both audio and visual data to analyze participants' emotions, including tone of voice, pitch, rhythm of speech, and facial expressions, to assess participants' emotions, level of understanding, and degree of agreement, achieving more accurate emotion recognition.

[0300] Step 9:

[0301] The server generates real-time feedback based on these evaluation results and emotion data, and the generated feedback is sent to the device.

[0302] Step 10:

[0303] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[0304] Step 11:

[0305] After the meeting, the server generates a detailed feedback report based on all the evaluation and emotion data, and provides the report to the user.

[0306] Step 12:

[0307] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[0308] Example 2

[0309] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] Conventional meeting management systems have difficulty in providing real-time feedback or generating detailed feedback reports, limiting their ability to improve meeting efficiency and productivity. Furthermore, they have been unable to accurately grasp participants' emotions and levels of understanding and immediately implement improvement measures based on that information, making it difficult to ensure effective communication.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for collecting audio data and converting it into text in real time, a means for analyzing the converted text data and evaluating the information transmission, a means for capturing and collecting image materials and evaluating their effectiveness, a means for performing emotion analysis based on the audio data and evaluating the participants' levels of understanding and satisfaction, a means for providing opinions in real time based on these evaluation results, and a means for generating a detailed opinion report after the meeting. This enables real-time feedback during the meeting and optimization of communication taking into account the participants' emotions and levels of understanding. Furthermore, generating a detailed feedback report can suggest specific improvement measures for the next meeting.

[0312] "Sound data" refers to digital or analog signals collected as sound.

[0313] A "character string" refers to a series of characters or words that are output as a result of analyzing sound data.

[0314] "Communication evaluation" refers to the process of analyzing and evaluating the quality, content appropriateness, and clarity of communication.

[0315] "Image material" refers to digital or analog image data containing visual information, such as presentations or whiteboards.

[0316] "Emotion analysis" refers to the process of analyzing participants' emotions and psychological states from audio and image data.

[0317] "Real-time feedback" refers to evaluations and opinions that are instantly communicated to users based on the results of analysis of sound and image data.

[0318] An "opinion report" is a document that details the analysis of a meeting, including an evaluation of communication, visual support, and sentiment analysis results.

[0319] A "follow-up plan" is a plan that specifies the methods and steps for follow-up if necessary for specific participants based on the evaluation data.

[0320] "User's information processing device" refers to an electronic device, such as a personal computer or smartphone, used to receive feedback from the system.

[0321] The system of the present invention collects audio data, converts it into text, evaluates the communication of information, collects and evaluates visual materials, analyzes emotions, provides real-time feedback, generates detailed opinion reports, and presents follow-up plans to improve the efficiency and productivity of corporate meetings. Specific embodiments are described below.

[0322] System Configuration

[0323] This system mainly consists of three elements: a server, a terminal, and a user.

[0324] User operations

[0325] Users use their terminals to set the type of meeting (presentation, one-on-one, regular gathering, etc.) and register it with the server. After the meeting starts, users can have regular conversations or make presentations.

[0326] Acoustic data collection and conversion

[0327] During a meeting, the device collects audio data in real time through the microphone and sends it to the server. The server converts the received audio data into text using a speech recognition system, such as the Google Speech-to-Text API. This converted text data is then stored in a database on the server.

[0328] Evaluation of information transmission

[0329] The server analyzes the converted text data using natural language processing (NLP) technology, such as SpaCy or Google NLP API, to evaluate the quality of the communication, the appropriateness of the content, and clarity. The evaluation results are saved as feedback data, such as the "appropriateness of the explanation" and "appropriateness of the words."

[0330] Collection and evaluation of image materials

[0331] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server. The server then uses image recognition technology such as Google Cloud Vision API to evaluate the readability and visual effect of the visual support materials. For example, it generates feedback such as "The figures are complex and difficult to read."

[0332] Emotion analysis

[0333] The server analyzes the audio data using emotion engines such as IBM Watson Tone Analyzer. It evaluates participants' emotions, level of understanding, and degree of satisfaction based on the tone of voice, pitch, and rhythm of speech. It also analyzes image data to achieve more accurate emotion recognition. The analysis results are saved as feedback, such as "a particular employee may be feeling anxious."

[0334] Real-time feedback

[0335] The server generates real-time feedback based on all evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback during the meeting and immediately implement any necessary improvements.

[0336] Present a detailed opinion report and follow-up plan

[0337] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and a follow-up plan for specific participants. This detailed feedback report provides specific improvement suggestions for the next meeting.

[0338] Example: 1-on-1 meeting

[0339] 1. Setup and Start

[0340] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[0341] 2. Acoustic data collection

[0342] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0343] 3. Evaluation of information transmission and emotion analysis

[0344] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[0345] 4. Evaluation of image materials

[0346] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[0347] 5. Real-time feedback

[0348] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[0349] 6. Generate a detailed opinion report

[0350] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for anxious employees.

[0351] Prompt Sentence Examples

[0352] Below is an example of a prompt sentence to input to the generative AI model.

[0353] Prepare a detailed feedback report for your 1:1 meetings, including a communication evaluation, visual support evaluation, sentiment analysis results, and a plan to follow up with anxious employees based on the following data:

[0354] Audio Data Text: "I feel like I didn't explain this project well enough."

[0355] Visual support: "The diagram on the whiteboard was complicated."

[0356] Sentiment analysis results: "Certain employees may be feeling anxious."

[0357] In this way, the system of the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[0358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0359] Step 1: Set up users and start a meeting

[0360] How it works: A user turns on their device, opens the Meetings application, selects the type of meeting (presentation, one-on-one, regular get-together, etc.), and configures it.

[0361] Input: Enter the meeting type and participant information.

[0362] Data processing: Format the entered meeting information into the appropriate format.

[0363] Output: Sends the configured meeting information to the server, which stores it in the database and completes the meeting preparation.

[0364] Step 2: Collecting sound data and sending it to the server

[0365] Action: A user presses the "Start Meeting" button to start a meeting. The device's microphone collects audio data.

[0366] Input: User's voice.

[0367] Data processing: Collect sound data in real time and convert it into an appropriate format.

[0368] Output: Sends sound data to the server.

[0369] Step 3: Converting sound data into a string

[0370] How it works: The server converts the received audio data into a string using a speech recognition system (e.g., Google Speech-to-Text API).

[0371] Input: Sound data.

[0372] Data processing: Converting sound data into text using a voice recognition system.

[0373] Output: The converted character data is saved in a database on the server.

[0374] Step 4: Analyze text data and evaluate information transmission

[0375] How it works: The server analyzes the converted text data using natural language processing (NLP) technology (e.g., SpaCy or Google NLP API) to evaluate the quality of the communication, the appropriateness of the content, and clarity.

[0376] Input: The converted character data.

[0377] Data processing: Analyzing text data using NLP techniques and calculating specific evaluation indicators (e.g., appropriateness, clarity, etc.).

[0378] Output: The evaluation results are saved as feedback data on the server.

[0379] Step 5: Collecting image data and sending it to the server

[0380] What it does: If a user is using a presentation, the device will capture the presentation using the camera or screen capture function.

[0381] Input: Visual materials (presentation slides, whiteboard images, etc.).

[0382] Data processing: Converting collected images into the appropriate format.

[0383] Output: Send the image data to the server.

[0384] Step 6: Analysis and evaluation of imagery

[0385] How it works: The server analyzes the image data using image recognition technology such as the Google Cloud Vision API to evaluate the readability and visual effectiveness of the visual support materials.

[0386] Input: Collected image data.

[0387] Data processing: Image data is analyzed using image recognition technology to calculate visual evaluation indicators.

[0388] Output: Evaluation results are generated and stored as feedback data on the server.

[0389] Step 7: Analyze the sentiment data

[0390] How it works: The server analyzes the audio data using an emotion engine (e.g., IBM Watson Tone Analyzer) and evaluates the participants' emotions, level of understanding, and degree of agreement based on the tone, pitch, and rhythm of their voices.

[0391] Input: Sound data.

[0392] Data processing: Analyzes sound data using an emotion engine to evaluate emotions and comprehension.

[0393] Output: The sentiment analysis results are saved as feedback data on the server.

[0394] Step 8: Generate and communicate real-time feedback

[0395] How it works: The server generates real-time feedback based on all evaluation results and emotion data, and sends the generated feedback to the device.

[0396] Input: Various evaluation results and sentiment analysis results.

[0397] Data processing: Various evaluation results and sentiment analysis results are combined to generate feedback.

[0398] Output: Feedback data is sent to the device, where the user can review it and take any necessary corrective action.

[0399] Step 9: Generate and submit a detailed opinion report

[0400] How it works: After the meeting, the server generates a detailed opinion report based on all the evaluation data.

[0401] Input: All rating data and sentiment analysis results.

[0402] Data processing: Compile various evaluation data to prepare a detailed opinion report.

[0403] Output: Provide users with a feedback report with concrete improvements for the next meeting.

[0404] The above is a concrete explanation of the processing steps of this system.

[0405] (Application example 2)

[0406] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] In conventional factory meetings, there are insufficient means to evaluate the quality of communication, the effectiveness of visual support materials, and the emotions and satisfaction of participants in real time, making it difficult to improve productivity and provide efficient feedback.In addition, factory robots cannot adaptively control their operations based on the content of meetings and feedback results, which limits the improvement of productivity throughout the factory.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting voice data and converting it into text in real time; means for analyzing the converted text data and evaluating communication; means for photographing and collecting visual support materials and evaluating their effectiveness; means for performing emotion analysis based on the voice data and evaluating participants' levels of understanding and satisfaction; means for providing real-time feedback based on these evaluation results; means for generating a detailed feedback report after the meeting; means for improving productivity within the factory and the quality of human communication; and means for adaptively controlling the operation of factory robots using the analyzed data in real time. This makes it possible to improve the quality of communication and the effectiveness of visual support, accurately evaluate participants' emotions and satisfaction, and provide immediate feedback. Furthermore, factory robots can also adaptively control their operation based on the meeting results, improving the productivity of the entire factory.

[0409] "Audio Data" means voice information collected through a voice input device such as a microphone.

[0410] "Real-time" means having a time response where data is collected, processed, and fed back immediately.

[0411] "String" refers to data that has been converted from audio data into text format.

[0412] "Communication evaluation" refers to the act of analyzing the converted text data and evaluating the appropriateness of the communication, clarity, and validity of the content.

[0413] "Visual support materials" means materials that visually support information, such as presentation materials and whiteboards.

[0414] "Effectiveness of visual support" refers to evaluating how effective visual support materials are in conveying information.

[0415] "Sentiment analysis" refers to a technology that analyzes participants' emotions based on audio data and evaluates their level of understanding and agreement.

[0416] "Real-time feedback" means providing immediate feedback based on collected and analyzed data.

[0417] A "detailed feedback report" refers to a report generated after a meeting that includes communication evaluation results, sentiment analysis results, specific improvement suggestions, etc.

[0418] "Productivity improvement" means optimizing factory production processes to increase efficiency and results.

[0419] "Quality of communication" refers to indicators that reflect the clarity, appropriateness, and level of understanding of information exchange.

[0420] "Data analysis" refers to the act of analyzing collected data and extracting valuable information.

[0421] "Factory robot motion control" refers to the technology that manages the movements and actions of robots in factories to accomplish designated tasks.

[0422] System Configuration

[0423] The present invention provides a system for making meetings in a factory more efficient and improving productivity. The configuration and operation of the system for carrying out the invention will be described below.

[0424] 1. Hardware Configuration

[0425] This system uses the following hardware:

[0426] Microphone: A device for collecting audio data.

[0427] Camera: A device for capturing visual support materials.

[0428] Robot body: A device that moves around the factory and collects, analyzes, and provides feedback on various data.

[0429] Display: A device for displaying feedback and reports.

[0430] 2. Software Configuration

[0431] The system software utilizes the following technologies:

[0432] Speech recognition: Converts collected voice data into text in real time using the Google Cloud Speech-to-Text API.

[0433] Natural language processing: Using generative AI models such as GPT-4, the converted text data is analyzed and communication is evaluated.

[0434] Image Recognition: Analyze visual support materials using OpenCV and evaluate their effectiveness.

[0435] Sentiment analysis: IBM Watson Tone Analyzer is used to analyze voice tone and pitch to assess participants' emotions.

[0436] Real-time data processing and communication: Process data in real time and provide feedback using the MQTT protocol.

[0437] Processing flow

[0438] 1. Data Collection

[0439] When a user starts a meeting, the robot's microphone and camera collect audio and visual support material, which is then transmitted in real time to a server.

[0440] 2. Speech recognition and conversion

[0441] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text format.

[0442] 3. Natural Language Processing (NLP)

[0443] The converted text data is analyzed using GPT-4 to evaluate the quality of communication.

[0444] 4. Visual Support Assessment

[0445] Use image recognition with OpenCV to evaluate the content of presentation materials and whiteboards.

[0446] 5. Sentiment analysis

[0447] IBM Watson Tone Analyzer analyzes the tone and pitch of audio data to assess participants' emotions in real time.

[0448] 6. Real-time feedback generation

[0449] The server generates real-time feedback based on the analysis results and displays it on a display or mobile device.

[0450] 7. Generate detailed feedback reports

[0451] After the meeting, the server generates a detailed feedback report and provides it to the user.

[0452] Specific examples

[0453] A meeting to improve production line efficiency is held in a factory conference room. When a user starts a meeting, a robot collects audio and whiteboard content. The speech is then converted into text using the Google Cloud Speech-to-Text API, and GPT-4 analyzes the content of the communication. OpenCV also evaluates the diagrams and text on the whiteboard, and IBM Watson Tone Analyzer analyzes the emotions of the participants. Based on this data, real-time feedback is displayed on the screen, and any necessary improvements are immediately implemented. After the meeting, a detailed feedback report is generated, providing suggestions for improvement for the next meeting.

[0454] Prompt Sentence Examples

[0455] Analyze the following text data, evaluate the quality of communication, visual support, and analyze sentiment to generate real-time feedback, and generate a detailed feedback report after the meeting.

[0456] Text data:

[0457] Participant A: How can we increase production speed?

[0458] Participant B: It's impossible unless we hire more staff.

[0459] Participant C: No, if we optimize the process, we may be able to manage without increasing manpower.

[0460] Visual Data:

[0461] (Whiteboard image)

[0462] Output formats:

[0463] 1. Communication Evaluation

[0464] 2. Visual Support Evaluation

[0465] 3. Sentiment analysis results

[0466] 4. Real-time feedback

[0467] 5. Detailed feedback report

[0468] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0469] Step 1:

[0470] When a user starts a meeting, the device's microphone and camera begin to collect audio and visual support data. The audio and video data are collected in real time and sent to the server. The input is the audio and video data, and the output is the collected raw data.

[0471] Step 2:

[0472] The server uses the Google Cloud Speech-to-Text API to convert the transmitted voice data into text in real time. This conversion process converts the phonological information of the voice into text data. The input is voice data, and the output is text data converted from the voice into text format.

[0473] Step 3:

[0474] The server uses a generative AI model such as GPT-4 to analyze the converted text data. Specifically, it uses natural language processing (NLP) technology to evaluate the content and quality of the communication, determining factors such as clarity of explanation, appropriateness to the question, and use of specific examples. The input is text data, and the output is the evaluation result of the communication.

[0475] Step 4:

[0476] The server uses OpenCV to analyze the visual support materials (image data) sent from the device. This includes the readability of the images and the placement of figures and text. The analysis results are used to evaluate the effectiveness of the visual support. The input is the image data, and the output is the evaluation result of the visual support.

[0477] Step 5:

[0478] The server uses IBM Watson Tone Analyzer to analyze the tone and pitch of the audio data and perform sentiment analysis. This analysis evaluates participants' emotions, understanding, and degree of agreement. The input is the audio data, and the output is the sentiment analysis results.

[0479] Step 6:

[0480] The server generates real-time feedback based on these evaluation results and displays it on the terminal display or mobile device. This feedback includes immediately actionable improvement measures. The input is communication evaluation, visual support evaluation, and sentiment analysis data, and the output is real-time feedback.

[0481] Step 7:

[0482] After the meeting, the server generates a detailed feedback report based on all the evaluation data. This report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and specific improvement suggestions for the next meeting. The input is all the evaluation data, and the output is a detailed feedback report.

[0483] Step 8:

[0484] The server adaptively controls the factory robot's operation based on the data analyzed in real time. This includes adjusting the factory robot's operation patterns and work procedures based on the meeting content and feedback results. The input is the meeting analysis data and feedback results, and the output is the factory robot's operation control commands.

[0485] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0486] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0487] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0488] [Second embodiment]

[0489] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0490] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0491] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0492] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0493] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0494] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0495] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0496] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0497] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0498] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0499] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0500] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0501] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[0502] System Configuration

[0503] 1. User operations

[0504] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[0505] 2. Audio data collection and conversion

[0506] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[0507] 3. Communication Evaluation

[0508] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[0509] 4. Evaluating visual support

[0510] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[0511] 5. Sentiment analysis

[0512] The server performs sentiment analysis based on the audio data, specifically analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, level of understanding, and degree of agreement.

[0513] 6. Real-time feedback

[0514] The server generates real-time feedback based on the evaluation results and sends it to the device, where users can check the feedback and immediately implement improvements during the meeting.

[0515] 7. Generate detailed feedback reports

[0516] After the meeting, the server generates a detailed feedback report based on all the evaluation data and provides it to the user, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0517] Specific examples

[0518] For presentations

[0519] 1. Setup and Start

[0520] The user sets up a "presentation" on the device and starts the meeting, the server loads the template, and the setup is complete.

[0521] 2. Collection of audio data

[0522] When a user gives a presentation, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0523] 3. Communication evaluation and sentiment analysis

[0524] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0525] 4. Evaluating visual support

[0526] When a user is using a slide, the device collects images of the slide and sends them to the server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[0527] 5. Real-time feedback

[0528] The server generates feedback and sends it to the device in real time, allowing users to review the feedback and implement improvements during the meeting.

[0529] 6. Generate detailed feedback reports

[0530] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user, including the adequacy of the explanation, clarity of communication, visual support, and sentiment analysis results.

[0531] In this way, the present invention can greatly improve the efficiency and productivity of meetings.

[0532] The processing flow will be explained below.

[0533] Step 1:

[0534] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[0535] Step 2:

[0536] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[0537] Step 3:

[0538] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[0539] Step 4:

[0540] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[0541] Step 5:

[0542] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[0543] Step 6:

[0544] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[0545] Step 7:

[0546] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[0547] Step 8:

[0548] The server performs sentiment analysis based on the audio data, analyzing the tone of voice, pitch, rhythm of speech, etc. to assess participants' emotions, level of understanding, and degree of agreement.

[0549] Step 9:

[0550] The server generates real-time feedback based on these evaluation results, and the generated feedback is sent to the device.

[0551] Step 10:

[0552] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[0553] Step 11:

[0554] After the meeting, the server generates a detailed feedback report based on all the evaluation data, and provides the report to the user.

[0555] Step 12:

[0556] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[0557] Example 1

[0558] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0559] Many modern corporate meetings require increased efficiency and productivity. However, in traditional meetings, it is difficult to evaluate the quality of communication between participants, their level of understanding, and the appropriateness of visual support in real time and provide immediate feedback. Furthermore, creating detailed evaluation reports after meetings and creating follow-up plans for specific participants can be time-consuming and laborious, leading to delays in implementation. A system that can effectively resolve these issues and improve the quality of meetings is needed.

[0560] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0561] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, means for setting and recording the type of meeting, and means for using natural language processing technology and image recognition technology to evaluate the voice and visual data. This makes it possible to provide effective feedback in real time during meetings and quickly create a detailed evaluation report after the meeting ends, thereby improving the overall quality and productivity of meetings.

[0562] "Means for collecting audio data" refers to a function in which the device uses a microphone to record audio during a meeting in real time and acquires this audio as data.

[0563] The "means for converting into text in real time" is a function that uses voice recognition technology to instantly convert collected voice data into text format.

[0564] The "means for analyzing the converted character string data" is a function that uses natural language processing technology to analyze text data and evaluate the quality and effectiveness of communication.

[0565] "Means for capturing and collecting visual support materials" refers to the ability to record and collect presentation materials and visual support using a camera or screen capture function.

[0566] The "means for evaluating the effectiveness" is a function that uses image recognition technology to analyze collected visual support materials and evaluate their visual effectiveness and readability.

[0567] "Means for performing emotion analysis based on audio data" is a function that analyzes the tone, pitch, rhythm, etc. of the voice from collected audio data to evaluate the emotions and level of understanding of participants.

[0568] "Means for assessing participants' level of understanding and satisfaction" is a function that evaluates how well each participant understood and agreed with the meeting content based on the results of sentiment analysis.

[0569] "Means for providing feedback in real time" refers to a function that instantly generates feedback based on the analyzed evaluation results and notifies the user's device.

[0570] The "means for generating a detailed feedback report after the meeting" is a function for generating a report including detailed evaluation results after the meeting based on all evaluation data during the meeting.

[0571] The "means for setting and recording the type of meeting" is a function that allows the user to select and set the type of meeting and record that information in the database.

[0572] "Natural language processing technology" is a technology for analyzing text data and understanding meaning and context, and for evaluating communication based on the analysis results.

[0573] "Image recognition technology" is a technology for analyzing image data and evaluating the characteristics and effects of visual information.

[0574] The system of the present invention is designed to improve the efficiency and productivity of corporate meetings by providing voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan suggestions.

[0575] 1. User operations

[0576] The user first selects the type of meeting on their device and registers the information with the server. For example, if the user selects "presentation," the server loads the corresponding template and settings. Once the settings are complete, the user can start the meeting as usual.

[0577] 2. Audio data collection and conversion

[0578] As the meeting progresses, the device uses its microphone to collect real-time audio data and transmits it to a server, which then converts the collected audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, which stores the converted text data for analysis.

[0579] 3. Communication Evaluation

[0580] The server analyzes the stored text data using natural language processing (NLP) technology, such as SpaCy or Google NLP. This analysis evaluates the appropriateness of the explanation, the appropriateness of the language, and the clarity of the communication. For example, it can generate feedback such as "There are not enough specific examples."

[0581] 4. Evaluating visual support

[0582] When a user uses presentation materials, the device captures the materials using the camera or screen capture function. The captured image data is sent to a server, which then evaluates it using image recognition technology such as Google Cloud Vision API. For example, the server generates feedback such as, "The font on the slides is small and difficult to read."

[0583] 5. Sentiment analysis

[0584] The server analyzes the tone, pitch, rhythm, and other aspects of the voice data and performs emotional analysis, which evaluates participants' emotions and intelligibility and can generate alerts such as "a specific participant may have a question."

[0585] 6. Real-time feedback

[0586] The server generates real-time feedback based on the analysis results and sends it to the device. Users can check this feedback during the meeting and take immediate action. For example, feedback such as "Please make the font larger on the slides" may be displayed on the device.

[0587] 7. Generate detailed feedback reports

[0588] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants. Users can receive the report in PDF format or other formats.

[0589] Specific examples

[0590] 1. Setup and Start

[0591] The user sets up a "presentation" on the device and starts a meeting, and the server loads the relevant template and completes the setup.

[0592] 2. Collection of audio data

[0593] When a user gives a presentation, the device collects the audio in real time and sends it to the server, which converts the audio data into text and stores it.

[0594] 3. Communication evaluation and sentiment analysis

[0595] The server analyzes the stored text data using NLP technology and generates feedback such as "You should use more concrete examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0596] 4. Evaluating visual support

[0597] When a user is using a slide deck, the device collects images of the slide deck and sends them to the server, which evaluates the font size, color usage, etc. and generates feedback such as "The font on the slide deck is small and difficult to read."

[0598] 5. Real-time feedback

[0599] The feedback generated by the server is sent to the terminal in real time, and the user can check the content and implement improvements during the meeting.

[0600] 6. Generate detailed feedback reports

[0601] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user.

[0602] This system can significantly improve the efficiency and productivity of meetings.

[0603] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0604] Step 1:

[0605] The user sets the meeting type on the device. Specifically, the user launches the device's application, selects the appropriate meeting type from "Presentation," "1-on-1," or "Regular Meeting," and sends the information to the server. At this time, the device receives the meeting type and other related setting information (e.g., scheduled time, participant list, etc.) as input data and sends it to the server. Based on the received data, the server loads the templates and tools required for the meeting.

[0606] Step 2:

[0607] During meetings, the device uses a microphone to collect voice data in real time. Specifically, the device's microphone records what is being said and sends the voice data sequentially to the server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API or similar. In this process, speech recognition technology is applied to the input voice data to generate text data. The converted text data is stored on the server for later analysis.

[0608] Step 3:

[0609] The server analyzes the stored text data using natural language processing (NLP) technology. Specifically, it receives the text data as input and evaluates the appropriateness of the explanation, the appropriateness of the words, the clarity of the communication, and so on. For example, based on identified keywords and phrases, it generates feedback such as "There are not enough specific examples" or "There is too much technical terminology." This feedback information is output as the analysis result.

[0610] Step 4:

[0611] When a user uses presentation materials, the device collects the materials using the device's camera or screen capture function. Specifically, the device's camera takes a picture of the slides or printed materials, or the screen capture function captures the digital materials. The image data of these visual support materials is sent as input to a server. The server then uses image recognition technology such as Google Cloud Vision API to analyze the image data and evaluate the visual support. For example, specific feedback such as "The font on the slides is small and difficult to read" is output.

[0612] Step 5:

[0613] The server performs sentiment analysis on the collected voice data. Specifically, it receives the voice data as input and analyzes the tone, pitch, rhythm, etc. of the voice. Using sentiment analysis technology such as IBM Watson's Tone Analyzer, it evaluates the participants' emotions and level of understanding, and outputs a judgment such as "a particular participant may have a question."

[0614] Step 6:

[0615] The server generates real-time feedback based on the analysis results and sends it to the device. Specifically, it integrates the results of the communication evaluation, the visual support evaluation, and the sentiment analysis, and immediately notifies the user of areas for improvement. The device displays this to the user in real time, and the user can review the feedback and implement improvements during the meeting. Examples of feedback include "Please make the font larger on the slides" and "Please use more specific examples."

[0616] Step 7:

[0617] After the meeting, the server generates a detailed feedback report based on all the evaluation data. Specifically, it integrates the results of the voice-to-text conversion, communication evaluation, visual support evaluation, and sentiment analysis to create the report. The report is generated in a format such as PDF and sent to the user via email or cloud storage service. The report also includes a follow-up plan for specific participants.

[0618] (Application example 1)

[0619] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] Conventional meeting systems make it difficult to provide real-time feedback on participants' communication skills, the appropriateness of visual support, their emotions, and their level of understanding. Real-time feedback also lacks the ability to provide immediate improvement measures, preventing participants from making improvements on the spot. Furthermore, the lack of a function to generate detailed feedback reports and follow-up plans for specific participants prevents meetings from being efficiently and productively implemented.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0622] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, and means for displaying the feedback and evaluation results on the smart glasses in real time, thereby enabling improved communication during meetings, appropriate use of visual support, understanding of participants' emotions and levels of understanding, providing immediate feedback, and generating a detailed feedback report and follow-up plan.

[0623] "Audio data" is information in which audio signals are recorded in digital format, and is basic data for using collected audio for analysis and processing.

[0624] A "character string" is data that is generated by analyzing voice data using voice recognition technology and expressing it in a text format that can be understood as language.

[0625] "Communication evaluation" is the process of analyzing the text data converted from audio data to evaluate the content of participants' statements, the validity of their explanations, the appropriateness of their words, and whether or not they communicated clearly.

[0626] "Visual support materials" are visual content such as slides, charts, and images used to support a presentation or explanation.

[0627] "Sentiment analysis" is the process of analyzing voice tone, pitch, and speaking rhythm based on audio data to assess participants' emotions, understanding, and agreement.

[0628] "Real-time feedback" refers to feedback information provided to participants immediately during a meeting based on the evaluation results, allowing participants to implement improvement measures on the spot.

[0629] A "feedback report" is a detailed report generated after a meeting that includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and a follow-up plan.

[0630] "Smart glasses" are glasses-type devices that have the ability to overlay digital information onto virtual and real space, allowing users to check the information in real time.

[0631] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[0632] System configuration

[0633] The system uses smart glasses to collect and convert voice data, evaluate communication, analyze sentiment, provide real-time feedback, generate detailed feedback reports, evaluate visual support, and display feedback.

[0634] 1. User operations

[0635] The user uses a device connected to the smart glasses to set the type of meeting (presentation, one-on-one, regular meeting, etc.) and register it on the server. Once the meeting starts, the user only needs to conduct regular conversations and presentations.

[0636] 2. Audio data collection and conversion

[0637] The smart glasses' microphone collects voice data in real time and sends it to a cloud server, which then converts the data into text using a voice recognition system.

[0638] 3. Communication Evaluation

[0639] The server analyzes the text data and uses natural language processing (NLP) techniques to evaluate the communication, including whether the explanation is clear, the words used are appropriate, and the delivery style is easy for participants to understand.

[0640] 4. Sentiment analysis

[0641] The server performs sentiment analysis on the audio data, analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, understanding, and satisfaction, and generates an alert if a specific participant may have a question.

[0642] 5. Evaluating visual support

[0643] When a user uses presentation materials, the smart glasses use their built-in camera and screen capture function to collect the materials and send them to a server, which then uses image recognition technology to evaluate the materials' readability and visual effect.

[0644] 6. Real-time feedback

[0645] The server generates real-time feedback based on the evaluation results, and users can check the feedback displayed on their smart glasses and immediately implement improvements during the meeting.

[0646] 7. Generate detailed feedback reports

[0647] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0648] Specific examples

[0649] For example, when presenting emergency response measures at a security meeting, the following steps are taken:

[0650] 1. Setup and Start

[0651] The user puts on the smart glasses, sets up an "emergency response presentation" on the device, and starts the meeting.

[0652] 2. Collection of audio data

[0653] When a user gives a presentation, the microphone in the smart glasses collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0654] 3. Communication evaluation and sentiment analysis

[0655] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0656] 4. Evaluating visual support

[0657] When a user is using a slide, the camera in the smart glasses collects images of the slide and sends them to a server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[0658] 5. Real-time feedback

[0659] The feedback generated by the server is sent to the smart glasses in real time, allowing users to review the feedback and take immediate action during the meeting.

[0660] 6. Generate detailed feedback reports

[0661] Once the presentation is over, the server generates and provides a detailed feedback report to the user, including the adequacy of the explanation, clarity of communication, visual support, sentiment analysis results, and a follow-up plan for specific participants.

[0662] Prompt Sentence Examples

[0663] By using prompts like the following, you can get specific feedback from the generative AI model based on its analysis results:

[0664] Example prompt sentence:

[0665] "Please rate the following meeting:

[0666] 1. The explanation was easy to understand.

[0667] 2. The points were clearly communicated.

[0668] 3. Participants responded actively.

[0669] Please take these factors into consideration and provide a specific assessment and suggestions for improvement."

[0670] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0671] Step 1:

[0672] Audio data collection

[0673] The user puts on the smart glasses and starts the meeting. The microphone in the smart glasses connected to the device collects audio data in real time and transmits it to the server.

[0674] <input>: what the user says

[0675] <Output>: Audio data

[0676] <Specific operation>: The microphone built into the smart glasses captures audio signals and transmits them to the device via wireless communication.

[0677] Step 2:

[0678] Converting audio data to a string

[0679] The server converts the voice data collected in step 1 into a string of text in real time using a speech recognition system (e.g., Google Speech-to-Text API).

[0680] <Input>: Audio data

[0681] <Output>: Converted string data

[0682] <Specific operation>: The server calls the speech recognition API and converts the speech data into text data. The converted text is stored in a database.

[0683] Step 3:

[0684] Communication Evaluation

[0685] The server analyzes the string data converted in step 2 using natural language processing (NLP) technology (e.g., the Transformers library) and evaluates the communication.

[0686] <Input>: String data

[0687] <Output>: Communication evaluation results

[0688] <Specific operation>: The server uses the NLP model to check the grammar of the text data, extract keywords, and evaluate the clarity and appropriateness of the explanations, and generates the results as feedback.

[0689] Step 4:

[0690] sentiment analysis

[0691] The server performs sentiment analysis based on the voice data from step 1 and the text data from step 2. It analyzes the tone, pitch, and rhythm of the voice to evaluate the participants' emotions, level of understanding, and degree of agreement.

[0692] <Input>: Audio data and text data

[0693] <Output>:Sentiment analysis results

[0694] <Specific operation>: The server uses a voice analysis algorithm to analyze the emotional tone and pitch fluctuations of the voice and identify the emotional state based on that.

[0695] Step 5:

[0696] Evaluating visual support

[0697] When a user uses presentation materials, the smart glasses' camera and screen capture function are used to collect the materials and send them to the server, which then uses image recognition technology (e.g., OpenCV) to evaluate the readability and visual effect of the materials.

[0698] <Input>: Image data of presentation materials

[0699] <Output>: Visual support evaluation results

[0700] <Specific operation>: The camera in the smart glasses takes a photo of the document and sends the image to the server, which uses an image processing algorithm to evaluate the font size and color of the text.

[0701] Step 6:

[0702] Generate real-time feedback

[0703] The server generates real-time feedback based on the evaluation results of steps 3, 4, and 5 and notifies the smart glasses, allowing the user to check the feedback on the spot and take corrective measures.

[0704] <Input>: Communication evaluation results, emotion analysis results, visual support evaluation results

[0705] <Output>: Real-time feedback

[0706] <Specific operation>: The server consolidates the evaluation results, generates feedback messages containing important points and improvement measures, and displays them on the smart glasses display.

[0707] Step 7:

[0708] Generate detailed feedback reports

[0709] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0710] <input>: All evaluation data

[0711] <output>:Detailed feedback report

[0712] <Specific operation>: The server aggregates all evaluation results and generates a detailed feedback report based on the report template. The report is provided to the user in a format such as PDF.

[0713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0714] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, emotion analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Furthermore, this system is combined with an emotion engine that recognizes the user's emotions, allowing the emotion data to be reflected in the feedback in real time. Specific embodiments will be described below.

[0715] System Configuration

[0716] 1. User operations

[0717] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[0718] 2. Audio data collection and conversion

[0719] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[0720] 3. Communication Evaluation

[0721] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[0722] 4. Evaluating visual support

[0723] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[0724] 5. Sentiment analysis using an emotion engine

[0725] The server includes an emotion engine for analyzing emotions based on voice data. Specifically, the emotion engine analyzes the tone of voice, pitch, rhythm of speech, etc., to evaluate participants' emotions, level of understanding, and degree of agreement. The emotion engine also analyzes image data for more accurate emotion recognition.

[0726] 6. Real-time feedback

[0727] The server generates real-time feedback based on the evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback and immediately implement improvements during the meeting.

[0728] 7. Generate detailed feedback reports and follow-up plans

[0729] After the meeting, the server generates a detailed feedback report based on all the evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants, allowing users to obtain specific improvement suggestions for the next meeting.

[0730] Specific examples

[0731] For 1-on-1 meetings

[0732] 1. Setup and Start

[0733] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[0734] 2. Collection of audio data

[0735] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0736] 3. Communication evaluation and sentiment analysis

[0737] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[0738] 4. Evaluating visual support

[0739] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[0740] 5. Real-time feedback

[0741] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[0742] 6. Generate detailed feedback reports

[0743] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for any anxious employees.

[0744] In this way, the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[0745] The processing flow will be explained below.

[0746] Step 1:

[0747] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[0748] Step 2:

[0749] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[0750] Step 3:

[0751] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[0752] Step 4:

[0753] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[0754] Step 5:

[0755] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[0756] Step 6:

[0757] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[0758] Step 7:

[0759] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[0760] Step 8:

[0761] The server uses an emotion engine to analyze both audio and visual data to analyze participants' emotions, including tone of voice, pitch, rhythm of speech, and facial expressions, to assess participants' emotions, level of understanding, and degree of agreement, achieving more accurate emotion recognition.

[0762] Step 9:

[0763] The server generates real-time feedback based on these evaluation results and emotion data, and the generated feedback is sent to the device.

[0764] Step 10:

[0765] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[0766] Step 11:

[0767] After the meeting, the server generates a detailed feedback report based on all the evaluation and emotion data, and provides the report to the user.

[0768] Step 12:

[0769] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[0770] Example 2

[0771] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0772] Conventional meeting management systems have difficulty in providing real-time feedback or generating detailed feedback reports, limiting their ability to improve meeting efficiency and productivity. Furthermore, they have been unable to accurately grasp participants' emotions and levels of understanding and immediately implement improvement measures based on that information, making it difficult to ensure effective communication.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for collecting audio data and converting it into text in real time, a means for analyzing the converted text data and evaluating the information transmission, a means for capturing and collecting image materials and evaluating their effectiveness, a means for performing emotion analysis based on the audio data and evaluating the participants' levels of understanding and satisfaction, a means for providing opinions in real time based on these evaluation results, and a means for generating a detailed opinion report after the meeting. This enables real-time feedback during the meeting and optimization of communication taking into account the participants' emotions and levels of understanding. Furthermore, generating a detailed feedback report can suggest specific improvement measures for the next meeting.

[0774] "Sound data" refers to digital or analog signals collected as sound.

[0775] A "character string" refers to a series of characters or words that are output as a result of analyzing sound data.

[0776] "Communication evaluation" refers to the process of analyzing and evaluating the quality, content appropriateness, and clarity of communication.

[0777] "Image material" refers to digital or analog image data containing visual information, such as presentations or whiteboards.

[0778] "Emotion analysis" refers to the process of analyzing participants' emotions and psychological states from audio and image data.

[0779] "Real-time feedback" refers to evaluations and opinions that are instantly communicated to users based on the results of analysis of sound and image data.

[0780] An "opinion report" is a document that details the analysis of a meeting, including an evaluation of communication, visual support, and sentiment analysis results.

[0781] A "follow-up plan" is a plan that specifies the methods and steps for follow-up if necessary for specific participants based on the evaluation data.

[0782] "User's information processing device" refers to an electronic device, such as a personal computer or smartphone, used to receive feedback from the system.

[0783] The system of the present invention collects audio data, converts it into text, evaluates the communication of information, collects and evaluates visual materials, analyzes emotions, provides real-time feedback, generates detailed opinion reports, and presents follow-up plans to improve the efficiency and productivity of corporate meetings. Specific embodiments are described below.

[0784] System Configuration

[0785] This system mainly consists of three elements: a server, a terminal, and a user.

[0786] User operations

[0787] Users use their terminals to set the type of meeting (presentation, one-on-one, regular gathering, etc.) and register it with the server. After the meeting starts, users can have regular conversations or make presentations.

[0788] Acoustic data collection and conversion

[0789] During a meeting, the device collects audio data in real time through the microphone and sends it to the server. The server converts the received audio data into text using a speech recognition system, such as the Google Speech-to-Text API. This converted text data is then stored in a database on the server.

[0790] Evaluation of information transmission

[0791] The server analyzes the converted text data using natural language processing (NLP) technology, such as SpaCy or Google NLP API, to evaluate the quality of the communication, the appropriateness of the content, and clarity. The evaluation results are saved as feedback data, such as the "appropriateness of the explanation" and "appropriateness of the words."

[0792] Collection and evaluation of image materials

[0793] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server. The server then uses image recognition technology such as Google Cloud Vision API to evaluate the readability and visual effect of the visual support materials. For example, it generates feedback such as "The figures are complex and difficult to read."

[0794] Emotion analysis

[0795] The server analyzes the audio data using emotion engines such as IBM Watson Tone Analyzer. It evaluates participants' emotions, level of understanding, and degree of satisfaction based on the tone of voice, pitch, and rhythm of speech. It also analyzes image data to achieve more accurate emotion recognition. The analysis results are saved as feedback, such as "a particular employee may be feeling anxious."

[0796] Real-time feedback

[0797] The server generates real-time feedback based on all evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback during the meeting and immediately implement any necessary improvements.

[0798] Present a detailed opinion report and follow-up plan

[0799] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and a follow-up plan for specific participants. This detailed feedback report provides specific improvement suggestions for the next meeting.

[0800] Example: 1-on-1 meeting

[0801] 1. Setup and Start

[0802] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[0803] 2. Acoustic data collection

[0804] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0805] 3. Evaluation of information transmission and emotion analysis

[0806] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[0807] 4. Evaluation of image materials

[0808] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[0809] 5. Real-time feedback

[0810] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[0811] 6. Generate a detailed opinion report

[0812] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for anxious employees.

[0813] Prompt Sentence Examples

[0814] Below is an example of a prompt sentence to input to the generative AI model.

[0815] Prepare a detailed feedback report for your 1:1 meetings, including a communication evaluation, visual support evaluation, sentiment analysis results, and a plan to follow up with anxious employees based on the following data:

[0816] Audio Data Text: "I feel like I didn't explain this project well enough."

[0817] Visual support: "The diagram on the whiteboard was complicated."

[0818] Sentiment analysis results: "Certain employees may be feeling anxious."

[0819] In this way, the system of the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[0820] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0821] Step 1: Set up users and start a meeting

[0822] How it works: A user turns on their device, opens the Meetings application, selects the type of meeting (presentation, one-on-one, regular get-together, etc.), and configures it.

[0823] Input: Enter the meeting type and participant information.

[0824] Data processing: Format the entered meeting information into the appropriate format.

[0825] Output: Sends the configured meeting information to the server, which stores it in the database and completes the meeting preparation.

[0826] Step 2: Collecting sound data and sending it to the server

[0827] Action: A user presses the "Start Meeting" button to start a meeting. The device's microphone collects audio data.

[0828] Input: User's voice.

[0829] Data processing: Collect sound data in real time and convert it into an appropriate format.

[0830] Output: Sends sound data to the server.

[0831] Step 3: Converting sound data into a string

[0832] How it works: The server converts the received audio data into a string using a speech recognition system (e.g., Google Speech-to-Text API).

[0833] Input: Sound data.

[0834] Data processing: Converting sound data into text using a voice recognition system.

[0835] Output: The converted character data is saved in a database on the server.

[0836] Step 4: Analyze text data and evaluate information transmission

[0837] How it works: The server analyzes the converted text data using natural language processing (NLP) technology (e.g., SpaCy or Google NLP API) to evaluate the quality of the communication, the appropriateness of the content, and clarity.

[0838] Input: The converted character data.

[0839] Data processing: Analyzing text data using NLP techniques and calculating specific evaluation indicators (e.g., appropriateness, clarity, etc.).

[0840] Output: The evaluation results are saved as feedback data on the server.

[0841] Step 5: Collecting image data and sending it to the server

[0842] What it does: If a user is using a presentation, the device will capture the presentation using the camera or screen capture function.

[0843] Input: Visual materials (presentation slides, whiteboard images, etc.).

[0844] Data processing: Converting collected images into the appropriate format.

[0845] Output: Send the image data to the server.

[0846] Step 6: Analysis and evaluation of imagery

[0847] How it works: The server analyzes the image data using image recognition technology such as the Google Cloud Vision API to evaluate the readability and visual effectiveness of the visual support materials.

[0848] Input: Collected image data.

[0849] Data processing: Image data is analyzed using image recognition technology to calculate visual evaluation indicators.

[0850] Output: Evaluation results are generated and stored as feedback data on the server.

[0851] Step 7: Analyze the sentiment data

[0852] How it works: The server analyzes the audio data using an emotion engine (e.g., IBM Watson Tone Analyzer) and evaluates the participants' emotions, level of understanding, and degree of agreement based on the tone, pitch, and rhythm of their voices.

[0853] Input: Sound data.

[0854] Data processing: Analyzes sound data using an emotion engine to evaluate emotions and comprehension.

[0855] Output: The sentiment analysis results are saved as feedback data on the server.

[0856] Step 8: Generate and communicate real-time feedback

[0857] How it works: The server generates real-time feedback based on all evaluation results and emotion data, and sends the generated feedback to the device.

[0858] Input: Various evaluation results and sentiment analysis results.

[0859] Data processing: Various evaluation results and sentiment analysis results are combined to generate feedback.

[0860] Output: Feedback data is sent to the device, where the user can review it and take any necessary corrective action.

[0861] Step 9: Generate and submit a detailed opinion report

[0862] How it works: After the meeting, the server generates a detailed opinion report based on all the evaluation data.

[0863] Input: All rating data and sentiment analysis results.

[0864] Data processing: Compile various evaluation data to prepare a detailed opinion report.

[0865] Output: Provide users with a feedback report with concrete improvements for the next meeting.

[0866] The above is a concrete explanation of the processing steps of this system.

[0867] (Application example 2)

[0868] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0869] In conventional factory meetings, there are insufficient means to evaluate the quality of communication, the effectiveness of visual support materials, and the emotions and satisfaction of participants in real time, making it difficult to improve productivity and provide efficient feedback.In addition, factory robots cannot adaptively control their operations based on the content of meetings and feedback results, which limits the improvement of productivity throughout the factory.

[0870] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting voice data and converting it into text in real time; means for analyzing the converted text data and evaluating communication; means for photographing and collecting visual support materials and evaluating their effectiveness; means for performing emotion analysis based on the voice data and evaluating participants' levels of understanding and satisfaction; means for providing real-time feedback based on these evaluation results; means for generating a detailed feedback report after the meeting; means for improving productivity within the factory and the quality of human communication; and means for adaptively controlling the operation of factory robots using the analyzed data in real time. This makes it possible to improve the quality of communication and the effectiveness of visual support, accurately evaluate participants' emotions and satisfaction, and provide immediate feedback. Furthermore, factory robots can also adaptively control their operation based on the meeting results, improving the productivity of the entire factory.

[0871] "Audio Data" means voice information collected through a voice input device such as a microphone.

[0872] "Real-time" means having a time response where data is collected, processed, and fed back immediately.

[0873] "String" refers to data that has been converted from audio data into text format.

[0874] "Communication evaluation" refers to the act of analyzing the converted text data and evaluating the appropriateness of the communication, clarity, and validity of the content.

[0875] "Visual support materials" means materials that visually support information, such as presentation materials and whiteboards.

[0876] "Effectiveness of visual support" refers to evaluating how effective visual support materials are in conveying information.

[0877] "Sentiment analysis" refers to a technology that analyzes participants' emotions based on audio data and evaluates their level of understanding and agreement.

[0878] "Real-time feedback" means providing immediate feedback based on collected and analyzed data.

[0879] A "detailed feedback report" refers to a report generated after a meeting that includes communication evaluation results, sentiment analysis results, specific improvement suggestions, etc.

[0880] "Productivity improvement" means optimizing factory production processes to increase efficiency and results.

[0881] "Quality of communication" refers to indicators that reflect the clarity, appropriateness, and level of understanding of information exchange.

[0882] "Data analysis" refers to the act of analyzing collected data and extracting valuable information.

[0883] "Factory robot motion control" refers to the technology that manages the movements and actions of robots in factories to accomplish designated tasks.

[0884] System Configuration

[0885] The present invention provides a system for making meetings in a factory more efficient and improving productivity. The configuration and operation of the system for carrying out the invention will be described below.

[0886] 1. Hardware Configuration

[0887] This system uses the following hardware:

[0888] Microphone: A device for collecting audio data.

[0889] Camera: A device for capturing visual support materials.

[0890] Robot body: A device that moves around the factory and collects, analyzes, and provides feedback on various data.

[0891] Display: A device for displaying feedback and reports.

[0892] 2. Software Configuration

[0893] The system software utilizes the following technologies:

[0894] Speech recognition: Converts collected voice data into text in real time using the Google Cloud Speech-to-Text API.

[0895] Natural language processing: Using generative AI models such as GPT-4, the converted text data is analyzed and communication is evaluated.

[0896] Image Recognition: Analyze visual support materials using OpenCV and evaluate their effectiveness.

[0897] Sentiment analysis: IBM Watson Tone Analyzer is used to analyze voice tone and pitch to assess participants' emotions.

[0898] Real-time data processing and communication: Process data in real time and provide feedback using the MQTT protocol.

[0899] Processing flow

[0900] 1. Data Collection

[0901] When a user starts a meeting, the robot's microphone and camera collect audio and visual support material, which is then transmitted in real time to a server.

[0902] 2. Speech recognition and conversion

[0903] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text format.

[0904] 3. Natural Language Processing (NLP)

[0905] The converted text data is analyzed using GPT-4 to evaluate the quality of communication.

[0906] 4. Visual Support Assessment

[0907] Use image recognition with OpenCV to evaluate the content of presentation materials and whiteboards.

[0908] 5. Sentiment analysis

[0909] IBM Watson Tone Analyzer analyzes the tone and pitch of audio data to assess participants' emotions in real time.

[0910] 6. Real-time feedback generation

[0911] The server generates real-time feedback based on the analysis results and displays it on a display or mobile device.

[0912] 7. Generate detailed feedback reports

[0913] After the meeting, the server generates a detailed feedback report and provides it to the user.

[0914] Specific examples

[0915] A meeting to improve production line efficiency is held in a factory conference room. When a user starts a meeting, a robot collects audio and whiteboard content. The speech is then converted into text using the Google Cloud Speech-to-Text API, and GPT-4 analyzes the content of the communication. OpenCV also evaluates the diagrams and text on the whiteboard, and IBM Watson Tone Analyzer analyzes the emotions of the participants. Based on this data, real-time feedback is displayed on the screen, and any necessary improvements are immediately implemented. After the meeting, a detailed feedback report is generated, providing suggestions for improvement for the next meeting.

[0916] Prompt Sentence Examples

[0917] Analyze the following text data, evaluate the quality of communication, visual support, and analyze sentiment to generate real-time feedback, and generate a detailed feedback report after the meeting.

[0918] Text data:

[0919] Participant A: How can we increase production speed?

[0920] Participant B: It's impossible unless we hire more staff.

[0921] Participant C: No, if we optimize the process, we may be able to manage without increasing manpower.

[0922] Visual Data:

[0923] (Whiteboard image)

[0924] Output formats:

[0925] 1. Communication Evaluation

[0926] 2. Visual Support Evaluation

[0927] 3. Sentiment analysis results

[0928] 4. Real-time feedback

[0929] 5. Detailed feedback report

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] When a user starts a meeting, the device's microphone and camera begin to collect audio and visual support data. The audio and video data are collected in real time and sent to the server. The input is the audio and video data, and the output is the collected raw data.

[0933] Step 2:

[0934] The server uses the Google Cloud Speech-to-Text API to convert the transmitted voice data into text in real time. This conversion process converts the phonological information of the voice into text data. The input is voice data, and the output is text data converted from the voice into text format.

[0935] Step 3:

[0936] The server uses a generative AI model such as GPT-4 to analyze the converted text data. Specifically, it uses natural language processing (NLP) technology to evaluate the content and quality of the communication, determining factors such as clarity of explanation, appropriateness to the question, and use of specific examples. The input is text data, and the output is the evaluation result of the communication.

[0937] Step 4:

[0938] The server uses OpenCV to analyze the visual support materials (image data) sent from the device. This includes the readability of the images and the placement of figures and text. The analysis results are used to evaluate the effectiveness of the visual support. The input is the image data, and the output is the evaluation result of the visual support.

[0939] Step 5:

[0940] The server uses IBM Watson Tone Analyzer to analyze the tone and pitch of the audio data and perform sentiment analysis. This analysis evaluates participants' emotions, understanding, and degree of agreement. The input is the audio data, and the output is the sentiment analysis results.

[0941] Step 6:

[0942] The server generates real-time feedback based on these evaluation results and displays it on the terminal display or mobile device. This feedback includes immediately actionable improvement measures. The input is communication evaluation, visual support evaluation, and sentiment analysis data, and the output is real-time feedback.

[0943] Step 7:

[0944] After the meeting, the server generates a detailed feedback report based on all the evaluation data. This report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and specific improvement suggestions for the next meeting. The input is all the evaluation data, and the output is a detailed feedback report.

[0945] Step 8:

[0946] The server adaptively controls the factory robot's operation based on the data analyzed in real time. This includes adjusting the factory robot's operation patterns and work procedures based on the meeting content and feedback results. The input is the meeting analysis data and feedback results, and the output is the factory robot's operation control commands.

[0947] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0948] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0949] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0950] [Third embodiment]

[0951] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0952] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0953] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0954] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0955] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0956] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0957] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0958] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0959] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0960] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0961] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0962] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0963] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[0964] System Configuration

[0965] 1. User operations

[0966] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[0967] 2. Audio data collection and conversion

[0968] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[0969] 3. Communication Evaluation

[0970] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[0971] 4. Evaluating visual support

[0972] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[0973] 5. Sentiment analysis

[0974] The server performs sentiment analysis based on the audio data, specifically analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, level of understanding, and degree of agreement.

[0975] 6. Real-time feedback

[0976] The server generates real-time feedback based on the evaluation results and sends it to the device, where users can check the feedback and immediately implement improvements during the meeting.

[0977] 7. Generate detailed feedback reports

[0978] After the meeting, the server generates a detailed feedback report based on all the evaluation data and provides it to the user, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[0979] Specific examples

[0980] For presentations

[0981] 1. Setup and Start

[0982] The user sets up a "presentation" on the device and starts the meeting, the server loads the template, and the setup is complete.

[0983] 2. Collection of audio data

[0984] When a user gives a presentation, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[0985] 3. Communication evaluation and sentiment analysis

[0986] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[0987] 4. Evaluating visual support

[0988] When a user is using a slide, the device collects images of the slide and sends them to the server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[0989] 5. Real-time feedback

[0990] The server generates feedback and sends it to the device in real time, allowing users to review the feedback and implement improvements during the meeting.

[0991] 6. Generate detailed feedback reports

[0992] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user, including the adequacy of the explanation, clarity of communication, visual support, and sentiment analysis results.

[0993] In this way, the present invention can greatly improve the efficiency and productivity of meetings.

[0994] The processing flow will be explained below.

[0995] Step 1:

[0996] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[0997] Step 2:

[0998] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[0999] Step 3:

[1000] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[1001] Step 4:

[1002] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[1003] Step 5:

[1004] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[1005] Step 6:

[1006] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[1007] Step 7:

[1008] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[1009] Step 8:

[1010] The server performs sentiment analysis based on the audio data, analyzing the tone of voice, pitch, rhythm of speech, etc. to assess participants' emotions, level of understanding, and degree of agreement.

[1011] Step 9:

[1012] The server generates real-time feedback based on these evaluation results, and the generated feedback is sent to the device.

[1013] Step 10:

[1014] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[1015] Step 11:

[1016] After the meeting, the server generates a detailed feedback report based on all the evaluation data, and provides the report to the user.

[1017] Step 12:

[1018] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[1019] Example 1

[1020] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1021] Many modern corporate meetings require increased efficiency and productivity. However, in traditional meetings, it is difficult to evaluate the quality of communication between participants, their level of understanding, and the appropriateness of visual support in real time and provide immediate feedback. Furthermore, creating detailed evaluation reports after meetings and creating follow-up plans for specific participants can be time-consuming and laborious, leading to delays in implementation. A system that can effectively resolve these issues and improve the quality of meetings is needed.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1023] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, means for setting and recording the type of meeting, and means for using natural language processing technology and image recognition technology to evaluate the voice and visual data. This makes it possible to provide effective feedback in real time during meetings and quickly create a detailed evaluation report after the meeting ends, thereby improving the overall quality and productivity of meetings.

[1024] "Means for collecting audio data" refers to a function in which the device uses a microphone to record audio during a meeting in real time and acquires this audio as data.

[1025] The "means for converting into text in real time" is a function that uses voice recognition technology to instantly convert collected voice data into text format.

[1026] The "means for analyzing the converted character string data" is a function that uses natural language processing technology to analyze text data and evaluate the quality and effectiveness of communication.

[1027] "Means for capturing and collecting visual support materials" refers to the ability to record and collect presentation materials and visual support using a camera or screen capture function.

[1028] The "means for evaluating the effectiveness" is a function that uses image recognition technology to analyze collected visual support materials and evaluate their visual effectiveness and readability.

[1029] "Means for performing emotion analysis based on audio data" is a function that analyzes the tone, pitch, rhythm, etc. of the voice from collected audio data to evaluate the emotions and level of understanding of participants.

[1030] "Means for assessing participants' level of understanding and satisfaction" is a function that evaluates how well each participant understood and agreed with the meeting content based on the results of sentiment analysis.

[1031] "Means for providing feedback in real time" refers to a function that instantly generates feedback based on the analyzed evaluation results and notifies the user's device.

[1032] The "means for generating a detailed feedback report after the meeting" is a function for generating a report including detailed evaluation results after the meeting based on all evaluation data during the meeting.

[1033] The "means for setting and recording the type of meeting" is a function that allows the user to select and set the type of meeting and record that information in the database.

[1034] "Natural language processing technology" is a technology for analyzing text data and understanding meaning and context, and for evaluating communication based on the analysis results.

[1035] "Image recognition technology" is a technology for analyzing image data and evaluating the characteristics and effects of visual information.

[1036] The system of the present invention is designed to improve the efficiency and productivity of corporate meetings by providing voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan suggestions.

[1037] 1. User operations

[1038] The user first selects the type of meeting on their device and registers the information with the server. For example, if the user selects "presentation," the server loads the corresponding template and settings. Once the settings are complete, the user can start the meeting as usual.

[1039] 2. Audio data collection and conversion

[1040] As the meeting progresses, the device uses its microphone to collect real-time audio data and transmits it to a server, which then converts the collected audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, which stores the converted text data for analysis.

[1041] 3. Communication Evaluation

[1042] The server analyzes the stored text data using natural language processing (NLP) technology, such as SpaCy or Google NLP. This analysis evaluates the appropriateness of the explanation, the appropriateness of the language, and the clarity of the communication. For example, it can generate feedback such as "There are not enough specific examples."

[1043] 4. Evaluating visual support

[1044] When a user uses presentation materials, the device captures the materials using the camera or screen capture function. The captured image data is sent to a server, which then evaluates it using image recognition technology such as Google Cloud Vision API. For example, the server generates feedback such as, "The font on the slides is small and difficult to read."

[1045] 5. Sentiment analysis

[1046] The server analyzes the tone, pitch, rhythm, and other aspects of the voice data and performs emotional analysis, which evaluates participants' emotions and intelligibility and can generate alerts such as "a specific participant may have a question."

[1047] 6. Real-time feedback

[1048] The server generates real-time feedback based on the analysis results and sends it to the device. Users can check this feedback during the meeting and take immediate action. For example, feedback such as "Please make the font larger on the slides" may be displayed on the device.

[1049] 7. Generate detailed feedback reports

[1050] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants. Users can receive the report in PDF format or other formats.

[1051] Specific examples

[1052] 1. Setup and Start

[1053] The user sets up a "presentation" on the device and starts a meeting, and the server loads the relevant template and completes the setup.

[1054] 2. Collection of audio data

[1055] When a user gives a presentation, the device collects the audio in real time and sends it to the server, which converts the audio data into text and stores it.

[1056] 3. Communication evaluation and sentiment analysis

[1057] The server analyzes the stored text data using NLP technology and generates feedback such as "You should use more concrete examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[1058] 4. Evaluating visual support

[1059] When a user is using a slide deck, the device collects images of the slide deck and sends them to the server, which evaluates the font size, color usage, etc. and generates feedback such as "The font on the slide deck is small and difficult to read."

[1060] 5. Real-time feedback

[1061] The feedback generated by the server is sent to the terminal in real time, and the user can check the content and implement improvements during the meeting.

[1062] 6. Generate detailed feedback reports

[1063] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user.

[1064] This system can significantly improve the efficiency and productivity of meetings.

[1065] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1066] Step 1:

[1067] The user sets the meeting type on the device. Specifically, the user launches the device's application, selects the appropriate meeting type from "Presentation," "1-on-1," or "Regular Meeting," and sends the information to the server. At this time, the device receives the meeting type and other related setting information (e.g., scheduled time, participant list, etc.) as input data and sends it to the server. Based on the received data, the server loads the templates and tools required for the meeting.

[1068] Step 2:

[1069] During meetings, the device uses a microphone to collect voice data in real time. Specifically, the device's microphone records what is being said and sends the voice data sequentially to the server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API or similar. In this process, speech recognition technology is applied to the input voice data to generate text data. The converted text data is stored on the server for later analysis.

[1070] Step 3:

[1071] The server analyzes the stored text data using natural language processing (NLP) technology. Specifically, it receives the text data as input and evaluates the appropriateness of the explanation, the appropriateness of the words, the clarity of the communication, and so on. For example, based on identified keywords and phrases, it generates feedback such as "There are not enough specific examples" or "There is too much technical terminology." This feedback information is output as the analysis result.

[1072] Step 4:

[1073] When a user uses presentation materials, the device collects the materials using the device's camera or screen capture function. Specifically, the device's camera takes a picture of the slides or printed materials, or the screen capture function captures the digital materials. The image data of these visual support materials is sent as input to a server. The server then uses image recognition technology such as Google Cloud Vision API to analyze the image data and evaluate the visual support. For example, specific feedback such as "The font on the slides is small and difficult to read" is output.

[1074] Step 5:

[1075] The server performs sentiment analysis on the collected voice data. Specifically, it receives the voice data as input and analyzes the tone, pitch, rhythm, etc. of the voice. Using sentiment analysis technology such as IBM Watson's Tone Analyzer, it evaluates the participants' emotions and level of understanding, and outputs a judgment such as "a particular participant may have a question."

[1076] Step 6:

[1077] The server generates real-time feedback based on the analysis results and sends it to the device. Specifically, it integrates the results of the communication evaluation, the visual support evaluation, and the sentiment analysis, and immediately notifies the user of areas for improvement. The device displays this to the user in real time, and the user can review the feedback and implement improvements during the meeting. Examples of feedback include "Please make the font larger on the slides" and "Please use more specific examples."

[1078] Step 7:

[1079] After the meeting, the server generates a detailed feedback report based on all the evaluation data. Specifically, it integrates the results of the voice-to-text conversion, communication evaluation, visual support evaluation, and sentiment analysis to create the report. The report is generated in a format such as PDF and sent to the user via email or cloud storage service. The report also includes a follow-up plan for specific participants.

[1080] (Application example 1)

[1081] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1082] Conventional meeting systems make it difficult to provide real-time feedback on participants' communication skills, the appropriateness of visual support, their emotions, and their level of understanding. Real-time feedback also lacks the ability to provide immediate improvement measures, preventing participants from making improvements on the spot. Furthermore, the lack of a function to generate detailed feedback reports and follow-up plans for specific participants prevents meetings from being efficiently and productively implemented.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1084] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, and means for displaying the feedback and evaluation results on the smart glasses in real time, thereby enabling improved communication during meetings, appropriate use of visual support, understanding of participants' emotions and levels of understanding, providing immediate feedback, and generating a detailed feedback report and follow-up plan.

[1085] "Audio data" is information in which audio signals are recorded in digital format, and is basic data for using collected audio for analysis and processing.

[1086] A "character string" is data that is generated by analyzing voice data using voice recognition technology and expressing it in a text format that can be understood as language.

[1087] "Communication evaluation" is the process of analyzing the text data converted from audio data to evaluate the content of participants' statements, the validity of their explanations, the appropriateness of their words, and whether or not they communicated clearly.

[1088] "Visual support materials" are visual content such as slides, charts, and images used to support a presentation or explanation.

[1089] "Sentiment analysis" is the process of analyzing voice tone, pitch, and speaking rhythm based on audio data to assess participants' emotions, understanding, and agreement.

[1090] "Real-time feedback" refers to feedback information provided to participants immediately during a meeting based on the evaluation results, allowing participants to implement improvement measures on the spot.

[1091] A "feedback report" is a detailed report generated after a meeting that includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and a follow-up plan.

[1092] "Smart glasses" are glasses-type devices that have the ability to overlay digital information onto virtual and real space, allowing users to check the information in real time.

[1093] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[1094] System configuration

[1095] The system uses smart glasses to collect and convert voice data, evaluate communication, analyze sentiment, provide real-time feedback, generate detailed feedback reports, evaluate visual support, and display feedback.

[1096] 1. User operations

[1097] The user uses a device connected to the smart glasses to set the type of meeting (presentation, one-on-one, regular meeting, etc.) and register it on the server. Once the meeting starts, the user only needs to conduct regular conversations and presentations.

[1098] 2. Audio data collection and conversion

[1099] The smart glasses' microphone collects voice data in real time and sends it to a cloud server, which then converts the data into text using a voice recognition system.

[1100] 3. Communication Evaluation

[1101] The server analyzes the text data and uses natural language processing (NLP) techniques to evaluate the communication, including whether the explanation is clear, the words used are appropriate, and the delivery style is easy for participants to understand.

[1102] 4. Sentiment analysis

[1103] The server performs sentiment analysis on the audio data, analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, understanding, and satisfaction, and generates an alert if a specific participant may have a question.

[1104] 5. Evaluating visual support

[1105] When a user uses presentation materials, the smart glasses use their built-in camera and screen capture function to collect the materials and send them to a server, which then uses image recognition technology to evaluate the materials' readability and visual effect.

[1106] 6. Real-time feedback

[1107] The server generates real-time feedback based on the evaluation results, and users can check the feedback displayed on their smart glasses and immediately implement improvements during the meeting.

[1108] 7. Generate detailed feedback reports

[1109] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[1110] Specific examples

[1111] For example, when presenting emergency response measures at a security meeting, the following steps are taken:

[1112] 1. Setup and Start

[1113] The user puts on the smart glasses, sets up an "emergency response presentation" on the device, and starts the meeting.

[1114] 2. Collection of audio data

[1115] When a user gives a presentation, the microphone in the smart glasses collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1116] 3. Communication evaluation and sentiment analysis

[1117] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[1118] 4. Evaluating visual support

[1119] When a user is using a slide, the camera in the smart glasses collects images of the slide and sends them to a server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[1120] 5. Real-time feedback

[1121] The feedback generated by the server is sent to the smart glasses in real time, allowing users to review the feedback and take immediate action during the meeting.

[1122] 6. Generate detailed feedback reports

[1123] Once the presentation is over, the server generates and provides a detailed feedback report to the user, including the adequacy of the explanation, clarity of communication, visual support, sentiment analysis results, and a follow-up plan for specific participants.

[1124] Prompt Sentence Examples

[1125] By using prompts like the following, you can get specific feedback from the generative AI model based on its analysis results:

[1126] Example prompt sentence:

[1127] "Please rate the following meeting:

[1128] 1. The explanation was easy to understand.

[1129] 2. The points were clearly communicated.

[1130] 3. Participants responded actively.

[1131] Please take these factors into consideration and provide a specific assessment and suggestions for improvement."

[1132] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1133] Step 1:

[1134] Audio data collection

[1135] The user puts on the smart glasses and starts the meeting. The microphone in the smart glasses connected to the device collects audio data in real time and transmits it to the server.

[1136] <input>: what the user says

[1137] <Output>: Audio data

[1138] <Specific operation>: The microphone built into the smart glasses captures audio signals and transmits them to the device via wireless communication.

[1139] Step 2:

[1140] Converting audio data to a string

[1141] The server converts the voice data collected in step 1 into a string of text in real time using a speech recognition system (e.g., Google Speech-to-Text API).

[1142] <Input>: Audio data

[1143] <Output>: Converted string data

[1144] <Specific operation>: The server calls the speech recognition API and converts the speech data into text data. The converted text is stored in a database.

[1145] Step 3:

[1146] Communication Evaluation

[1147] The server analyzes the string data converted in step 2 using natural language processing (NLP) technology (e.g., the Transformers library) and evaluates the communication.

[1148] <Input>: String data

[1149] <Output>: Communication evaluation results

[1150] <Specific operation>: The server uses the NLP model to check the grammar of the text data, extract keywords, and evaluate the clarity and appropriateness of the explanations, and generates the results as feedback.

[1151] Step 4:

[1152] sentiment analysis

[1153] The server performs sentiment analysis based on the voice data from step 1 and the text data from step 2. It analyzes the tone, pitch, and rhythm of the voice to evaluate the participants' emotions, level of understanding, and degree of agreement.

[1154] <Input>: Audio data and text data

[1155] <Output>:Sentiment analysis results

[1156] <Specific operation>: The server uses a voice analysis algorithm to analyze the emotional tone and pitch fluctuations of the voice and identify the emotional state based on that.

[1157] Step 5:

[1158] Evaluating visual support

[1159] When a user uses presentation materials, the smart glasses' camera and screen capture function are used to collect the materials and send them to the server, which then uses image recognition technology (e.g., OpenCV) to evaluate the readability and visual effect of the materials.

[1160] <Input>: Image data of presentation materials

[1161] <Output>: Visual support evaluation results

[1162] <Specific operation>: The camera in the smart glasses takes a photo of the document and sends the image to the server, which uses an image processing algorithm to evaluate the font size and color of the text.

[1163] Step 6:

[1164] Generate real-time feedback

[1165] The server generates real-time feedback based on the evaluation results of steps 3, 4, and 5 and notifies the smart glasses, allowing the user to check the feedback on the spot and take corrective measures.

[1166] <Input>: Communication evaluation results, emotion analysis results, visual support evaluation results

[1167] <Output>: Real-time feedback

[1168] <Specific operation>: The server consolidates the evaluation results, generates feedback messages containing important points and improvement measures, and displays them on the smart glasses display.

[1169] Step 7:

[1170] Generate detailed feedback reports

[1171] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[1172] <input>: All evaluation data

[1173] <output>:Detailed feedback report

[1174] <Specific operation>: The server aggregates all evaluation results and generates a detailed feedback report based on the report template. The report is provided to the user in a format such as PDF.

[1175] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1176] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, emotion analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Furthermore, this system is combined with an emotion engine that recognizes the user's emotions, allowing the emotion data to be reflected in the feedback in real time. Specific embodiments will be described below.

[1177] System Configuration

[1178] 1. User operations

[1179] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[1180] 2. Audio data collection and conversion

[1181] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[1182] 3. Communication Evaluation

[1183] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[1184] 4. Evaluating visual support

[1185] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[1186] 5. Sentiment analysis using an emotion engine

[1187] The server includes an emotion engine for analyzing emotions based on voice data. Specifically, the emotion engine analyzes the tone of voice, pitch, rhythm of speech, etc., to evaluate participants' emotions, level of understanding, and degree of agreement. The emotion engine also analyzes image data for more accurate emotion recognition.

[1188] 6. Real-time feedback

[1189] The server generates real-time feedback based on the evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback and immediately implement improvements during the meeting.

[1190] 7. Generate detailed feedback reports and follow-up plans

[1191] After the meeting, the server generates a detailed feedback report based on all the evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants, allowing users to obtain specific improvement suggestions for the next meeting.

[1192] Specific examples

[1193] For 1-on-1 meetings

[1194] 1. Setup and Start

[1195] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[1196] 2. Collection of audio data

[1197] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1198] 3. Communication evaluation and sentiment analysis

[1199] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[1200] 4. Evaluating visual support

[1201] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[1202] 5. Real-time feedback

[1203] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[1204] 6. Generate detailed feedback reports

[1205] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for any anxious employees.

[1206] In this way, the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[1207] The processing flow will be explained below.

[1208] Step 1:

[1209] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[1210] Step 2:

[1211] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[1212] Step 3:

[1213] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[1214] Step 4:

[1215] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[1216] Step 5:

[1217] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[1218] Step 6:

[1219] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[1220] Step 7:

[1221] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[1222] Step 8:

[1223] The server uses an emotion engine to analyze both audio and visual data to analyze participants' emotions, including tone of voice, pitch, rhythm of speech, and facial expressions, to assess participants' emotions, level of understanding, and degree of agreement, achieving more accurate emotion recognition.

[1224] Step 9:

[1225] The server generates real-time feedback based on these evaluation results and emotion data, and the generated feedback is sent to the device.

[1226] Step 10:

[1227] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[1228] Step 11:

[1229] After the meeting, the server generates a detailed feedback report based on all the evaluation and emotion data, and provides the report to the user.

[1230] Step 12:

[1231] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[1232] Example 2

[1233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] Conventional meeting management systems have difficulty in providing real-time feedback or generating detailed feedback reports, limiting their ability to improve meeting efficiency and productivity. Furthermore, they have been unable to accurately grasp participants' emotions and levels of understanding and immediately implement improvement measures based on that information, making it difficult to ensure effective communication.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for collecting audio data and converting it into text in real time, a means for analyzing the converted text data and evaluating the information transmission, a means for capturing and collecting image materials and evaluating their effectiveness, a means for performing emotion analysis based on the audio data and evaluating the participants' levels of understanding and satisfaction, a means for providing opinions in real time based on these evaluation results, and a means for generating a detailed opinion report after the meeting. This enables real-time feedback during the meeting and optimization of communication taking into account the participants' emotions and levels of understanding. Furthermore, generating a detailed feedback report can suggest specific improvement measures for the next meeting.

[1236] "Sound data" refers to digital or analog signals collected as sound.

[1237] A "character string" refers to a series of characters or words that are output as a result of analyzing sound data.

[1238] "Communication evaluation" refers to the process of analyzing and evaluating the quality, content appropriateness, and clarity of communication.

[1239] "Image material" refers to digital or analog image data containing visual information, such as presentations or whiteboards.

[1240] "Emotion analysis" refers to the process of analyzing participants' emotions and psychological states from audio and image data.

[1241] "Real-time feedback" refers to evaluations and opinions that are instantly communicated to users based on the results of analysis of sound and image data.

[1242] An "opinion report" is a document that details the analysis of a meeting, including an evaluation of communication, visual support, and sentiment analysis results.

[1243] A "follow-up plan" is a plan that specifies the methods and steps for follow-up if necessary for specific participants based on the evaluation data.

[1244] "User's information processing device" refers to an electronic device, such as a personal computer or smartphone, used to receive feedback from the system.

[1245] The system of the present invention collects audio data, converts it into text, evaluates the communication of information, collects and evaluates visual materials, analyzes emotions, provides real-time feedback, generates detailed opinion reports, and presents follow-up plans to improve the efficiency and productivity of corporate meetings. Specific embodiments are described below.

[1246] System Configuration

[1247] This system mainly consists of three elements: a server, a terminal, and a user.

[1248] User operations

[1249] Users use their terminals to set the type of meeting (presentation, one-on-one, regular gathering, etc.) and register it with the server. After the meeting starts, users can have regular conversations or make presentations.

[1250] Acoustic data collection and conversion

[1251] During a meeting, the device collects audio data in real time through the microphone and sends it to the server. The server converts the received audio data into text using a speech recognition system, such as the Google Speech-to-Text API. This converted text data is then stored in a database on the server.

[1252] Evaluation of information transmission

[1253] The server analyzes the converted text data using natural language processing (NLP) technology, such as SpaCy or Google NLP API, to evaluate the quality of the communication, the appropriateness of the content, and clarity. The evaluation results are saved as feedback data, such as the "appropriateness of the explanation" and "appropriateness of the words."

[1254] Collection and evaluation of image materials

[1255] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server. The server then uses image recognition technology such as Google Cloud Vision API to evaluate the readability and visual effect of the visual support materials. For example, it generates feedback such as "The figures are complex and difficult to read."

[1256] Emotion analysis

[1257] The server analyzes the audio data using emotion engines such as IBM Watson Tone Analyzer. It evaluates participants' emotions, level of understanding, and degree of satisfaction based on the tone of voice, pitch, and rhythm of speech. It also analyzes image data to achieve more accurate emotion recognition. The analysis results are saved as feedback, such as "a particular employee may be feeling anxious."

[1258] Real-time feedback

[1259] The server generates real-time feedback based on all evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback during the meeting and immediately implement any necessary improvements.

[1260] Present a detailed opinion report and follow-up plan

[1261] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and a follow-up plan for specific participants. This detailed feedback report provides specific improvement suggestions for the next meeting.

[1262] Example: 1-on-1 meeting

[1263] 1. Setup and Start

[1264] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[1265] 2. Acoustic data collection

[1266] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1267] 3. Evaluation of information transmission and emotion analysis

[1268] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[1269] 4. Evaluation of image materials

[1270] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[1271] 5. Real-time feedback

[1272] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[1273] 6. Generate a detailed opinion report

[1274] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for anxious employees.

[1275] Prompt Sentence Examples

[1276] Below is an example of a prompt sentence to input to the generative AI model.

[1277] Prepare a detailed feedback report for your 1:1 meetings, including a communication evaluation, visual support evaluation, sentiment analysis results, and a plan to follow up with anxious employees based on the following data:

[1278] Audio Data Text: "I feel like I didn't explain this project well enough."

[1279] Visual support: "The diagram on the whiteboard was complicated."

[1280] Sentiment analysis results: "Certain employees may be feeling anxious."

[1281] In this way, the system of the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[1282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1283] Step 1: Set up users and start a meeting

[1284] How it works: A user turns on their device, opens the Meetings application, selects the type of meeting (presentation, one-on-one, regular get-together, etc.), and configures it.

[1285] Input: Enter the meeting type and participant information.

[1286] Data processing: Format the entered meeting information into the appropriate format.

[1287] Output: Sends the configured meeting information to the server, which stores it in the database and completes the meeting preparation.

[1288] Step 2: Collecting sound data and sending it to the server

[1289] Action: A user presses the "Start Meeting" button to start a meeting. The device's microphone collects audio data.

[1290] Input: User's voice.

[1291] Data processing: Collect sound data in real time and convert it into an appropriate format.

[1292] Output: Sends sound data to the server.

[1293] Step 3: Converting sound data into a string

[1294] How it works: The server converts the received audio data into a string using a speech recognition system (e.g., Google Speech-to-Text API).

[1295] Input: Sound data.

[1296] Data processing: Converting sound data into text using a voice recognition system.

[1297] Output: The converted character data is saved in a database on the server.

[1298] Step 4: Analyze text data and evaluate information transmission

[1299] How it works: The server analyzes the converted text data using natural language processing (NLP) technology (e.g., SpaCy or Google NLP API) to evaluate the quality of the communication, the appropriateness of the content, and clarity.

[1300] Input: The converted character data.

[1301] Data processing: Analyzing text data using NLP techniques and calculating specific evaluation indicators (e.g., appropriateness, clarity, etc.).

[1302] Output: The evaluation results are saved as feedback data on the server.

[1303] Step 5: Collecting image data and sending it to the server

[1304] What it does: If a user is using a presentation, the device will capture the presentation using the camera or screen capture function.

[1305] Input: Visual materials (presentation slides, whiteboard images, etc.).

[1306] Data processing: Converting collected images into the appropriate format.

[1307] Output: Send the image data to the server.

[1308] Step 6: Analysis and evaluation of imagery

[1309] How it works: The server analyzes the image data using image recognition technology such as the Google Cloud Vision API to evaluate the readability and visual effectiveness of the visual support materials.

[1310] Input: Collected image data.

[1311] Data processing: Image data is analyzed using image recognition technology to calculate visual evaluation indicators.

[1312] Output: Evaluation results are generated and stored as feedback data on the server.

[1313] Step 7: Analyze the sentiment data

[1314] How it works: The server analyzes the audio data using an emotion engine (e.g., IBM Watson Tone Analyzer) and evaluates the participants' emotions, level of understanding, and degree of agreement based on the tone, pitch, and rhythm of their voices.

[1315] Input: Sound data.

[1316] Data processing: Analyzes sound data using an emotion engine to evaluate emotions and comprehension.

[1317] Output: The sentiment analysis results are saved as feedback data on the server.

[1318] Step 8: Generate and communicate real-time feedback

[1319] How it works: The server generates real-time feedback based on all evaluation results and emotion data, and sends the generated feedback to the device.

[1320] Input: Various evaluation results and sentiment analysis results.

[1321] Data processing: Various evaluation results and sentiment analysis results are combined to generate feedback.

[1322] Output: Feedback data is sent to the device, where the user can review it and take any necessary corrective action.

[1323] Step 9: Generate and submit a detailed opinion report

[1324] How it works: After the meeting, the server generates a detailed opinion report based on all the evaluation data.

[1325] Input: All rating data and sentiment analysis results.

[1326] Data processing: Compile various evaluation data to prepare a detailed opinion report.

[1327] Output: Provide users with a feedback report with concrete improvements for the next meeting.

[1328] The above is a concrete explanation of the processing steps of this system.

[1329] (Application example 2)

[1330] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1331] In conventional factory meetings, there are insufficient means to evaluate the quality of communication, the effectiveness of visual support materials, and the emotions and satisfaction of participants in real time, making it difficult to improve productivity and provide efficient feedback.In addition, factory robots cannot adaptively control their operations based on the content of meetings and feedback results, which limits the improvement of productivity throughout the factory.

[1332] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting voice data and converting it into text in real time; means for analyzing the converted text data and evaluating communication; means for photographing and collecting visual support materials and evaluating their effectiveness; means for performing emotion analysis based on the voice data and evaluating participants' levels of understanding and satisfaction; means for providing real-time feedback based on these evaluation results; means for generating a detailed feedback report after the meeting; means for improving productivity within the factory and the quality of human communication; and means for adaptively controlling the operation of factory robots using the analyzed data in real time. This makes it possible to improve the quality of communication and the effectiveness of visual support, accurately evaluate participants' emotions and satisfaction, and provide immediate feedback. Furthermore, factory robots can also adaptively control their operation based on the meeting results, improving the productivity of the entire factory.

[1333] "Audio Data" means voice information collected through a voice input device such as a microphone.

[1334] "Real-time" means having a time response where data is collected, processed, and fed back immediately.

[1335] "String" refers to data that has been converted from audio data into text format.

[1336] "Communication evaluation" refers to the act of analyzing the converted text data and evaluating the appropriateness of the communication, clarity, and validity of the content.

[1337] "Visual support materials" means materials that visually support information, such as presentation materials and whiteboards.

[1338] "Effectiveness of visual support" refers to evaluating how effective visual support materials are in conveying information.

[1339] "Sentiment analysis" refers to a technology that analyzes participants' emotions based on audio data and evaluates their level of understanding and agreement.

[1340] "Real-time feedback" means providing immediate feedback based on collected and analyzed data.

[1341] A "detailed feedback report" refers to a report generated after a meeting that includes communication evaluation results, sentiment analysis results, specific improvement suggestions, etc.

[1342] "Productivity improvement" means optimizing factory production processes to increase efficiency and results.

[1343] "Quality of communication" refers to indicators that reflect the clarity, appropriateness, and level of understanding of information exchange.

[1344] "Data analysis" refers to the act of analyzing collected data and extracting valuable information.

[1345] "Factory robot motion control" refers to the technology that manages the movements and actions of robots in factories to accomplish designated tasks.

[1346] System Configuration

[1347] The present invention provides a system for making meetings in a factory more efficient and improving productivity. The configuration and operation of the system for carrying out the invention will be described below.

[1348] 1. Hardware Configuration

[1349] This system uses the following hardware:

[1350] Microphone: A device for collecting audio data.

[1351] Camera: A device for capturing visual support materials.

[1352] Robot body: A device that moves around the factory and collects, analyzes, and provides feedback on various data.

[1353] Display: A device for displaying feedback and reports.

[1354] 2. Software Configuration

[1355] The system software utilizes the following technologies:

[1356] Speech recognition: Converts collected voice data into text in real time using the Google Cloud Speech-to-Text API.

[1357] Natural language processing: Using generative AI models such as GPT-4, the converted text data is analyzed and communication is evaluated.

[1358] Image Recognition: Analyze visual support materials using OpenCV and evaluate their effectiveness.

[1359] Sentiment analysis: IBM Watson Tone Analyzer is used to analyze voice tone and pitch to assess participants' emotions.

[1360] Real-time data processing and communication: Process data in real time and provide feedback using the MQTT protocol.

[1361] Processing flow

[1362] 1. Data Collection

[1363] When a user starts a meeting, the robot's microphone and camera collect audio and visual support material, which is then transmitted in real time to a server.

[1364] 2. Speech recognition and conversion

[1365] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text format.

[1366] 3. Natural Language Processing (NLP)

[1367] The converted text data is analyzed using GPT-4 to evaluate the quality of communication.

[1368] 4. Visual Support Assessment

[1369] Use image recognition with OpenCV to evaluate the content of presentation materials and whiteboards.

[1370] 5. Sentiment analysis

[1371] IBM Watson Tone Analyzer analyzes the tone and pitch of audio data to assess participants' emotions in real time.

[1372] 6. Real-time feedback generation

[1373] The server generates real-time feedback based on the analysis results and displays it on a display or mobile device.

[1374] 7. Generate detailed feedback reports

[1375] After the meeting, the server generates a detailed feedback report and provides it to the user.

[1376] Specific examples

[1377] A meeting to improve production line efficiency is held in a factory conference room. When a user starts a meeting, a robot collects audio and whiteboard content. The speech is then converted into text using the Google Cloud Speech-to-Text API, and GPT-4 analyzes the content of the communication. OpenCV also evaluates the diagrams and text on the whiteboard, and IBM Watson Tone Analyzer analyzes the emotions of the participants. Based on this data, real-time feedback is displayed on the screen, and any necessary improvements are immediately implemented. After the meeting, a detailed feedback report is generated, providing suggestions for improvement for the next meeting.

[1378] Prompt Sentence Examples

[1379] Analyze the following text data, evaluate the quality of communication, visual support, and analyze sentiment to generate real-time feedback, and generate a detailed feedback report after the meeting.

[1380] Text data:

[1381] Participant A: How can we increase production speed?

[1382] Participant B: It's impossible unless we hire more staff.

[1383] Participant C: No, if we optimize the process, we may be able to manage without increasing manpower.

[1384] Visual Data:

[1385] (Whiteboard image)

[1386] Output formats:

[1387] 1. Communication Evaluation

[1388] 2. Visual Support Evaluation

[1389] 3. Sentiment analysis results

[1390] 4. Real-time feedback

[1391] 5. Detailed feedback report

[1392] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1393] Step 1:

[1394] When a user starts a meeting, the device's microphone and camera begin to collect audio and visual support data. The audio and video data are collected in real time and sent to the server. The input is the audio and video data, and the output is the collected raw data.

[1395] Step 2:

[1396] The server uses the Google Cloud Speech-to-Text API to convert the transmitted voice data into text in real time. This conversion process converts the phonological information of the voice into text data. The input is voice data, and the output is text data converted from the voice into text format.

[1397] Step 3:

[1398] The server uses a generative AI model such as GPT-4 to analyze the converted text data. Specifically, it uses natural language processing (NLP) technology to evaluate the content and quality of the communication, determining factors such as clarity of explanation, appropriateness to the question, and use of specific examples. The input is text data, and the output is the evaluation result of the communication.

[1399] Step 4:

[1400] The server uses OpenCV to analyze the visual support materials (image data) sent from the device. This includes the readability of the images and the placement of figures and text. The analysis results are used to evaluate the effectiveness of the visual support. The input is the image data, and the output is the evaluation result of the visual support.

[1401] Step 5:

[1402] The server uses IBM Watson Tone Analyzer to analyze the tone and pitch of the audio data and perform sentiment analysis. This analysis evaluates participants' emotions, understanding, and degree of agreement. The input is the audio data, and the output is the sentiment analysis results.

[1403] Step 6:

[1404] The server generates real-time feedback based on these evaluation results and displays it on the terminal display or mobile device. This feedback includes immediately actionable improvement measures. The input is communication evaluation, visual support evaluation, and sentiment analysis data, and the output is real-time feedback.

[1405] Step 7:

[1406] After the meeting, the server generates a detailed feedback report based on all the evaluation data. This report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and specific improvement suggestions for the next meeting. The input is all the evaluation data, and the output is a detailed feedback report.

[1407] Step 8:

[1408] The server adaptively controls the factory robot's operation based on the data analyzed in real time. This includes adjusting the factory robot's operation patterns and work procedures based on the meeting content and feedback results. The input is the meeting analysis data and feedback results, and the output is the factory robot's operation control commands.

[1409] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1410] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1411] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1412] [Fourth embodiment]

[1413] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1414] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1415] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1416] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1417] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1419] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1420] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1421] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1422] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1423] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1424] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1425] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1426] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[1427] System Configuration

[1428] 1. User operations

[1429] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[1430] 2. Audio data collection and conversion

[1431] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[1432] 3. Communication Evaluation

[1433] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[1434] 4. Evaluating visual support

[1435] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[1436] 5. Sentiment analysis

[1437] The server performs sentiment analysis based on the audio data, specifically analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, level of understanding, and degree of agreement.

[1438] 6. Real-time feedback

[1439] The server generates real-time feedback based on the evaluation results and sends it to the device, where users can check the feedback and immediately implement improvements during the meeting.

[1440] 7. Generate detailed feedback reports

[1441] After the meeting, the server generates a detailed feedback report based on all the evaluation data and provides it to the user, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[1442] Specific examples

[1443] For presentations

[1444] 1. Setup and Start

[1445] The user sets up a "presentation" on the device and starts the meeting, the server loads the template, and the setup is complete.

[1446] 2. Collection of audio data

[1447] When a user gives a presentation, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1448] 3. Communication evaluation and sentiment analysis

[1449] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[1450] 4. Evaluating visual support

[1451] When a user is using a slide, the device collects images of the slide and sends them to the server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[1452] 5. Real-time feedback

[1453] The server generates feedback and sends it to the device in real time, allowing users to review the feedback and implement improvements during the meeting.

[1454] 6. Generate detailed feedback reports

[1455] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user, including the adequacy of the explanation, clarity of communication, visual support, and sentiment analysis results.

[1456] In this way, the present invention can greatly improve the efficiency and productivity of meetings.

[1457] The processing flow will be explained below.

[1458] Step 1:

[1459] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[1460] Step 2:

[1461] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[1462] Step 3:

[1463] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[1464] Step 4:

[1465] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[1466] Step 5:

[1467] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[1468] Step 6:

[1469] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[1470] Step 7:

[1471] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[1472] Step 8:

[1473] The server performs sentiment analysis based on the audio data, analyzing the tone of voice, pitch, rhythm of speech, etc. to assess participants' emotions, level of understanding, and degree of agreement.

[1474] Step 9:

[1475] The server generates real-time feedback based on these evaluation results, and the generated feedback is sent to the device.

[1476] Step 10:

[1477] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[1478] Step 11:

[1479] After the meeting, the server generates a detailed feedback report based on all the evaluation data, and provides the report to the user.

[1480] Step 12:

[1481] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[1482] Example 1

[1483] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1484] Many modern corporate meetings require increased efficiency and productivity. However, in traditional meetings, it is difficult to evaluate the quality of communication between participants, their level of understanding, and the appropriateness of visual support in real time and provide immediate feedback. Furthermore, creating detailed evaluation reports after meetings and creating follow-up plans for specific participants can be time-consuming and laborious, leading to delays in implementation. A system that can effectively resolve these issues and improve the quality of meetings is needed.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1486] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, means for setting and recording the type of meeting, and means for using natural language processing technology and image recognition technology to evaluate the voice and visual data. This makes it possible to provide effective feedback in real time during meetings and quickly create a detailed evaluation report after the meeting ends, thereby improving the overall quality and productivity of meetings.

[1487] "Means for collecting audio data" refers to a function in which the device uses a microphone to record audio during a meeting in real time and acquires this audio as data.

[1488] The "means for converting into text in real time" is a function that uses voice recognition technology to instantly convert collected voice data into text format.

[1489] The "means for analyzing the converted character string data" is a function that uses natural language processing technology to analyze text data and evaluate the quality and effectiveness of communication.

[1490] "Means for capturing and collecting visual support materials" refers to the ability to record and collect presentation materials and visual support using a camera or screen capture function.

[1491] The "means for evaluating the effectiveness" is a function that uses image recognition technology to analyze collected visual support materials and evaluate their visual effectiveness and readability.

[1492] "Means for performing emotion analysis based on audio data" is a function that analyzes the tone, pitch, rhythm, etc. of the voice from collected audio data to evaluate the emotions and level of understanding of participants.

[1493] "Means for assessing participants' level of understanding and satisfaction" is a function that evaluates how well each participant understood and agreed with the meeting content based on the results of sentiment analysis.

[1494] "Means for providing feedback in real time" refers to a function that instantly generates feedback based on the analyzed evaluation results and notifies the user's device.

[1495] The "means for generating a detailed feedback report after the meeting" is a function for generating a report including detailed evaluation results after the meeting based on all evaluation data during the meeting.

[1496] The "means for setting and recording the type of meeting" is a function that allows the user to select and set the type of meeting and record that information in the database.

[1497] "Natural language processing technology" is a technology for analyzing text data and understanding meaning and context, and for evaluating communication based on the analysis results.

[1498] "Image recognition technology" is a technology for analyzing image data and evaluating the characteristics and effects of visual information.

[1499] The system of the present invention is designed to improve the efficiency and productivity of corporate meetings by providing voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan suggestions.

[1500] 1. User operations

[1501] The user first selects the type of meeting on their device and registers the information with the server. For example, if the user selects "presentation," the server loads the corresponding template and settings. Once the settings are complete, the user can start the meeting as usual.

[1502] 2. Audio data collection and conversion

[1503] As the meeting progresses, the device uses its microphone to collect real-time audio data and transmits it to a server, which then converts the collected audio data into text using speech recognition technology such as the Google Cloud Speech-to-Text API, which stores the converted text data for analysis.

[1504] 3. Communication Evaluation

[1505] The server analyzes the stored text data using natural language processing (NLP) technology, such as SpaCy or Google NLP. This analysis evaluates the appropriateness of the explanation, the appropriateness of the language, and the clarity of the communication. For example, it can generate feedback such as "There are not enough specific examples."

[1506] 4. Evaluating visual support

[1507] When a user uses presentation materials, the device captures the materials using the camera or screen capture function. The captured image data is sent to a server, which then evaluates it using image recognition technology such as Google Cloud Vision API. For example, the server generates feedback such as, "The font on the slides is small and difficult to read."

[1508] 5. Sentiment analysis

[1509] The server analyzes the tone, pitch, rhythm, and other aspects of the voice data and performs emotional analysis, which evaluates participants' emotions and intelligibility and can generate alerts such as "a specific participant may have a question."

[1510] 6. Real-time feedback

[1511] The server generates real-time feedback based on the analysis results and sends it to the device. Users can check this feedback during the meeting and take immediate action. For example, feedback such as "Please make the font larger on the slides" may be displayed on the device.

[1512] 7. Generate detailed feedback reports

[1513] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants. Users can receive the report in PDF format or other formats.

[1514] Specific examples

[1515] 1. Setup and Start

[1516] The user sets up a "presentation" on the device and starts a meeting, and the server loads the relevant template and completes the setup.

[1517] 2. Collection of audio data

[1518] When a user gives a presentation, the device collects the audio in real time and sends it to the server, which converts the audio data into text and stores it.

[1519] 3. Communication evaluation and sentiment analysis

[1520] The server analyzes the stored text data using NLP technology and generates feedback such as "You should use more concrete examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[1521] 4. Evaluating visual support

[1522] When a user is using a slide deck, the device collects images of the slide deck and sends them to the server, which evaluates the font size, color usage, etc. and generates feedback such as "The font on the slide deck is small and difficult to read."

[1523] 5. Real-time feedback

[1524] The feedback generated by the server is sent to the terminal in real time, and the user can check the content and implement improvements during the meeting.

[1525] 6. Generate detailed feedback reports

[1526] Once the presentation is complete, the server generates a detailed feedback report and provides it to the user.

[1527] This system can significantly improve the efficiency and productivity of meetings.

[1528] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1529] Step 1:

[1530] The user sets the meeting type on the device. Specifically, the user launches the device's application, selects the appropriate meeting type from "Presentation," "1-on-1," or "Regular Meeting," and sends the information to the server. At this time, the device receives the meeting type and other related setting information (e.g., scheduled time, participant list, etc.) as input data and sends it to the server. Based on the received data, the server loads the templates and tools required for the meeting.

[1531] Step 2:

[1532] During meetings, the device uses a microphone to collect voice data in real time. Specifically, the device's microphone records what is being said and sends the voice data sequentially to the server. The server then converts the received voice data into text data using the Google Cloud Speech-to-Text API or similar. In this process, speech recognition technology is applied to the input voice data to generate text data. The converted text data is stored on the server for later analysis.

[1533] Step 3:

[1534] The server analyzes the stored text data using natural language processing (NLP) technology. Specifically, it receives the text data as input and evaluates the appropriateness of the explanation, the appropriateness of the words, the clarity of the communication, and so on. For example, based on identified keywords and phrases, it generates feedback such as "There are not enough specific examples" or "There is too much technical terminology." This feedback information is output as the analysis result.

[1535] Step 4:

[1536] When a user uses presentation materials, the device collects the materials using the device's camera or screen capture function. Specifically, the device's camera takes a picture of the slides or printed materials, or the screen capture function captures the digital materials. The image data of these visual support materials is sent as input to a server. The server then uses image recognition technology such as Google Cloud Vision API to analyze the image data and evaluate the visual support. For example, specific feedback such as "The font on the slides is small and difficult to read" is output.

[1537] Step 5:

[1538] The server performs sentiment analysis on the collected voice data. Specifically, it receives the voice data as input and analyzes the tone, pitch, rhythm, etc. of the voice. Using sentiment analysis technology such as IBM Watson's Tone Analyzer, it evaluates the participants' emotions and level of understanding, and outputs a judgment such as "a particular participant may have a question."

[1539] Step 6:

[1540] The server generates real-time feedback based on the analysis results and sends it to the device. Specifically, it integrates the results of the communication evaluation, the visual support evaluation, and the sentiment analysis, and immediately notifies the user of areas for improvement. The device displays this to the user in real time, and the user can review the feedback and implement improvements during the meeting. Examples of feedback include "Please make the font larger on the slides" and "Please use more specific examples."

[1541] Step 7:

[1542] After the meeting, the server generates a detailed feedback report based on all the evaluation data. Specifically, it integrates the results of the voice-to-text conversion, communication evaluation, visual support evaluation, and sentiment analysis to create the report. The report is generated in a format such as PDF and sent to the user via email or cloud storage service. The report also includes a follow-up plan for specific participants.

[1543] (Application example 1)

[1544] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1545] Conventional meeting systems make it difficult to provide real-time feedback on participants' communication skills, the appropriateness of visual support, their emotions, and their level of understanding. Real-time feedback also lacks the ability to provide immediate improvement measures, preventing participants from making improvements on the spot. Furthermore, the lack of a function to generate detailed feedback reports and follow-up plans for specific participants prevents meetings from being efficiently and productively implemented.

[1546] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1547] In this invention, the server includes means for collecting voice data and converting it into text in real time, means for analyzing the converted text data and evaluating communication, means for capturing and collecting visual support materials and evaluating their effectiveness, means for performing sentiment analysis based on the voice data and evaluating participants' levels of understanding and satisfaction, means for providing real-time feedback based on these evaluation results, means for generating a detailed feedback report after the meeting ends, and means for displaying the feedback and evaluation results on the smart glasses in real time, thereby enabling improved communication during meetings, appropriate use of visual support, understanding of participants' emotions and levels of understanding, providing immediate feedback, and generating a detailed feedback report and follow-up plan.

[1548] "Audio data" is information in which audio signals are recorded in digital format, and is basic data for using collected audio for analysis and processing.

[1549] A "character string" is data that is generated by analyzing voice data using voice recognition technology and expressing it in a text format that can be understood as language.

[1550] "Communication evaluation" is the process of analyzing the text data converted from audio data to evaluate the content of participants' statements, the validity of their explanations, the appropriateness of their words, and whether or not they communicated clearly.

[1551] "Visual support materials" are visual content such as slides, charts, and images used to support a presentation or explanation.

[1552] "Sentiment analysis" is the process of analyzing voice tone, pitch, and speaking rhythm based on audio data to assess participants' emotions, understanding, and agreement.

[1553] "Real-time feedback" refers to feedback information provided to participants immediately during a meeting based on the evaluation results, allowing participants to implement improvement measures on the spot.

[1554] A "feedback report" is a detailed report generated after a meeting that includes communication evaluation results, the effectiveness of visual support, sentiment analysis results, and a follow-up plan.

[1555] "Smart glasses" are glasses-type devices that have the ability to overlay digital information onto virtual and real space, allowing users to check the information in real time.

[1556] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, sentiment analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Specific embodiments will be described below.

[1557] System configuration

[1558] The system uses smart glasses to collect and convert voice data, evaluate communication, analyze sentiment, provide real-time feedback, generate detailed feedback reports, evaluate visual support, and display feedback.

[1559] 1. User operations

[1560] The user uses a device connected to the smart glasses to set the type of meeting (presentation, one-on-one, regular meeting, etc.) and register it on the server. Once the meeting starts, the user only needs to conduct regular conversations and presentations.

[1561] 2. Audio data collection and conversion

[1562] The smart glasses' microphone collects voice data in real time and sends it to a cloud server, which then converts the data into text using a voice recognition system.

[1563] 3. Communication Evaluation

[1564] The server analyzes the text data and uses natural language processing (NLP) techniques to evaluate the communication, including whether the explanation is clear, the words used are appropriate, and the delivery style is easy for participants to understand.

[1565] 4. Sentiment analysis

[1566] The server performs sentiment analysis on the audio data, analyzing the tone of voice, pitch, and rhythm of speech to assess participants' emotions, understanding, and satisfaction, and generates an alert if a specific participant may have a question.

[1567] 5. Evaluating visual support

[1568] When a user uses presentation materials, the smart glasses use their built-in camera and screen capture function to collect the materials and send them to a server, which then uses image recognition technology to evaluate the materials' readability and visual effect.

[1569] 6. Real-time feedback

[1570] The server generates real-time feedback based on the evaluation results, and users can check the feedback displayed on their smart glasses and immediately implement improvements during the meeting.

[1571] 7. Generate detailed feedback reports

[1572] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[1573] Specific examples

[1574] For example, when presenting emergency response measures at a security meeting, the following steps are taken:

[1575] 1. Setup and Start

[1576] The user puts on the smart glasses, sets up an "emergency response presentation" on the device, and starts the meeting.

[1577] 2. Collection of audio data

[1578] When a user gives a presentation, the microphone in the smart glasses collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1579] 3. Communication evaluation and sentiment analysis

[1580] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "You should use more specific examples." It also performs sentiment analysis and generates alerts that "certain participants may have questions."

[1581] 4. Evaluating visual support

[1582] When a user is using a slide, the camera in the smart glasses collects images of the slide and sends them to a server, which evaluates the visual effects, such as font size and color usage, and generates feedback such as "The font on the slide is small and difficult to read."

[1583] 5. Real-time feedback

[1584] The feedback generated by the server is sent to the smart glasses in real time, allowing users to review the feedback and take immediate action during the meeting.

[1585] 6. Generate detailed feedback reports

[1586] Once the presentation is over, the server generates and provides a detailed feedback report to the user, including the adequacy of the explanation, clarity of communication, visual support, sentiment analysis results, and a follow-up plan for specific participants.

[1587] Prompt Sentence Examples

[1588] By using prompts like the following, you can get specific feedback from the generative AI model based on its analysis results:

[1589] Example prompt sentence:

[1590] "Please rate the following meeting:

[1591] 1. The explanation was easy to understand.

[1592] 2. The points were clearly communicated.

[1593] 3. Participants responded actively.

[1594] Please take these factors into consideration and provide a specific assessment and suggestions for improvement."

[1595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1596] Step 1:

[1597] Audio data collection

[1598] The user puts on the smart glasses and starts the meeting. The microphone in the smart glasses connected to the device collects audio data in real time and transmits it to the server.

[1599] <input>: what the user says

[1600] <Output>: Audio data

[1601] <Specific operation>: The microphone built into the smart glasses captures audio signals and transmits them to the device via wireless communication.

[1602] Step 2:

[1603] Converting audio data to a string

[1604] The server converts the voice data collected in step 1 into a string of text in real time using a speech recognition system (e.g., Google Speech-to-Text API).

[1605] <Input>: Audio data

[1606] <Output>: Converted string data

[1607] <Specific operation>: The server calls the speech recognition API and converts the speech data into text data. The converted text is stored in a database.

[1608] Step 3:

[1609] Communication Evaluation

[1610] The server analyzes the string data converted in step 2 using natural language processing (NLP) technology (e.g., the Transformers library) and evaluates the communication.

[1611] <Input>: String data

[1612] <Output>: Communication evaluation results

[1613] <Specific operation>: The server uses the NLP model to check the grammar of the text data, extract keywords, and evaluate the clarity and appropriateness of the explanations, and generates the results as feedback.

[1614] Step 4:

[1615] sentiment analysis

[1616] The server performs sentiment analysis based on the voice data from step 1 and the text data from step 2. It analyzes the tone, pitch, and rhythm of the voice to evaluate the participants' emotions, level of understanding, and degree of agreement.

[1617] <Input>: Audio data and text data

[1618] <Output>:Sentiment analysis results

[1619] <Specific operation>: The server uses a voice analysis algorithm to analyze the emotional tone and pitch fluctuations of the voice and identify the emotional state based on that.

[1620] Step 5:

[1621] Evaluating visual support

[1622] When a user uses presentation materials, the smart glasses' camera and screen capture function are used to collect the materials and send them to the server, which then uses image recognition technology (e.g., OpenCV) to evaluate the readability and visual effect of the materials.

[1623] <Input>: Image data of presentation materials

[1624] <Output>: Visual support evaluation results

[1625] <Specific operation>: The camera in the smart glasses takes a photo of the document and sends the image to the server, which uses an image processing algorithm to evaluate the font size and color of the text.

[1626] Step 6:

[1627] Generate real-time feedback

[1628] The server generates real-time feedback based on the evaluation results of steps 3, 4, and 5 and notifies the smart glasses, allowing the user to check the feedback on the spot and take corrective measures.

[1629] <Input>: Communication evaluation results, emotion analysis results, visual support evaluation results

[1630] <Output>: Real-time feedback

[1631] <Specific operation>: The server consolidates the evaluation results, generates feedback messages containing important points and improvement measures, and displays them on the smart glasses display.

[1632] Step 7:

[1633] Generate detailed feedback reports

[1634] After the meeting, the server generates a detailed feedback report based on all evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants.

[1635] <input>: All evaluation data

[1636] <output>:Detailed feedback report

[1637] <Specific operation>: The server aggregates all evaluation results and generates a detailed feedback report based on the report template. The report is provided to the user in a format such as PDF.

[1638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1639] The system of the present invention performs voice conversion, communication evaluation, visual support evaluation, emotion analysis, real-time feedback, detailed feedback report generation, and follow-up plan presentation to improve the efficiency and productivity of corporate meetings. Furthermore, this system is combined with an emotion engine that recognizes the user's emotions, allowing the emotion data to be reflected in the feedback in real time. Specific embodiments will be described below.

[1640] System Configuration

[1641] 1. User operations

[1642] The user uses a terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and registers it on the server. After the meeting starts, the user can have a regular conversation or make a presentation.

[1643] 2. Audio data collection and conversion

[1644] During a meeting, the device collects voice data in real time through the microphone and sends it to the server, which then converts the received voice data into text using a voice recognition system.

[1645] 3. Communication Evaluation

[1646] The server analyzes the converted text data using natural language processing (NLP) technology and evaluates the communication, for example, evaluating the appropriateness of the explanation, the appropriateness of the words, and whether the communication was clear.

[1647] 4. Evaluating visual support

[1648] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server, which then uses image recognition technology to evaluate the readability and visual effect of the visual support materials.

[1649] 5. Sentiment analysis using an emotion engine

[1650] The server includes an emotion engine for analyzing emotions based on voice data. Specifically, the emotion engine analyzes the tone of voice, pitch, rhythm of speech, etc., to evaluate participants' emotions, level of understanding, and degree of agreement. The emotion engine also analyzes image data for more accurate emotion recognition.

[1651] 6. Real-time feedback

[1652] The server generates real-time feedback based on the evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback and immediately implement improvements during the meeting.

[1653] 7. Generate detailed feedback reports and follow-up plans

[1654] After the meeting, the server generates a detailed feedback report based on all the evaluation data, including communication evaluation results, the effectiveness of visual support, sentiment analysis results, and follow-up plans for specific participants, allowing users to obtain specific improvement suggestions for the next meeting.

[1655] Specific examples

[1656] For 1-on-1 meetings

[1657] 1. Setup and Start

[1658] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[1659] 2. Collection of audio data

[1660] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1661] 3. Communication evaluation and sentiment analysis

[1662] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[1663] 4. Evaluating visual support

[1664] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[1665] 5. Real-time feedback

[1666] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[1667] 6. Generate detailed feedback reports

[1668] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for any anxious employees.

[1669] In this way, the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[1670] The processing flow will be explained below.

[1671] Step 1:

[1672] The user uses the terminal to set the type of meeting (presentation, 1-on-1, regular meeting, etc.) and sends the setting to the server.

[1673] Step 2:

[1674] The server loads the meeting template based on the received setting information and notifies the terminal that the loading is complete.

[1675] Step 3:

[1676] When a meeting starts, the device collects audio data in real time through the microphone, and the collected audio data is sent to the server.

[1677] Step 4:

[1678] The server converts the received voice data into text data using a voice recognition system, and stores the converted text data for analysis.

[1679] Step 5:

[1680] The server analyzes the text data using natural language processing (NLP) technology, and based on the analysis results, evaluates the appropriateness of the explanation, the appropriateness of the words, and the clarity of the communication.

[1681] Step 6:

[1682] When a user is using presentation materials, the device uses the camera or screen capture function to capture and collect images of the materials, and the collected data is sent to the server.

[1683] Step 7:

[1684] The server uses image recognition technology to evaluate visual support materials, specifically evaluating font size, color usage, layout, and other factors to determine the material's readability and visual impact.

[1685] Step 8:

[1686] The server uses an emotion engine to analyze both audio and visual data to analyze participants' emotions, including tone of voice, pitch, rhythm of speech, and facial expressions, to assess participants' emotions, level of understanding, and degree of agreement, achieving more accurate emotion recognition.

[1687] Step 9:

[1688] The server generates real-time feedback based on these evaluation results and emotion data, and the generated feedback is sent to the device.

[1689] Step 10:

[1690] Users can view real-time feedback through their devices and implement improvements during the meeting if necessary.

[1691] Step 11:

[1692] After the meeting, the server generates a detailed feedback report based on all the evaluation and emotion data, and provides the report to the user.

[1693] Step 12:

[1694] The server generates a follow-up plan for the specific participant based on the evaluation results and notifies the user via the terminal.

[1695] Example 2

[1696] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1697] Conventional meeting management systems have difficulty in providing real-time feedback or generating detailed feedback reports, limiting their ability to improve meeting efficiency and productivity. Furthermore, they have been unable to accurately grasp participants' emotions and levels of understanding and immediately implement improvement measures based on that information, making it difficult to ensure effective communication.

[1698] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for collecting audio data and converting it into text in real time, a means for analyzing the converted text data and evaluating the information transmission, a means for capturing and collecting image materials and evaluating their effectiveness, a means for performing emotion analysis based on the audio data and evaluating the participants' levels of understanding and satisfaction, a means for providing opinions in real time based on these evaluation results, and a means for generating a detailed opinion report after the meeting. This enables real-time feedback during the meeting and optimization of communication taking into account the participants' emotions and levels of understanding. Furthermore, generating a detailed feedback report can suggest specific improvement measures for the next meeting.

[1699] "Sound data" refers to digital or analog signals collected as sound.

[1700] A "character string" refers to a series of characters or words that are output as a result of analyzing sound data.

[1701] "Communication evaluation" refers to the process of analyzing and evaluating the quality, content appropriateness, and clarity of communication.

[1702] "Image material" refers to digital or analog image data containing visual information, such as presentations or whiteboards.

[1703] "Emotion analysis" refers to the process of analyzing participants' emotions and psychological states from audio and image data.

[1704] "Real-time feedback" refers to evaluations and opinions that are instantly communicated to users based on the results of analysis of sound and image data.

[1705] An "opinion report" is a document that details the analysis of a meeting, including an evaluation of communication, visual support, and sentiment analysis results.

[1706] A "follow-up plan" is a plan that specifies the methods and steps for follow-up if necessary for specific participants based on the evaluation data.

[1707] "User's information processing device" refers to an electronic device, such as a personal computer or smartphone, used to receive feedback from the system.

[1708] The system of the present invention collects audio data, converts it into text, evaluates the communication of information, collects and evaluates visual materials, analyzes emotions, provides real-time feedback, generates detailed opinion reports, and presents follow-up plans to improve the efficiency and productivity of corporate meetings. Specific embodiments are described below.

[1709] System Configuration

[1710] This system mainly consists of three elements: a server, a terminal, and a user.

[1711] User operations

[1712] Users use their terminals to set the type of meeting (presentation, one-on-one, regular gathering, etc.) and register it with the server. After the meeting starts, users can have regular conversations or make presentations.

[1713] Acoustic data collection and conversion

[1714] During a meeting, the device collects audio data in real time through the microphone and sends it to the server. The server converts the received audio data into text using a speech recognition system, such as the Google Speech-to-Text API. This converted text data is then stored in a database on the server.

[1715] Evaluation of information transmission

[1716] The server analyzes the converted text data using natural language processing (NLP) technology, such as SpaCy or Google NLP API, to evaluate the quality of the communication, the appropriateness of the content, and clarity. The evaluation results are saved as feedback data, such as the "appropriateness of the explanation" and "appropriateness of the words."

[1717] Collection and evaluation of image materials

[1718] When a user uses presentation materials, the device captures the materials using a camera or screen capture function and sends them to a server. The server then uses image recognition technology such as Google Cloud Vision API to evaluate the readability and visual effect of the visual support materials. For example, it generates feedback such as "The figures are complex and difficult to read."

[1719] Emotion analysis

[1720] The server analyzes the audio data using emotion engines such as IBM Watson Tone Analyzer. It evaluates participants' emotions, level of understanding, and degree of satisfaction based on the tone of voice, pitch, and rhythm of speech. It also analyzes image data to achieve more accurate emotion recognition. The analysis results are saved as feedback, such as "a particular employee may be feeling anxious."

[1721] Real-time feedback

[1722] The server generates real-time feedback based on all evaluation results and emotion data, and the generated feedback is sent to the device, where users can check the feedback during the meeting and immediately implement any necessary improvements.

[1723] Present a detailed opinion report and follow-up plan

[1724] After the meeting, the server generates a detailed feedback report based on all the evaluation data. The report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and a follow-up plan for specific participants. This detailed feedback report provides specific improvement suggestions for the next meeting.

[1725] Example: 1-on-1 meeting

[1726] 1. Setup and Start

[1727] The user sets up a "1-on-1 meeting" on their device and registers it on the server. The server loads the template and the setup is complete.

[1728] 2. Acoustic data collection

[1729] When a user holds a one-on-one meeting, the device collects audio and transmits it in real time to a server, which converts the audio data into text and stores it for analysis.

[1730] 3. Evaluation of information transmission and emotion analysis

[1731] The server uses NLP technology to analyze the text data stored for analysis and generates feedback such as "There are not enough concrete examples in the explanations." In addition, it uses an emotion engine to perform sentiment analysis and generates an alert that "a particular employee may be feeling anxious."

[1732] 4. Evaluation of image materials

[1733] When a user uses a whiteboard, the device collects images of the whiteboard and sends them to the server, which uses image recognition technology to evaluate the content and generate feedback such as "The diagram is too complex to understand."

[1734] 5. Real-time feedback

[1735] The server generates feedback and sends it to the device in real time, allowing the user to review the feedback and take necessary corrective measures during the meeting.

[1736] 6. Generate a detailed opinion report

[1737] After the 1-on-1 meeting is over, the server generates a detailed feedback report and provides it to the user, including the appropriateness of the explanation, clear communication, visual support, sentiment analysis results, and a follow-up plan for anxious employees.

[1738] Prompt Sentence Examples

[1739] Below is an example of a prompt sentence to input to the generative AI model.

[1740] Prepare a detailed feedback report for your 1:1 meetings, including a communication evaluation, visual support evaluation, sentiment analysis results, and a plan to follow up with anxious employees based on the following data:

[1741] Audio Data Text: "I feel like I didn't explain this project well enough."

[1742] Visual support: "The diagram on the whiteboard was complicated."

[1743] Sentiment analysis results: "Certain employees may be feeling anxious."

[1744] In this way, the system of the present invention can significantly improve the efficiency and productivity of meetings and provide more effective feedback that takes into account the user's emotions.

[1745] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1746] Step 1: Set up users and start a meeting

[1747] How it works: A user turns on their device, opens the Meetings application, selects the type of meeting (presentation, one-on-one, regular get-together, etc.), and configures it.

[1748] Input: Enter the meeting type and participant information.

[1749] Data processing: Format the entered meeting information into the appropriate format.

[1750] Output: Sends the configured meeting information to the server, which stores it in the database and completes the meeting preparation.

[1751] Step 2: Collecting sound data and sending it to the server

[1752] Action: A user presses the "Start Meeting" button to start a meeting. The device's microphone collects audio data.

[1753] Input: User's voice.

[1754] Data processing: Collect sound data in real time and convert it into an appropriate format.

[1755] Output: Sends sound data to the server.

[1756] Step 3: Converting sound data into a string

[1757] How it works: The server converts the received audio data into a string using a speech recognition system (e.g., Google Speech-to-Text API).

[1758] Input: Sound data.

[1759] Data processing: Converting sound data into text using a voice recognition system.

[1760] Output: The converted character data is saved in a database on the server.

[1761] Step 4: Analyze text data and evaluate information transmission

[1762] How it works: The server analyzes the converted text data using natural language processing (NLP) technology (e.g., SpaCy or Google NLP API) to evaluate the quality of the communication, the appropriateness of the content, and clarity.

[1763] Input: The converted character data.

[1764] Data processing: Analyzing text data using NLP techniques and calculating specific evaluation indicators (e.g., appropriateness, clarity, etc.).

[1765] Output: The evaluation results are saved as feedback data on the server.

[1766] Step 5: Collecting image data and sending it to the server

[1767] What it does: If a user is using a presentation, the device will capture the presentation using the camera or screen capture function.

[1768] Input: Visual materials (presentation slides, whiteboard images, etc.).

[1769] Data processing: Converting collected images into the appropriate format.

[1770] Output: Send the image data to the server.

[1771] Step 6: Analysis and evaluation of imagery

[1772] How it works: The server analyzes the image data using image recognition technology such as the Google Cloud Vision API to evaluate the readability and visual effectiveness of the visual support materials.

[1773] Input: Collected image data.

[1774] Data processing: Image data is analyzed using image recognition technology to calculate visual evaluation indicators.

[1775] Output: Evaluation results are generated and stored as feedback data on the server.

[1776] Step 7: Analyze the sentiment data

[1777] How it works: The server analyzes the audio data using an emotion engine (e.g., IBM Watson Tone Analyzer) and evaluates the participants' emotions, level of understanding, and degree of agreement based on the tone, pitch, and rhythm of their voices.

[1778] Input: Sound data.

[1779] Data processing: Analyzes sound data using an emotion engine to evaluate emotions and comprehension.

[1780] Output: The sentiment analysis results are saved as feedback data on the server.

[1781] Step 8: Generate and communicate real-time feedback

[1782] How it works: The server generates real-time feedback based on all evaluation results and emotion data, and sends the generated feedback to the device.

[1783] Input: Various evaluation results and sentiment analysis results.

[1784] Data processing: Various evaluation results and sentiment analysis results are combined to generate feedback.

[1785] Output: Feedback data is sent to the device, where the user can review it and take any necessary corrective action.

[1786] Step 9: Generate and submit a detailed opinion report

[1787] How it works: After the meeting, the server generates a detailed opinion report based on all the evaluation data.

[1788] Input: All rating data and sentiment analysis results.

[1789] Data processing: Compile various evaluation data to prepare a detailed opinion report.

[1790] Output: Provide users with a feedback report with concrete improvements for the next meeting.

[1791] The above is a concrete explanation of the processing steps of this system.

[1792] (Application example 2)

[1793] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1794] In conventional factory meetings, there are insufficient means to evaluate the quality of communication, the effectiveness of visual support materials, and the emotions and satisfaction of participants in real time, making it difficult to improve productivity and provide efficient feedback.In addition, factory robots cannot adaptively control their operations based on the content of meetings and feedback results, which limits the improvement of productivity throughout the factory.

[1795] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting voice data and converting it into text in real time; means for analyzing the converted text data and evaluating communication; means for photographing and collecting visual support materials and evaluating their effectiveness; means for performing emotion analysis based on the voice data and evaluating participants' levels of understanding and satisfaction; means for providing real-time feedback based on these evaluation results; means for generating a detailed feedback report after the meeting; means for improving productivity within the factory and the quality of human communication; and means for adaptively controlling the operation of factory robots using the analyzed data in real time. This makes it possible to improve the quality of communication and the effectiveness of visual support, accurately evaluate participants' emotions and satisfaction, and provide immediate feedback. Furthermore, factory robots can also adaptively control their operation based on the meeting results, improving the productivity of the entire factory.

[1796] "Audio Data" means voice information collected through a voice input device such as a microphone.

[1797] "Real-time" means having a time response where data is collected, processed, and fed back immediately.

[1798] "String" refers to data that has been converted from audio data into text format.

[1799] "Communication evaluation" refers to the act of analyzing the converted text data and evaluating the appropriateness of the communication, clarity, and validity of the content.

[1800] "Visual support materials" means materials that visually support information, such as presentation materials and whiteboards.

[1801] "Effectiveness of visual support" refers to evaluating how effective visual support materials are in conveying information.

[1802] "Sentiment analysis" refers to a technology that analyzes participants' emotions based on audio data and evaluates their level of understanding and agreement.

[1803] "Real-time feedback" means providing immediate feedback based on collected and analyzed data.

[1804] A "detailed feedback report" refers to a report generated after a meeting that includes communication evaluation results, sentiment analysis results, specific improvement suggestions, etc.

[1805] "Productivity improvement" means optimizing factory production processes to increase efficiency and results.

[1806] "Quality of communication" refers to indicators that reflect the clarity, appropriateness, and level of understanding of information exchange.

[1807] "Data analysis" refers to the act of analyzing collected data and extracting valuable information.

[1808] "Factory robot motion control" refers to the technology that manages the movements and actions of robots in factories to accomplish designated tasks.

[1809] System Configuration

[1810] The present invention provides a system for making meetings in a factory more efficient and improving productivity. The configuration and operation of the system for carrying out the invention will be described below.

[1811] 1. Hardware Configuration

[1812] This system uses the following hardware:

[1813] Microphone: A device for collecting audio data.

[1814] Camera: A device for capturing visual support materials.

[1815] Robot body: A device that moves around the factory and collects, analyzes, and provides feedback on various data.

[1816] Display: A device for displaying feedback and reports.

[1817] 2. Software Configuration

[1818] The system software utilizes the following technologies:

[1819] Speech recognition: Converts collected voice data into text in real time using the Google Cloud Speech-to-Text API.

[1820] Natural language processing: Using generative AI models such as GPT-4, the converted text data is analyzed and communication is evaluated.

[1821] Image Recognition: Analyze visual support materials using OpenCV and evaluate their effectiveness.

[1822] Sentiment analysis: IBM Watson Tone Analyzer is used to analyze voice tone and pitch to assess participants' emotions.

[1823] Real-time data processing and communication: Process data in real time and provide feedback using the MQTT protocol.

[1824] Processing flow

[1825] 1. Data Collection

[1826] When a user starts a meeting, the robot's microphone and camera collect audio and visual support material, which is then transmitted in real time to a server.

[1827] 2. Speech recognition and conversion

[1828] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text format.

[1829] 3. Natural Language Processing (NLP)

[1830] The converted text data is analyzed using GPT-4 to evaluate the quality of communication.

[1831] 4. Visual Support Assessment

[1832] Use image recognition with OpenCV to evaluate the content of presentation materials and whiteboards.

[1833] 5. Sentiment analysis

[1834] IBM Watson Tone Analyzer analyzes the tone and pitch of audio data to assess participants' emotions in real time.

[1835] 6. Real-time feedback generation

[1836] The server generates real-time feedback based on the analysis results and displays it on a display or mobile device.

[1837] 7. Generate detailed feedback reports

[1838] After the meeting, the server generates a detailed feedback report and provides it to the user.

[1839] Specific examples

[1840] A meeting to improve production line efficiency is held in a factory conference room. When a user starts a meeting, a robot collects audio and whiteboard content. The speech is then converted into text using the Google Cloud Speech-to-Text API, and GPT-4 analyzes the content of the communication. OpenCV also evaluates the diagrams and text on the whiteboard, and IBM Watson Tone Analyzer analyzes the emotions of the participants. Based on this data, real-time feedback is displayed on the screen, and any necessary improvements are immediately implemented. After the meeting, a detailed feedback report is generated, providing suggestions for improvement for the next meeting.

[1841] Prompt Sentence Examples

[1842] Analyze the following text data, evaluate the quality of communication, visual support, and analyze sentiment to generate real-time feedback, and generate a detailed feedback report after the meeting.

[1843] Text data:

[1844] Participant A: How can we increase production speed?

[1845] Participant B: It's impossible unless we hire more staff.

[1846] Participant C: No, if we optimize the process, we may be able to manage without increasing manpower.

[1847] Visual Data:

[1848] (Whiteboard image)

[1849] Output formats:

[1850] 1. Communication Evaluation

[1851] 2. Visual Support Evaluation

[1852] 3. Sentiment analysis results

[1853] 4. Real-time feedback

[1854] 5. Detailed feedback report

[1855] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1856] Step 1:

[1857] When a user starts a meeting, the device's microphone and camera begin to collect audio and visual support data. The audio and video data are collected in real time and sent to the server. The input is the audio and video data, and the output is the collected raw data.

[1858] Step 2:

[1859] The server uses the Google Cloud Speech-to-Text API to convert the transmitted voice data into text in real time. This conversion process converts the phonological information of the voice into text data. The input is voice data, and the output is text data converted from the voice into text format.

[1860] Step 3:

[1861] The server uses a generative AI model such as GPT-4 to analyze the converted text data. Specifically, it uses natural language processing (NLP) technology to evaluate the content and quality of the communication, determining factors such as clarity of explanation, appropriateness to the question, and use of specific examples. The input is text data, and the output is the evaluation result of the communication.

[1862] Step 4:

[1863] The server uses OpenCV to analyze the visual support materials (image data) sent from the device. This includes the readability of the images and the placement of figures and text. The analysis results are used to evaluate the effectiveness of the visual support. The input is the image data, and the output is the evaluation result of the visual support.

[1864] Step 5:

[1865] The server uses IBM Watson Tone Analyzer to analyze the tone and pitch of the audio data and perform sentiment analysis. This analysis evaluates participants' emotions, understanding, and degree of agreement. The input is the audio data, and the output is the sentiment analysis results.

[1866] Step 6:

[1867] The server generates real-time feedback based on these evaluation results and displays it on the terminal display or mobile device. This feedback includes immediately actionable improvement measures. The input is communication evaluation, visual support evaluation, and sentiment analysis data, and the output is real-time feedback.

[1868] Step 7:

[1869] After the meeting, the server generates a detailed feedback report based on all the evaluation data. This report includes the communication evaluation results, the effectiveness of visual support, the results of sentiment analysis, and specific improvement suggestions for the next meeting. The input is all the evaluation data, and the output is a detailed feedback report.

[1870] Step 8:

[1871] The server adaptively controls the factory robot's operation based on the data analyzed in real time. This includes adjusting the factory robot's operation patterns and work procedures based on the meeting content and feedback results. The input is the meeting analysis data and feedback results, and the output is the factory robot's operation control commands.

[1872] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1873] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1874] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1875] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1876] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1877] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1878] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1879] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1880] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1881] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1882] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1883] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1884] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1885] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1886] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific p...

Claims

1. A means of collecting audio data and converting it into text in real time; A means for analyzing the converted character string data and evaluating the communication; A means of photographing and collecting visual support materials and evaluating their effectiveness; A means of evaluating participants' understanding and satisfaction by conducting sentiment analysis based on audio data; A means of providing real-time feedback based on these assessment results; A means to generate a detailed feedback report after the meeting; A system including:

2. 10. The system of claim 1, further comprising means for generating a follow-up plan for a particular participant based on the evaluation results.

3. 2. The system according to claim 1, wherein said real-time feedback means notifies the user of the feedback content at the user's terminal, enabling the user to take corrective measures on the spot.

Images