system

Abstract

Provide a system. [Solution] means for acquiring audio data; means for converting the voice data into text data; means for proofreading the text data; means for tagging the nuance of a statement based on the text data; means for analyzing the tagged text data and evaluating the degree of purpose achievement of the meeting; means for generating and providing feedback based on said evaluation; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the spread of online meetings, it is becoming increasingly difficult to properly evaluate the efficiency and effectiveness of meetings. Meeting minutes alone, in particular, make it difficult to properly grasp the progress of the meeting and the nuances of what was said, making it impossible to accurately determine whether the meeting's objectives were achieved. There is also a need to evaluate the quality and balance of communication between participants and identify specific areas for improvement, but current systems do not adequately address this. Therefore, it is necessary to develop a system that provides specific feedback to improve the overall efficiency of meetings and the quality of what is said. [Means for solving the problem]

[0005] The present invention provides the following means to solve the above-mentioned problems. It is a system including a means for acquiring audio data, a means for converting the audio data into text data, a means for proofreading the text data, a means for tagging the nuances of speech based on the text data, and a means for analyzing the tagged text data and evaluating the degree to which the purpose of the meeting was achieved. The system also includes a means for generating feedback based on the evaluation and providing it to the organizer. Furthermore, by adding a means for adding speaker information and speech time information to the audio data, evaluation of each speaker is possible, and by including a means for correcting mistranslations and grammatical errors in the text data, accurate minutes can be generated. This enables specific feedback on the degree to which the purpose of the meeting was achieved and areas for improvement, supporting high-quality meeting management.

[0006] "Audio data" is data that includes a recording of words and sounds spoken during a conference.

[0007] "Text data" is data in which voice data is expressed as characters.

[0008] "Proofreading" is the process of correcting mistranslations and grammatical errors contained in text data.

[0009] "Nuance" indicates emotions and intentions based on facial expressions, tone, etc. of what is being said.

[0010] "Tagging" is the process of attaching relevant information and attributes to text data.

[0011] "Analysis" is the process of examining data in detail to reveal its content and patterns.

[0012] "Evaluation" is the process of determining the degree of achievement and efficiency of objectives based on data.

[0013] "Feedback" is the process of providing information about meeting performance and areas for improvement.

[0014] A "means" refers to a method or device used to achieve a particular purpose. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0037] System flow overview

[0038] System configuration

[0039] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0040] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0041] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text.

[0042] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0043] 5. Data analysis method: Analyze the tagged text data and evaluate the achievement of the meeting objectives.

[0044] 6. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[0045] Program processing overview

[0046] Acquiring voice data and converting voice to text

[0047] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[0048] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[0049] Text Proofreading

[0050] Server: Proofread the generated text data to ensure there are no mistranslations or grammatical errors. This proofreading process is very important to ensure the accuracy of the text information. After proofreading, the text is easier to read and the content is more accurate.

[0051] Nuance tagging of statements

[0052] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[0053] Data analysis and feedback

[0054] Server: Analyzes the tagged text data and evaluates the achievement of the meeting's objectives. This evaluation includes the progress of the meeting, balance of comments, time management, etc. The evaluation results are generated as specific feedback and provided to the organizer.

[0055] Specific examples

[0056] Acquiring and converting audio data

[0057] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0058] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0059] Proofreading text and tagging nuances

[0060] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[0061] Analyzing data and providing feedback

[0062] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0063] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[0067] Step 2:

[0068] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[0069] Step 3:

[0070] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[0071] Step 4:

[0072] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0073] Step 5:

[0074] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[0075] Step 6:

[0076] Server: Sends the nuance-tagged text data to the analysis system, which evaluates the progress of the meeting and the degree to which the objectives were achieved. Evaluation points include time management and balance of speech.

[0077] Step 7:

[0078] Server: Generates feedback based on the evaluation results received from the analysis system. This feedback provides an overall assessment of the meeting and specific areas for improvement.

[0079] Step 8:

[0080] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[0081] Step 9:

[0082] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[0083] The above is the specific processing flow of this system.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Current online meeting systems lack a means to objectively evaluate the progress of meetings and the degree to which their objectives were achieved. This makes it difficult for organizers to obtain specific feedback to evaluate and improve the results of meetings. Furthermore, automated processes for accurately recording what is said during meetings and understanding the nuances of what is said are also uncommon. This makes it difficult to improve meeting efficiency.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the proofread text data, means for analyzing the tagged text data and evaluating the degree of achievement of the conference goal, and means for generating feedback based on the evaluation and providing it to the organizer. This makes it possible to objectively evaluate the progress of the conference and the degree of achievement of the goal, and to provide specific feedback to the organizer.

[0089] "Audio Data" refers to audio information recorded during a conference.

[0090] "Text data" refers to data in which voice data is converted into text information.

[0091] "Proofreading" refers to correcting mistranslations and grammatical errors in converted text data.

[0092] "Nuance tagging" refers to adding tags that indicate the emotion or intention of a statement based on text data.

[0093] "Analysis" refers to analyzing data based on tagged text data and evaluating the degree to which the meeting's objectives were achieved.

[0094] "Feedback" refers to evaluations and improvement suggestions generated based on the analysis results and provided to the meeting organizer.

[0095] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0096] System flow overview

[0097] Step 1: Acquire audio data

[0098] User: When the conference host starts the conference, the user enables the recording function on the device and records the audio data until the end of the conference. After the conference ends, the user uploads the recorded audio file to the server. Uploading is done using a dedicated uploader application or a browser.

[0099] Step 2: Speech to text conversion

[0100] Server: After receiving the uploaded audio files, the server uses a speech recognition system to convert the audio data into text data. The speech recognition system typically uses the Google® Cloud Speech-to-Text API. This system analyzes the received audio data and generates text data for each speaker. For example, the format might be "Tanaka-san (09:00:00): Thank you for your time today."

[0101] Step 3: Proofread your text

[0102] Server: Use the Content Moderation API to proofread the generated text data. This process corrects text translation errors and grammatical errors. After proofreading, the text data becomes more accurate and easier to read. It is recommended to use the Grammarly API as the Content Moderation API.

[0103] Step 4: Nuance tagging

[0104] Server: The proofread text data is sent to a sentiment analysis engine, which tags the nuances of the utterances. This analysis assigns tags that indicate the emotion and intention of the utterance. The IBM Watson (registered trademark) Tone Analyzer is commonly used as the sentiment analysis engine. For example, the format is "Tanaka-san (09:00:00): Thank you for your time today. [Positive]."

[0105] Step 5: Data analysis

[0106] Server: To analyze the tagged text data, data analysis libraries such as Python's Pandas and NumPy are used. This allows for evaluation of the progress of the meeting, balance of speeches, time management, etc. As a result of the analysis, the degree to which the meeting's objectives were achieved is scored.

[0107] Step 6: Provide feedback

[0108] Server: The server generates specific feedback based on the analysis results and provides it to the organizer. The feedback includes an overall evaluation of the conference and suggestions for improvement, making it useful for the organizer. The feedback is provided in the form of a report, which the organizer can receive by email or on the platform.

[0109] Specific examples

[0110] Acquiring and converting audio data

[0111] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0112] Server: The server receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0113] Proofreading text and tagging nuances

[0114] Server: Proofreads the generated text data and corrects typos. Then, a sentiment analysis engine is used to tag the corrected text with nuances such as "positive" or "negative." For example, a tag is added like this: "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Positive]."

[0115] Analyzing data and providing feedback

[0116] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0117] Prompt Sentence Examples

[0118] Below are examples of specific prompts used at each step:

[0119] Voice data conversion prompt

[0120] Example of a prompt to convert voice data to text:

[0121] Audio Data: <Recorded audio file>

[0122] Settings: Encoding = "LINEAR16", Sample rate = 16000, Language code = "ja-JP"

[0123] Proofreading prompt

[0124] Example of a proofreading prompt:

[0125] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0126] Nuanced tagging prompts

[0127] Nuance tagging prompt example:

[0128] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0129] Model: Sentiment Analysis Model 2023

[0130] The above is an embodiment of the present invention, which can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0131] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0132] Step 1: Acquire audio data

[0133] User: When the meeting host starts a meeting, the recording function of the device is enabled. The recording function uses the device's built-in microphone or an external microphone to record the meeting audio in real time. After the meeting ends, the recorded audio file is uploaded to the server.

[0134] Input: Audio recorded during the meeting.

[0135] Output: Audio file uploaded to the server.

[0136] Specific operation: After the meeting ends, the user clicks the "upload" button on the online platform to upload the recorded file to the server.

[0137] Step 2: Speech to text conversion

[0138] Server: The server receives the uploaded audio files. Once the files are received, it calls a speech recognition system (e.g., Google Cloud Speech-to-Text API) to convert the audio files into text data. The speech recognition system analyzes the audio data and generates text data for each speaker.

[0139] Input: An audio file.

[0140] Output: Text data containing utterances by time.

[0141] Specific operation: The server sends the audio data to the speech recognition API and receives the converted text data, which includes a timestamp and the speaker's name.

[0142] Step 3: Proofread your text

[0143] Server: Use a Content Moderation API (e.g., Grammarly API) to proofread the generated text data. The proofreading process corrects spelling errors and grammatical errors.

[0144] Input: The converted text data.

[0145] Output: Text data with mistranslations and grammatical errors corrected.

[0146] What happens: The server sends the text data to the proofreading API and receives the corrected text along with the plain text. The corrected items include typos and grammatical errors.

[0147] Step 4: Nuance tagging

[0148] Server: Analyzes the nuances of the speech based on the proofread text data. This analysis is performed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) and tags are added to indicate the sentiment and intent of the speech.

[0149] Input: Proofread text data.

[0150] Output: Nuance-tagged text data.

[0151] How it works: The server sends the proofread text data to the sentiment analysis engine and receives sentiment tags such as positive, negative, or neutral.

[0152] Step 5: Data analysis

[0153] Server: Analyzes the tagged text data using a data analysis system (e.g., Python's Pandas, NumPy, and machine learning algorithms) and evaluates the achievement of the meeting's objectives, including the progress of the meeting, balance of speeches, and time management.

[0154] Input: Tagged text data.

[0155] Output: Score and feedback data to evaluate the achievement of the meeting objectives.

[0156] Specific operation: The server inputs the tagged text data into a data analysis algorithm, evaluates the frequency of comments and time allocation, and outputs a score indicating the degree to which the meeting's objectives were achieved.

[0157] Step 6: Provide feedback

[0158] Server: The server generates feedback based on the analysis results and provides it to the organizer. This feedback includes an overall evaluation of the conference and suggestions for improvement, and is provided in the form of a report.

[0159] User: The user receives the feedback provided by the server and uses it as a reference for the next meeting.

[0160] Input: Analysis results.

[0161] Output: Feedback report provided to organizer.

[0162] Specific operation: The server compiles the analysis results into a feedback report and sends it to the organizer by email. Users can also check the report on the platform.

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] Autonomous vehicles require accurate understanding of passenger satisfaction and the status of requests, and provide specific feedback to improve service quality. However, currently, there is a lack of effective means to analyze communication between passengers and vehicles and generate feedback. It is necessary to solve this problem and improve passenger satisfaction.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a means for proofreading the text data. This makes it possible to accurately grasp the content of communication between passengers and autonomous vehicles. The server also includes a means for tagging the nuances of speech based on the text data, a means for analyzing the tagged text data to evaluate passenger satisfaction and response to requests in the autonomous vehicle, and a means for generating and providing feedback based on the evaluation. This makes it possible to improve the service quality of autonomous vehicles, increase passenger satisfaction, and provide a comfortable riding experience.

[0168] "Voice data" refers to data that records audio information, such as conversations and sounds with passengers inside an autonomous vehicle.

[0169] "Text data" is audio data converted into text information, and is data that records the content in an abstract manner.

[0170] "Proofreading" is the process of correcting typographical and grammatical errors in text data to make it accurate and easy to read.

[0171] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[0172] "Analysis" refers to the process of evaluating tagged text data and analyzing passenger satisfaction and requests within the autonomous vehicle.

[0173] "Evaluation" is the act of determining the degree of achievement of specific standards or objectives based on the results of analysis.

[0174] "Feedback" refers to information such as specific comments and suggestions that are generated based on the evaluation results.

[0175] "Service Quality" refers to the overall quality and performance of the service provided by an automated vehicle.

[0176] "Passenger satisfaction" is an indicator that shows the level of satisfaction felt by passengers using autonomous vehicles.

[0177] "Request response status" refers to the status of how appropriately passenger requests and demands are handled.

[0178] System Overview

[0179] This invention relates to a system for analyzing communication between passengers and the vehicle in an autonomous vehicle and providing feedback. This system includes a series of processes from acquiring voice data to evaluating it and generating feedback.

[0180] Acquiring and converting audio data

[0181] Server: To obtain voice data, collect voice data from microphones installed inside the autonomous vehicle. The collected voice data is converted into text data using a voice recognition system. For this voice recognition system, for example, the speech_recognition library is used.

[0182] Proofreading text data

[0183] Server: The converted text data is proofread to correct typos and grammatical errors. This proofreading process uses, for example, the TextBlob library.

[0184] Nuance tagging of statements

[0185] Server: Based on the proofread text data, tagging is performed to identify nuances that indicate the sentiment and intent of the speech. For tagging, the server uses the nltk library, for example, to classify the sentiment of words.

[0186] Analyzing data and providing feedback

[0187] Server: Analyzes the tagged text data and evaluates passenger satisfaction and request responsiveness. This evaluation includes determining the degree of achievement against specific criteria and objectives. Feedback generated based on the analysis results is provided to the operation manager. This feedback generation process allows for the improvement of the quality of service of the autonomous vehicle.

[0188] The specific hardware and software used

[0189] Hardware: Smartphones, microphones in self-driving vehicles

[0190] Software: Python program, speech_recognition library, TextBlob library, nltk library

[0191] Specific examples

[0192] For example, suppose a passenger reports a problem in an autonomous vehicle. The audio data is picked up by a microphone and converted into text data using a speech recognition system. This text data is then proofread using the TextBlob library and tagged with emotional nuances using the nltk library. Finally, the tagged data is analyzed to evaluate the passenger's level of dissatisfaction and how well the request was handled, and feedback is generated for the operation manager, such as "Passengers are very satisfied. We are providing good service."

[0193] Prompt Sentence Examples

[0194] "Analyze audio files recorded inside the autonomous vehicle to generate feedback that assesses passenger satisfaction and how well requests are handled."

[0195] In this way, by properly analyzing communication with passengers inside autonomous vehicles and providing feedback, operation managers can improve service quality and increase passenger satisfaction.

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] Acquiring voice data and converting it to text

[0199] The server acquires voice data from microphones installed in the autonomous vehicle. The acquired voice data is converted into text data using a voice recognition system. Specifically, the voice data is converted into character string information using the Python speech_recognition library. The input is an audio file, and the output is text data.

[0200] Step 2:

[0201] Proofreading text data

[0202] The server receives the converted text data as input and corrects typos and grammatical errors. This process uses the TextBlob library. Specifically, TextBlob is used to automatically correct grammatical errors in the text data. The input is the converted text data, and the output is the proofread text data.

[0203] Step 3:

[0204] Nuance tagging

[0205] The server receives the proofread text data and tags it with nuances that indicate the sentiment and intent of the utterance. This is done using the nltk library. Specifically, nltk is used to classify the sentiment of each word and assign tags such as positive, negative, and neutral. The input is the proofread text data, and the output is text data with nuance tags.

[0206] Step 4:

[0207] Data analysis

[0208] The server analyzes the tagged text data to evaluate passenger satisfaction and the response to their requests. This analysis determines the degree of achievement of specific criteria or objectives. Specifically, it calculates the number of positive, negative, and neutral tags and evaluates satisfaction based on that. The input is text data with nuanced tags, and the output is the analysis results.

[0209] Step 5:

[0210] Generating and Providing Feedback

[0211] The server generates feedback based on the analysis results and provides that feedback to the operation manager. Specifically, it generates specific suggestions and points to note for improving the service as text based on the analysis results. The input is the analysis results, and the output is the feedback text.

[0212] Specific examples of processing

[0213] 1. Voice data is acquired from the microphone saying, "The road is congested, please take a detour."

[0214] 2. The acquired voice data is converted into text data.

[0215] 3. The text data is proofread and typos are corrected, such as "The road is congested, please take a detour."

[0216] 4. The proofread text is tagged with nuances such as "The road is congested, so please take a detour [negative]."

[0217] 5. Analyze the tagged text data and evaluate the passenger dissatisfaction level.

[0218] 6. Based on the analysis results, feedback such as "Passengers are highly dissatisfied, so it is recommended that you consider proposing a detour route" is generated.

[0219] In this way, after going through each processing step, the autonomous vehicle operation manager can obtain specific feedback to improve the quality of service.

[0220] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0221] The present invention relates to a system that acquires audio data from an online conference, converts it into text data, proofreads it, tags it with nuances, analyzes it, and combines it with an emotion engine to recognize the user's emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[0222] System flow overview

[0223] System configuration

[0224] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0225] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0226] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text data.

[0227] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0228] 5. Emotion engine: Recognizes the user's emotions from voice data and generates emotional information.

[0229] 6. Data analysis method: Analyze tagged text data and sentiment data to evaluate the achievement of the meeting objectives.

[0230] 7. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[0231] Program processing overview

[0232] Acquiring voice data and converting voice to text

[0233] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[0234] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[0235] Text Proofreading

[0236] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0237] Nuance tagging of statements

[0238] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[0239] Use of emotion engine

[0240] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, speed, etc. of the voice to recognize the user's emotion. For example, it generates emotion information such as "relaxed," "tense," or "angry."

[0241] Data analysis and feedback

[0242] Server: Analyzes the nuance-tagged text data and the emotion data generated by the emotion engine. The analysis system evaluates the progress of the meeting and the degree to which the objectives were achieved, and generates more specific feedback taking into account the emotion information. The evaluation results are provided to the organizer as specific feedback.

[0243] Specific examples

[0244] Acquiring and converting audio data

[0245] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0246] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0247] Proofreading text and tagging nuances

[0248] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[0249] Use of emotion engine

[0250] Server: Sends the voice data to the emotion engine and analyzes it. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, it assigns emotional information in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[0251] Analyzing data and providing feedback

[0252] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0253] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0254] The processing flow will be explained below.

[0255] Step 1:

[0256] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[0257] Step 2:

[0258] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[0259] Step 3:

[0260] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[0261] Step 4:

[0262] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0263] Step 5:

[0264] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[0265] Step 6:

[0266] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions and generate emotion data. For example, emotional information such as "relaxed," "tense," or "angry" is generated.

[0267] Step 7:

[0268] Server: Sends the nuance-tagged text data and the emotion data generated by the emotion engine to the analysis system, which evaluates the progress of the meeting and the degree to which its objectives have been achieved based on the text data and emotion data.

[0269] Step 8:

[0270] Server: Generates feedback based on the evaluation results received from the analysis system, including an overall evaluation of the meeting and specific improvements.

[0271] Step 9:

[0272] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[0273] Step 10:

[0274] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[0275] The above is the specific processing flow of this system.

[0276] Example 2

[0277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0278] With conventional online meeting systems, recording and reviewing meeting content is often done manually, making it difficult to evaluate the efficiency of the meeting and the degree to which the purpose was achieved.In addition, it is not possible to accurately grasp the nuances of what is being said or the emotions of the speaker, which leads to a problem of reduced quality of feedback.

[0279] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for recognizing a user's emotion from the voice data and generating emotion information, means for analyzing the tagged text data and emotion information and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to record online meetings, analyze nuances and emotions, evaluate the degree of goal achievement, and provide high-quality feedback.

[0280] "Audio data" refers to audio information stored in digital format, and is used to record the remarks of conference participants.

[0281] "Text data" refers to data obtained by converting voice data into character information, and expressing the content of a speech in a sentence format.

[0282] "Proofreading" is the process of detecting and correcting typographical and grammatical errors in text data.

[0283] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[0284] "Emotion information" is information generated by analyzing the user's emotions from voice data, and indicates an emotional state such as relaxation, tension, or anger.

[0285] "Evaluation of the degree of achievement of objectives" is the process of analyzing the progress and content of a meeting and evaluating the extent to which the set objectives were achieved.

[0286] "Providing feedback" refers to the task of generating improvements and overall evaluations of the conference based on the analysis results and conveying this information to the organizer.

[0287] The present invention is a system that converts voice data in online conferences into text data, then proofreads, nuance tags, and analyzes the text data, combines it with an emotion engine to recognize user emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[0288] System configuration

[0289] 1. How to acquire audio data: When users hold online meetings, they acquire audio data using the built-in recording function of their devices. The audio data is saved on the device in formats such as WAV or MP3.

[0290] 2. Audio file uploading method: After the conference ends, the user uploads the acquired audio data to the server. The device sends the audio file to the specified server's upload API.

[0291] 3. Speech recognition and text conversion: The server receives the uploaded audio files and converts them into text using a speech recognition system (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text). Analysis generates text data for each speaker.

[0292] 4. Text proofreading: The server sends the generated text data to a proofreading system (e.g., Grammarly API or Microsoft® Text Analytics API) to correct mistranslations and grammatical errors. The proofread text data is then sent back to the server.

[0293] 5. Nuance tagging: Based on the proofread text data, the server uses a nuance tagging system (e.g., a Natural Language Processing (NLP) model) to tag the utterance with a tag that indicates its sentiment or intent. For example, tags such as "positive," "negative," and "question" are applied.

[0294] 6. Emotion Engine: The server sends the voice data to an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer), which recognizes the user's emotion from the tone, pitch, and speed of the voice and generates emotion information. The emotion information includes emotional states such as relaxed, tense, and angry.

[0295] 7. Data analysis method: The server analyzes the nuance-tagged text data and emotional information using an analysis system (e.g., a Python analysis library) to evaluate the achievement of the meeting objectives.

[0296] 8. Feedback provision method: Based on the analysis results, the server generates feedback including an overall evaluation of the conference and suggestions for improvement, and provides it to the organizer. The feedback is sent to the organizer via email or the dashboard.

[0297] Specific examples

[0298] Acquiring and converting audio data

[0299] User: Start an online meeting, enable the recording function, and upload the recorded audio file to the server after the meeting ends.

[0300] Server: The server receives the audio file and converts it into text using a speech recognition system. This generates text such as "Tanaka-san (09:05:30): Let's discuss the progress of this project."

[0301] Proofreading text and tagging nuances

[0302] Server: Proofreads the generated text data, corrects grammatical errors and mistranslations, and then assigns nuances such as "positive" or "negative" to the corrected text.

[0303] Use of emotion engine

[0304] Server: The voice data is sent to the emotion engine and analyzed. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, emotional information is added in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[0305] Analyzing data and providing feedback

[0306] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0307] Prompt Sentence Examples

[0308] "I would like to develop a system that acquires audio data from online meetings, converts it to text data, proofreads it, tags it with nuances, uses an emotion engine to recognize users' emotions, and evaluates the degree to which the meeting achieved its objectives. Specifically, I would like to record the audio of the meeting, convert it to text using a speech recognition system, correct any errors in the text using a proofreading system, tag the text with nuances and emotions, and generate emotion data using an emotion engine. Finally, I would like a system that evaluates the meeting using a data analysis system and provides feedback."

[0309] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0310] Step 1: Acquire audio data

[0311] User: When starting an online meeting, enable the recording function on the device. After the meeting ends, the audio data will be saved on the device.

[0312] Input: Audio data during the meeting

[0313] Output: Recorded audio file (WAV or MP3 format)

[0314] How it works: The device collects audio during the meeting through the microphone and records it in real time. When the meeting ends, the audio file is saved on the device.

[0315] Step 2: Upload your audio file

[0316] User: After the meeting ends, upload the recorded audio file to the server.

[0317] Input: Saved audio file

[0318] Output: Audio file uploaded to the server

[0319] Specific operation: The user selects an audio file on the device and uploads it to the server. The audio file is then transferred to the specified server using the upload API.

[0320] Step 3: Speech recognition and text conversion

[0321] Server: After receiving the audio file, convert the audio data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API).

[0322] Input: Audio file uploaded to the server

[0323] Output: Text data converted from audio

[0324] Specific operation: The server sends the audio file to a speech recognition system, which analyzes the audio and generates text data that includes the speaker's name, the time of speech, and the content of the speech.

[0325] Step 4: Proofread the text data

[0326] Server: Sends the generated text data to a proofreading system (e.g., Grammarly API) to correct mistranslations and grammatical errors.

[0327] Input: Text data generated by a speech recognition system

[0328] Output: Proofread text data

[0329] Specific operation: The server sends the text data to the proofreading system, which detects and automatically corrects typos and grammatical errors. The corrected text data is then returned to the server.

[0330] Step 5: Nuance tagging

[0331] Server: Based on the proofread text data, a nuance tagging system (NLP model) is used to assign tags that indicate the sentiment and intent of the utterances.

[0332] Input: Proofread text data

[0333] Output: Text data with nuance tags

[0334] How it works: The server passes the proofread text data to an NLP model, which generates tags such as "positive" or "negative" based on the content of the speech and assigns them to the text.

[0335] Step 6: Emotion Recognition with the Emotion Engine

[0336] Server: Sends voice data to an emotion engine (e.g., Affectiva), which recognizes the user's emotions from the tone, pitch, speed, etc. of the voice and generates emotion information.

[0337] Input: Audio data

[0338] Output: Data containing emotional information

[0339] Specific operation: The server sends voice data to the emotion engine, which analyzes the voice characteristics to recognize the user's emotional state (relaxed, nervous, angry, etc.) and generates information about the user's emotional state. The generated emotion information is returned to the server.

[0340] Step 7: Data analysis

[0341] Server: Analyzes the nuance-tagged text data and emotional information using an analysis system (data analysis platform) to evaluate the degree to which the meeting's objectives were achieved.

[0342] Input: Nuance-tagged text data, emotion information

[0343] Output: Analysis results

[0344] Specific operation: The server passes the text data and emotion data to the analysis system, which evaluates the frequency of comments, emotional fluctuations, progress, etc. The analysis then calculates the overall achievement of the meeting's objectives.

[0345] Step 8: Generate and provide feedback

[0346] Server: Based on the analysis results, it generates feedback including an overall evaluation of the meeting and suggestions for improvement, and provides it to the organizer.

[0347] Input: Analysis results

[0348] Output: Feedback message

[0349] Specific operation: The server compiles the analysis results and generates feedback such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the following improvements would make the meeting better: - Strengthen time management - Balance speeches." This feedback is provided to the organizer via email or dashboard.

[0350] (Application example 2)

[0351] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] In conventional online meetings and instruction sharing meetings at logistics centers, even though it is possible to capture and convert audio data into text, there is a lack of means to effectively recognize the nuances of what is being said and the emotions of participants, and to properly analyze and evaluate the progress of the meeting and the degree to which its objectives have been achieved. This makes it difficult to understand the overall content of the meeting or identify areas for improvement, which can lead to a decline in the efficiency and quality of the meeting.

[0353] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for generating emotion information from the tagged text data and voice data, means for analyzing the emotion information and tagged text data and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to evaluate the degree of goal achievement of the meeting and provide feedback.

[0354] "Audio data" refers to audio information from meetings or communications recorded in digital format.

[0355] "Text data" is character information obtained by analyzing voice data.

[0356] "Proofreading" is the act of correcting mistranslations and grammatical errors in converted text data.

[0357] "Nuance tagging" refers to assigning labels that indicate the sentiment or intent of a statement based on text data.

[0358] "Emotion information" is data that indicates the user's emotion recognized from voice data or text data.

[0359] "Analysis" is the process of using tagged text data and sentiment information to evaluate the progress of the meeting and the degree to which its objectives were achieved.

[0360] "Evaluation" means analyzing the degree to which the meeting's objectives were achieved and presenting the results quantitatively or qualitatively.

[0361] "Feedback" is information about improvements and progress of the meeting that is generated based on the evaluation results.

[0362] A "system" is a combination of devices and software that executes a series of processes, from acquiring audio data to providing feedback.

[0363] A system for implementing this invention performs a series of processes including acquiring voice data, converting the voice data into text, proofreading the text data, tagging nuances, generating emotional information, analyzing the data, and generating feedback.

[0364] Hardware and software used

[0365] Hardware: Smartphone for primary processing, server for final data processing

[0366] software:

[0367] Speech recognition: Google Cloud Speech-to-Text API

[0368] Emotion recognition: IBM Watson Tone Analyzer

[0369] Data Analysis: Python + Pandas

[0370] Feedback Generation: OpenAI® GPT-3®

[0371] Processing flow

[0372] Step 1: Acquire audio data

[0373] The server uses smartphones to record audio from meetings at the logistics center, and the recorded audio data is stored in digital format.

[0374] Step 2: Speech to text conversion

[0375] The server receives the recorded audio data and converts it into text using the Google Cloud Speech-to-Text API, which also includes speaker information and speaking time information.

[0376] Step 3: Proofread the text data

[0377] The server proofreads the generated text data, using an automatic proofreading engine to correct grammatical errors and mistranslations.

[0378] Step 4: Nuance tagging

[0379] Based on the proofread text data, the server adds tags that indicate the emotion and intention of the statement as nuances.

[0380] Step 5: Generating emotional information

[0381] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information.

[0382] Step 6: Data analysis

[0383] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree to which the goal was achieved. The evaluation is performed using Python and Pandas.

[0384] Step 7: Generate feedback

[0385] The server generates feedback based on the evaluation results and provides it to the organizer. The feedback is generated using OpenAI's GPT-3 model.

[0386] Specific examples

[0387] At a meeting held at a logistics center, the organizer records the audio of the meeting using a smartphone. After recording is complete, the audio data is uploaded to a server. The server then converts the audio to text using the Google Cloud Speech-to-Text API, correcting grammatical errors and mistranslations using an automatic proofreading engine. Nuances are tagged in the corrected text data, and emotional information about participants is generated using IBM Watson Tone Analyzer. The generated text data and emotional information are analyzed using Python and Pandas, and the degree to which the meeting's objectives were achieved is evaluated. Finally, feedback is generated using OpenAI GPT-3 based on the evaluation results and provided to the organizer.

[0388] Prompt Sentence Examples

[0389] The meeting achieved a goal of 0.75. Please give us your suggestions for improvement.

[0390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0391] Step 1:

[0392] A user uses a smartphone to record the audio of a meeting at a logistics center. The audio data is stored digitally on the smartphone. When the recording is complete, the user uploads the audio data to a server.

[0393] Step 2:

[0394] The server receives the uploaded audio data and converts it into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data containing speaker information and speech time information. The server sends the audio data to the API and generates corresponding text for each utterance.

[0395] Step 3:

[0396] The server sends the generated text data to an automatic proofreading engine, which corrects any mistranslations or grammatical errors. The input is the generated text data, and the output is the corrected text data. The automatic proofreading engine analyzes the text and makes appropriate corrections.

[0397] Step 4:

[0398] The server performs nuance tagging based on the corrected text data. The input is the corrected text data, and the output is the nuance-tagged text data. The server analyzes the text data and assigns tags based on its sentiment and intent.

[0399] Step 5:

[0400] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information. The input is voice data, and the output is emotional information. The server analyzes the voice data, and the emotional engine generates the emotional information.

[0401] Step 6:

[0402] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree of goal achievement. The input is the tagged text data and emotion information, and the output is the evaluation result of the degree of goal achievement of the meeting. The server analyzes the data using Python and Pandas and performs a quantitative evaluation.

[0403] Step 7:

[0404] The server generates feedback based on the evaluation results and provides it to the organizer. The input is the evaluation result of the meeting's goal achievement, and the output is feedback including areas for improvement. The feedback is generated using OpenAI's GPT-3 model, which generates prompts based on the evaluation results and creates feedback.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0406] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0408] [Second embodiment]

[0409] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0410] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0411] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0413] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0415] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0416] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0417] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0420] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0421] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0422] System flow overview

[0423] System configuration

[0424] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0425] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0426] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text.

[0427] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0428] 5. Data analysis method: Analyze the tagged text data and evaluate the achievement of the meeting objectives.

[0429] 6. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[0430] Program processing overview

[0431] Acquiring voice data and converting voice to text

[0432] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[0433] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[0434] Text Proofreading

[0435] Server: Proofread the generated text data to ensure there are no mistranslations or grammatical errors. This proofreading process is very important to ensure the accuracy of the text information. After proofreading, the text is easier to read and the content is more accurate.

[0436] Nuance tagging of statements

[0437] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[0438] Data analysis and feedback

[0439] Server: Analyzes the tagged text data and evaluates the achievement of the meeting's objectives. This evaluation includes the progress of the meeting, balance of comments, time management, etc. The evaluation results are generated as specific feedback and provided to the organizer.

[0440] Specific examples

[0441] Acquiring and converting audio data

[0442] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0443] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0444] Proofreading text and tagging nuances

[0445] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[0446] Analyzing data and providing feedback

[0447] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0448] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0449] The processing flow will be explained below.

[0450] Step 1:

[0451] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[0452] Step 2:

[0453] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[0454] Step 3:

[0455] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[0456] Step 4:

[0457] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0458] Step 5:

[0459] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[0460] Step 6:

[0461] Server: Sends the nuance-tagged text data to the analysis system, which evaluates the progress of the meeting and the degree to which the objectives were achieved. Evaluation points include time management and balance of speech.

[0462] Step 7:

[0463] Server: Generates feedback based on the evaluation results received from the analysis system. This feedback provides an overall assessment of the meeting and specific areas for improvement.

[0464] Step 8:

[0465] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[0466] Step 9:

[0467] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[0468] The above is the specific processing flow of this system.

[0469] Example 1

[0470] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0471] Current online meeting systems lack a means to objectively evaluate the progress of meetings and the degree to which their objectives were achieved. This makes it difficult for organizers to obtain specific feedback to evaluate and improve the results of meetings. Furthermore, automated processes for accurately recording what is said during meetings and understanding the nuances of what is said are also uncommon. This makes it difficult to improve meeting efficiency.

[0472] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0473] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the proofread text data, means for analyzing the tagged text data and evaluating the degree of achievement of the conference goal, and means for generating feedback based on the evaluation and providing it to the organizer. This makes it possible to objectively evaluate the progress of the conference and the degree of achievement of the goal, and to provide specific feedback to the organizer.

[0474] "Audio Data" refers to audio information recorded during a conference.

[0475] "Text data" refers to data in which voice data is converted into text information.

[0476] "Proofreading" refers to correcting mistranslations and grammatical errors in converted text data.

[0477] "Nuance tagging" refers to adding tags that indicate the emotion or intention of a statement based on text data.

[0478] "Analysis" refers to analyzing data based on tagged text data and evaluating the degree to which the meeting's objectives were achieved.

[0479] "Feedback" refers to evaluations and improvement suggestions generated based on the analysis results and provided to the meeting organizer.

[0480] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0481] System flow overview

[0482] Step 1: Acquire audio data

[0483] User: When the conference host starts the conference, the user enables the recording function on the device and records the audio data until the end of the conference. After the conference ends, the user uploads the recorded audio file to the server. Uploading is done using a dedicated uploader application or a browser.

[0484] Step 2: Speech to text conversion

[0485] Server: After receiving the uploaded audio files, the server uses a speech recognition system to convert the audio data into text data. Google Cloud Speech-to-Text API is commonly used as the speech recognition system. This system analyzes the received audio data and generates text data for each speaker. For example, the format might be "Tanaka-san (09:00:00): Thank you for your time today."

[0486] Step 3: Proofread your text

[0487] Server: Use the Content Moderation API to proofread the generated text data. This process corrects text translation errors and grammatical errors. After proofreading, the text data becomes more accurate and easier to read. It is recommended to use the Grammarly API as the Content Moderation API.

[0488] Step 4: Nuance tagging

[0489] Server: The proofread text data is sent to a sentiment analysis engine, which tags the nuances of the utterances. This analysis generates tags that indicate the sentiment and intent of the utterances. IBM Watson Tone Analyzer is commonly used as the sentiment analysis engine. For example, the format is something like "Tanaka-san (09:00:00): Thank you for your time today. [Positive]."

[0490] Step 5: Data analysis

[0491] Server: To analyze the tagged text data, data analysis libraries such as Python's Pandas and NumPy are used. This allows for evaluation of the progress of the meeting, balance of speeches, time management, etc. As a result of the analysis, the degree to which the meeting's objectives were achieved is scored.

[0492] Step 6: Provide feedback

[0493] Server: The server generates specific feedback based on the analysis results and provides it to the organizer. The feedback includes an overall evaluation of the conference and suggestions for improvement, making it useful for the organizer. The feedback is provided in the form of a report, which the organizer can receive by email or on the platform.

[0494] Specific examples

[0495] Acquiring and converting audio data

[0496] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0497] Server: The server receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0498] Proofreading text and tagging nuances

[0499] Server: Proofreads the generated text data and corrects typos. Then, a sentiment analysis engine is used to tag the corrected text with nuances such as "positive" or "negative." For example, a tag is added like this: "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Positive]."

[0500] Analyzing data and providing feedback

[0501] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0502] Prompt Sentence Examples

[0503] Below are examples of specific prompts used at each step:

[0504] Voice data conversion prompt

[0505] Example of a prompt to convert voice data to text:

[0506] Audio Data: <Recorded audio file>

[0507] Settings: Encoding = "LINEAR16", Sample rate = 16000, Language code = "ja-JP"

[0508] Proofreading prompt

[0509] Example of a proofreading prompt:

[0510] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0511] Nuanced tagging prompts

[0512] Nuance tagging prompt example:

[0513] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0514] Model: Sentiment Analysis Model 2023

[0515] The above is an embodiment of the present invention, which can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0516] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0517] Step 1: Acquire audio data

[0518] User: When the meeting host starts a meeting, the recording function of the device is enabled. The recording function uses the device's built-in microphone or an external microphone to record the meeting audio in real time. After the meeting ends, the recorded audio file is uploaded to the server.

[0519] Input: Audio recorded during the meeting.

[0520] Output: Audio file uploaded to the server.

[0521] Specific operation: After the meeting ends, the user clicks the "upload" button on the online platform to upload the recorded file to the server.

[0522] Step 2: Speech to text conversion

[0523] Server: The server receives the uploaded audio files. Once the files are received, it calls a speech recognition system (e.g., Google Cloud Speech-to-Text API) to convert the audio files into text data. The speech recognition system analyzes the audio data and generates text data for each speaker.

[0524] Input: An audio file.

[0525] Output: Text data containing utterances by time.

[0526] Specific operation: The server sends the audio data to the speech recognition API and receives the converted text data, which includes a timestamp and the speaker's name.

[0527] Step 3: Proofread your text

[0528] Server: Use a Content Moderation API (e.g., Grammarly API) to proofread the generated text data. The proofreading process corrects spelling errors and grammatical errors.

[0529] Input: The converted text data.

[0530] Output: Text data with mistranslations and grammatical errors corrected.

[0531] What happens: The server sends the text data to the proofreading API and receives the corrected text along with the plain text. The corrected items include typos and grammatical errors.

[0532] Step 4: Nuance tagging

[0533] Server: Analyzes the nuances of the speech based on the proofread text data. This analysis is performed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) and tags are added to indicate the sentiment and intent of the speech.

[0534] Input: Proofread text data.

[0535] Output: Nuance-tagged text data.

[0536] How it works: The server sends the proofread text data to the sentiment analysis engine and receives sentiment tags such as positive, negative, or neutral.

[0537] Step 5: Data analysis

[0538] Server: Analyzes the tagged text data using a data analysis system (e.g., Python's Pandas, NumPy, and machine learning algorithms) and evaluates the achievement of the meeting's objectives, including the progress of the meeting, balance of speeches, and time management.

[0539] Input: Tagged text data.

[0540] Output: Score and feedback data to evaluate the achievement of the meeting objectives.

[0541] Specific operation: The server inputs the tagged text data into a data analysis algorithm, evaluates the frequency of comments and time allocation, and outputs a score indicating the degree to which the meeting's objectives were achieved.

[0542] Step 6: Provide feedback

[0543] Server: The server generates feedback based on the analysis results and provides it to the organizer. This feedback includes an overall evaluation of the conference and suggestions for improvement, and is provided in the form of a report.

[0544] User: The user receives the feedback provided by the server and uses it as a reference for the next meeting.

[0545] Input: Analysis results.

[0546] Output: Feedback report provided to organizer.

[0547] Specific operation: The server compiles the analysis results into a feedback report and sends it to the organizer by email. Users can also check the report on the platform.

[0548] (Application example 1)

[0549] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0550] Autonomous vehicles require accurate understanding of passenger satisfaction and the status of requests, and provide specific feedback to improve service quality. However, currently, there is a lack of effective means to analyze communication between passengers and vehicles and generate feedback. It is necessary to solve this problem and improve passenger satisfaction.

[0551] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0552] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a means for proofreading the text data. This makes it possible to accurately grasp the content of communication between passengers and autonomous vehicles. The server also includes a means for tagging the nuances of speech based on the text data, a means for analyzing the tagged text data to evaluate passenger satisfaction and response to requests in the autonomous vehicle, and a means for generating and providing feedback based on the evaluation. This makes it possible to improve the service quality of autonomous vehicles, increase passenger satisfaction, and provide a comfortable riding experience.

[0553] "Voice data" refers to data that records audio information, such as conversations and sounds with passengers inside an autonomous vehicle.

[0554] "Text data" is audio data converted into text information, and is data that records the content in an abstract manner.

[0555] "Proofreading" is the process of correcting typographical and grammatical errors in text data to make it accurate and easy to read.

[0556] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[0557] "Analysis" refers to the process of evaluating tagged text data and analyzing passenger satisfaction and requests within the autonomous vehicle.

[0558] "Evaluation" is the act of determining the degree of achievement of specific standards or objectives based on the results of analysis.

[0559] "Feedback" refers to information such as specific comments and suggestions that are generated based on the evaluation results.

[0560] "Service Quality" refers to the overall quality and performance of the service provided by an automated vehicle.

[0561] "Passenger satisfaction" is an indicator that shows the level of satisfaction felt by passengers using autonomous vehicles.

[0562] "Request response status" refers to the status of how appropriately passenger requests and demands are handled.

[0563] System Overview

[0564] This invention relates to a system for analyzing communication between passengers and the vehicle in an autonomous vehicle and providing feedback. This system includes a series of processes from acquiring voice data to evaluating it and generating feedback.

[0565] Acquiring and converting audio data

[0566] Server: To obtain voice data, collect voice data from microphones installed inside the autonomous vehicle. The collected voice data is converted into text data using a voice recognition system. For this voice recognition system, for example, the speech_recognition library is used.

[0567] Proofreading text data

[0568] Server: The converted text data is proofread to correct typos and grammatical errors. This proofreading process uses, for example, the TextBlob library.

[0569] Nuance tagging of statements

[0570] Server: Based on the proofread text data, tagging is performed to identify nuances that indicate the sentiment and intent of the speech. For tagging, the server uses the nltk library, for example, to classify the sentiment of words.

[0571] Analyzing data and providing feedback

[0572] Server: Analyzes the tagged text data and evaluates passenger satisfaction and request responsiveness. This evaluation includes determining the degree of achievement against specific criteria and objectives. Feedback generated based on the analysis results is provided to the operation manager. This feedback generation process allows for the improvement of the quality of service of the autonomous vehicle.

[0573] The specific hardware and software used

[0574] Hardware: Smartphones, microphones in self-driving vehicles

[0575] Software: Python program, speech_recognition library, TextBlob library, nltk library

[0576] Specific examples

[0577] For example, suppose a passenger reports a problem in an autonomous vehicle. The audio data is picked up by a microphone and converted into text data using a speech recognition system. This text data is then proofread using the TextBlob library and tagged with emotional nuances using the nltk library. Finally, the tagged data is analyzed to evaluate the passenger's level of dissatisfaction and how well the request was handled, and feedback is generated for the operation manager, such as "Passengers are very satisfied. We are providing good service."

[0578] Prompt Sentence Examples

[0579] "Analyze audio files recorded inside the autonomous vehicle to generate feedback that assesses passenger satisfaction and how well requests are handled."

[0580] In this way, by properly analyzing communication with passengers inside autonomous vehicles and providing feedback, operation managers can improve service quality and increase passenger satisfaction.

[0581] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0582] Step 1:

[0583] Acquiring voice data and converting it to text

[0584] The server acquires voice data from microphones installed in the autonomous vehicle. The acquired voice data is converted into text data using a voice recognition system. Specifically, the voice data is converted into character string information using the Python speech_recognition library. The input is an audio file, and the output is text data.

[0585] Step 2:

[0586] Proofreading text data

[0587] The server receives the converted text data as input and corrects typos and grammatical errors. This process uses the TextBlob library. Specifically, TextBlob is used to automatically correct grammatical errors in the text data. The input is the converted text data, and the output is the proofread text data.

[0588] Step 3:

[0589] Nuance tagging

[0590] The server receives the proofread text data and tags it with nuances that indicate the sentiment and intent of the utterance. This is done using the nltk library. Specifically, nltk is used to classify the sentiment of each word and assign tags such as positive, negative, and neutral. The input is the proofread text data, and the output is text data with nuance tags.

[0591] Step 4:

[0592] Data analysis

[0593] The server analyzes the tagged text data to evaluate passenger satisfaction and the response to their requests. This analysis determines the degree of achievement of specific criteria or objectives. Specifically, it calculates the number of positive, negative, and neutral tags and evaluates satisfaction based on that. The input is text data with nuanced tags, and the output is the analysis results.

[0594] Step 5:

[0595] Generating and Providing Feedback

[0596] The server generates feedback based on the analysis results and provides that feedback to the operation manager. Specifically, it generates specific suggestions and points to note for improving the service as text based on the analysis results. The input is the analysis results, and the output is the feedback text.

[0597] Specific examples of processing

[0598] 1. Voice data is acquired from the microphone saying, "The road is congested, please take a detour."

[0599] 2. The acquired voice data is converted into text data.

[0600] 3. The text data is proofread and typos are corrected, such as "The road is congested, please take a detour."

[0601] 4. The proofread text is tagged with nuances such as "The road is congested, so please take a detour [negative]."

[0602] 5. Analyze the tagged text data and evaluate the passenger dissatisfaction level.

[0603] 6. Based on the analysis results, feedback such as "Passengers are highly dissatisfied, so it is recommended that you consider proposing a detour route" is generated.

[0604] In this way, after going through each processing step, the autonomous vehicle operation manager can obtain specific feedback to improve the quality of service.

[0605] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0606] The present invention relates to a system that acquires audio data from an online conference, converts it into text data, proofreads it, tags it with nuances, analyzes it, and combines it with an emotion engine to recognize the user's emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[0607] System flow overview

[0608] System configuration

[0609] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0610] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0611] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text data.

[0612] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0613] 5. Emotion engine: Recognizes the user's emotions from voice data and generates emotional information.

[0614] 6. Data analysis method: Analyze tagged text data and sentiment data to evaluate the achievement of the meeting objectives.

[0615] 7. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[0616] Program processing overview

[0617] Acquiring voice data and converting voice to text

[0618] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[0619] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[0620] Text Proofreading

[0621] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0622] Nuance tagging of statements

[0623] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[0624] Use of emotion engine

[0625] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, speed, etc. of the voice to recognize the user's emotion. For example, it generates emotion information such as "relaxed," "tense," or "angry."

[0626] Data analysis and feedback

[0627] Server: Analyzes the nuance-tagged text data and the emotion data generated by the emotion engine. The analysis system evaluates the progress of the meeting and the degree to which the objectives were achieved, and generates more specific feedback taking into account the emotion information. The evaluation results are provided to the organizer as specific feedback.

[0628] Specific examples

[0629] Acquiring and converting audio data

[0630] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0631] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0632] Proofreading text and tagging nuances

[0633] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[0634] Use of emotion engine

[0635] Server: Sends the voice data to the emotion engine and analyzes it. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, it assigns emotional information in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[0636] Analyzing data and providing feedback

[0637] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0638] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0639] The processing flow will be explained below.

[0640] Step 1:

[0641] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[0642] Step 2:

[0643] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[0644] Step 3:

[0645] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[0646] Step 4:

[0647] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0648] Step 5:

[0649] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[0650] Step 6:

[0651] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions and generate emotion data. For example, emotional information such as "relaxed," "tense," or "angry" is generated.

[0652] Step 7:

[0653] Server: Sends the nuance-tagged text data and the emotion data generated by the emotion engine to the analysis system, which evaluates the progress of the meeting and the degree to which its objectives have been achieved based on the text data and emotion data.

[0654] Step 8:

[0655] Server: Generates feedback based on the evaluation results received from the analysis system, including an overall evaluation of the meeting and specific improvements.

[0656] Step 9:

[0657] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[0658] Step 10:

[0659] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[0660] The above is the specific processing flow of this system.

[0661] Example 2

[0662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0663] With conventional online meeting systems, recording and reviewing meeting content is often done manually, making it difficult to evaluate the efficiency of the meeting and the degree to which the purpose was achieved.In addition, it is not possible to accurately grasp the nuances of what is being said or the emotions of the speaker, which leads to a problem of reduced quality of feedback.

[0664] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for recognizing a user's emotion from the voice data and generating emotion information, means for analyzing the tagged text data and emotion information and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to record online meetings, analyze nuances and emotions, evaluate the degree of goal achievement, and provide high-quality feedback.

[0665] "Audio data" refers to audio information stored in digital format, and is used to record the remarks of conference participants.

[0666] "Text data" refers to data obtained by converting voice data into character information, and expressing the content of a speech in a sentence format.

[0667] "Proofreading" is the process of detecting and correcting typographical and grammatical errors in text data.

[0668] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[0669] "Emotion information" is information generated by analyzing the user's emotions from voice data, and indicates an emotional state such as relaxation, tension, or anger.

[0670] "Evaluation of the degree of achievement of objectives" is the process of analyzing the progress and content of a meeting and evaluating the extent to which the set objectives were achieved.

[0671] "Providing feedback" refers to the task of generating improvements and overall evaluations of the conference based on the analysis results and conveying this information to the organizer.

[0672] The present invention is a system that converts voice data in online conferences into text data, then proofreads, nuance tags, and analyzes the text data, combines it with an emotion engine to recognize user emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[0673] System configuration

[0674] 1. How to acquire audio data: When users hold online meetings, they acquire audio data using the built-in recording function of their devices. The audio data is saved on the device in formats such as WAV or MP3.

[0675] 2. Audio file uploading method: After the conference ends, the user uploads the acquired audio data to the server. The device sends the audio file to the specified server's upload API.

[0676] 3. Speech recognition and text conversion: The server receives the uploaded audio files and converts them into text using a speech recognition system (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text). Analysis generates text data for each speaker.

[0677] 4. Text Proofreading: The server sends the generated text data to a proofreading system (e.g., Grammarly API or Microsoft Text Analytics API) to correct mistranslations and grammatical errors. The proofread text data is then sent back to the server.

[0678] 5. Nuance tagging: Based on the proofread text data, the server uses a nuance tagging system (e.g., a Natural Language Processing (NLP) model) to tag the utterance with a tag that indicates its sentiment or intent. For example, tags such as "positive," "negative," and "question" are applied.

[0679] 6. Emotion Engine: The server sends the voice data to an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer), which recognizes the user's emotion from the tone, pitch, and speed of the voice and generates emotion information. The emotion information includes emotional states such as relaxed, tense, and angry.

[0680] 7. Data analysis method: The server analyzes the nuance-tagged text data and emotional information using an analysis system (e.g., a Python analysis library) to evaluate the achievement of the meeting objectives.

[0681] 8. Feedback provision method: Based on the analysis results, the server generates feedback including an overall evaluation of the conference and suggestions for improvement, and provides it to the organizer. The feedback is sent to the organizer via email or the dashboard.

[0682] Specific examples

[0683] Acquiring and converting audio data

[0684] User: Start an online meeting, enable the recording function, and upload the recorded audio file to the server after the meeting ends.

[0685] Server: The server receives the audio file and converts it into text using a speech recognition system. This generates text such as "Tanaka-san (09:05:30): Let's discuss the progress of this project."

[0686] Proofreading text and tagging nuances

[0687] Server: Proofreads the generated text data, corrects grammatical errors and mistranslations, and then assigns nuances such as "positive" or "negative" to the corrected text.

[0688] Use of emotion engine

[0689] Server: The voice data is sent to the emotion engine and analyzed. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, emotional information is added in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[0690] Analyzing data and providing feedback

[0691] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0692] Prompt Sentence Examples

[0693] "I would like to develop a system that acquires audio data from online meetings, converts it to text data, proofreads it, tags it with nuances, uses an emotion engine to recognize users' emotions, and evaluates the degree to which the meeting achieved its objectives. Specifically, I would like to record the audio of the meeting, convert it to text using a speech recognition system, correct any errors in the text using a proofreading system, tag the text with nuances and emotions, and generate emotion data using an emotion engine. Finally, I would like a system that evaluates the meeting using a data analysis system and provides feedback."

[0694] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0695] Step 1: Acquire audio data

[0696] User: When starting an online meeting, enable the recording function on the device. After the meeting ends, the audio data will be saved on the device.

[0697] Input: Audio data during the meeting

[0698] Output: Recorded audio file (WAV or MP3 format)

[0699] How it works: The device collects audio during the meeting through the microphone and records it in real time. When the meeting ends, the audio file is saved on the device.

[0700] Step 2: Upload your audio file

[0701] User: After the meeting ends, upload the recorded audio file to the server.

[0702] Input: Saved audio file

[0703] Output: Audio file uploaded to the server

[0704] Specific operation: The user selects an audio file on the device and uploads it to the server. The audio file is then transferred to the specified server using the upload API.

[0705] Step 3: Speech recognition and text conversion

[0706] Server: After receiving the audio file, convert the audio data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API).

[0707] Input: Audio file uploaded to the server

[0708] Output: Text data converted from audio

[0709] Specific operation: The server sends the audio file to a speech recognition system, which analyzes the audio and generates text data that includes the speaker's name, the time of speech, and the content of the speech.

[0710] Step 4: Proofread the text data

[0711] Server: Sends the generated text data to a proofreading system (e.g., Grammarly API) to correct mistranslations and grammatical errors.

[0712] Input: Text data generated by a speech recognition system

[0713] Output: Proofread text data

[0714] Specific operation: The server sends the text data to the proofreading system, which detects and automatically corrects typos and grammatical errors. The corrected text data is then returned to the server.

[0715] Step 5: Nuance tagging

[0716] Server: Based on the proofread text data, a nuance tagging system (NLP model) is used to assign tags that indicate the sentiment and intent of the utterances.

[0717] Input: Proofread text data

[0718] Output: Text data with nuance tags

[0719] How it works: The server passes the proofread text data to an NLP model, which generates tags such as "positive" or "negative" based on the content of the speech and assigns them to the text.

[0720] Step 6: Emotion Recognition with the Emotion Engine

[0721] Server: Sends voice data to an emotion engine (e.g., Affectiva), which recognizes the user's emotions from the tone, pitch, speed, etc. of the voice and generates emotion information.

[0722] Input: Audio data

[0723] Output: Data containing emotional information

[0724] Specific operation: The server sends voice data to the emotion engine, which analyzes the voice characteristics to recognize the user's emotional state (relaxed, nervous, angry, etc.) and generates information about the user's emotional state. The generated emotion information is returned to the server.

[0725] Step 7: Data analysis

[0726] Server: Analyzes the nuance-tagged text data and emotional information using an analysis system (data analysis platform) to evaluate the degree to which the meeting's objectives were achieved.

[0727] Input: Nuance-tagged text data, emotion information

[0728] Output: Analysis results

[0729] Specific operation: The server passes the text data and emotion data to the analysis system, which evaluates the frequency of comments, emotional fluctuations, progress, etc. The analysis then calculates the overall achievement of the meeting's objectives.

[0730] Step 8: Generate and provide feedback

[0731] Server: Based on the analysis results, it generates feedback including an overall evaluation of the meeting and suggestions for improvement, and provides it to the organizer.

[0732] Input: Analysis results

[0733] Output: Feedback message

[0734] Specific operation: The server compiles the analysis results and generates feedback such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the following improvements would make the meeting better: - Strengthen time management - Balance speeches." This feedback is provided to the organizer via email or dashboard.

[0735] (Application example 2)

[0736] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0737] In conventional online meetings and instruction sharing meetings at logistics centers, even though it is possible to capture and convert audio data into text, there is a lack of means to effectively recognize the nuances of what is being said and the emotions of participants, and to properly analyze and evaluate the progress of the meeting and the degree to which its objectives have been achieved. This makes it difficult to understand the overall content of the meeting or identify areas for improvement, which can lead to a decline in the efficiency and quality of the meeting.

[0738] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for generating emotion information from the tagged text data and voice data, means for analyzing the emotion information and tagged text data and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to evaluate the degree of goal achievement of the meeting and provide feedback.

[0739] "Audio data" refers to audio information from meetings or communications recorded in digital format.

[0740] "Text data" is character information obtained by analyzing voice data.

[0741] "Proofreading" is the act of correcting mistranslations and grammatical errors in converted text data.

[0742] "Nuance tagging" refers to assigning labels that indicate the sentiment or intent of a statement based on text data.

[0743] "Emotion information" is data that indicates the user's emotion recognized from voice data or text data.

[0744] "Analysis" is the process of using tagged text data and sentiment information to evaluate the progress of the meeting and the degree to which its objectives were achieved.

[0745] "Evaluation" means analyzing the degree to which the meeting's objectives were achieved and presenting the results quantitatively or qualitatively.

[0746] "Feedback" is information about improvements and progress of the meeting that is generated based on the evaluation results.

[0747] A "system" is a combination of devices and software that executes a series of processes, from acquiring audio data to providing feedback.

[0748] A system for implementing this invention performs a series of processes including acquiring voice data, converting the voice data into text, proofreading the text data, tagging nuances, generating emotional information, analyzing the data, and generating feedback.

[0749] Hardware and software used

[0750] Hardware: Smartphone for primary processing, server for final data processing

[0751] software:

[0752] Speech recognition: Google Cloud Speech-to-Text API

[0753] Emotion recognition: IBM Watson Tone Analyzer

[0754] Data Analysis: Python + Pandas

[0755] Feedback Generation: OpenAI GPT-3

[0756] Processing flow

[0757] Step 1: Acquire audio data

[0758] The server uses smartphones to record audio from meetings at the logistics center, and the recorded audio data is stored in digital format.

[0759] Step 2: Speech to text conversion

[0760] The server receives the recorded audio data and converts it into text using the Google Cloud Speech-to-Text API, which also includes speaker information and speaking time information.

[0761] Step 3: Proofread the text data

[0762] The server proofreads the generated text data, using an automatic proofreading engine to correct grammatical errors and mistranslations.

[0763] Step 4: Nuance tagging

[0764] Based on the proofread text data, the server adds tags that indicate the emotion and intention of the statement as nuances.

[0765] Step 5: Generating emotional information

[0766] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information.

[0767] Step 6: Data analysis

[0768] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree to which the goal was achieved. The evaluation is performed using Python and Pandas.

[0769] Step 7: Generate feedback

[0770] The server generates feedback based on the evaluation results and provides it to the organizer. The feedback is generated using OpenAI's GPT-3 model.

[0771] Specific examples

[0772] At a meeting held at a logistics center, the organizer records the audio of the meeting using a smartphone. After recording is complete, the audio data is uploaded to a server. The server then converts the audio to text using the Google Cloud Speech-to-Text API, correcting grammatical errors and mistranslations using an automatic proofreading engine. Nuances are tagged in the corrected text data, and emotional information about participants is generated using IBM Watson Tone Analyzer. The generated text data and emotional information are analyzed using Python and Pandas, and the degree to which the meeting's objectives were achieved is evaluated. Finally, feedback is generated using OpenAI GPT-3 based on the evaluation results and provided to the organizer.

[0773] Prompt Sentence Examples

[0774] The meeting achieved a goal of 0.75. Please give us your suggestions for improvement.

[0775] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0776] Step 1:

[0777] A user uses a smartphone to record the audio of a meeting at a logistics center. The audio data is stored digitally on the smartphone. When the recording is complete, the user uploads the audio data to a server.

[0778] Step 2:

[0779] The server receives the uploaded audio data and converts it into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data containing speaker information and speech time information. The server sends the audio data to the API and generates corresponding text for each utterance.

[0780] Step 3:

[0781] The server sends the generated text data to an automatic proofreading engine, which corrects any mistranslations or grammatical errors. The input is the generated text data, and the output is the corrected text data. The automatic proofreading engine analyzes the text and makes appropriate corrections.

[0782] Step 4:

[0783] The server performs nuance tagging based on the corrected text data. The input is the corrected text data, and the output is the nuance-tagged text data. The server analyzes the text data and assigns tags based on its sentiment and intent.

[0784] Step 5:

[0785] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information. The input is voice data, and the output is emotional information. The server analyzes the voice data, and the emotional engine generates the emotional information.

[0786] Step 6:

[0787] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree of goal achievement. The input is the tagged text data and emotion information, and the output is the evaluation result of the degree of goal achievement of the meeting. The server analyzes the data using Python and Pandas and performs a quantitative evaluation.

[0788] Step 7:

[0789] The server generates feedback based on the evaluation results and provides it to the organizer. The input is the evaluation result of the meeting's goal achievement, and the output is feedback including areas for improvement. The feedback is generated using OpenAI's GPT-3 model, which generates prompts based on the evaluation results and creates feedback.

[0790] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0791] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0792] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0793] [Third embodiment]

[0794] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0795] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0796] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0797] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0798] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0799] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0800] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0801] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0802] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0803] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0804] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0805] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0806] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0807] System flow overview

[0808] System configuration

[0809] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0810] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0811] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text.

[0812] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0813] 5. Data analysis method: Analyze the tagged text data and evaluate the achievement of the meeting objectives.

[0814] 6. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[0815] Program processing overview

[0816] Acquiring voice data and converting voice to text

[0817] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[0818] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[0819] Text Proofreading

[0820] Server: Proofread the generated text data to ensure there are no mistranslations or grammatical errors. This proofreading process is very important to ensure the accuracy of the text information. After proofreading, the text is easier to read and the content is more accurate.

[0821] Nuance tagging of statements

[0822] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[0823] Data analysis and feedback

[0824] Server: Analyzes the tagged text data and evaluates the achievement of the meeting's objectives. This evaluation includes the progress of the meeting, balance of comments, time management, etc. The evaluation results are generated as specific feedback and provided to the organizer.

[0825] Specific examples

[0826] Acquiring and converting audio data

[0827] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0828] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0829] Proofreading text and tagging nuances

[0830] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[0831] Analyzing data and providing feedback

[0832] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0833] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0834] The processing flow will be explained below.

[0835] Step 1:

[0836] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[0837] Step 2:

[0838] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[0839] Step 3:

[0840] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[0841] Step 4:

[0842] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[0843] Step 5:

[0844] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[0845] Step 6:

[0846] Server: Sends the nuance-tagged text data to the analysis system, which evaluates the progress of the meeting and the degree to which the objectives were achieved. Evaluation points include time management and balance of speech.

[0847] Step 7:

[0848] Server: Generates feedback based on the evaluation results received from the analysis system. This feedback provides an overall assessment of the meeting and specific areas for improvement.

[0849] Step 8:

[0850] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[0851] Step 9:

[0852] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[0853] The above is the specific processing flow of this system.

[0854] Example 1

[0855] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0856] Current online meeting systems lack a means to objectively evaluate the progress of meetings and the degree to which their objectives were achieved. This makes it difficult for organizers to obtain specific feedback to evaluate and improve the results of meetings. Furthermore, automated processes for accurately recording what is said during meetings and understanding the nuances of what is said are also uncommon. This makes it difficult to improve meeting efficiency.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0858] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the proofread text data, means for analyzing the tagged text data and evaluating the degree of achievement of the conference goal, and means for generating feedback based on the evaluation and providing it to the organizer. This makes it possible to objectively evaluate the progress of the conference and the degree of achievement of the goal, and to provide specific feedback to the organizer.

[0859] "Audio Data" refers to audio information recorded during a conference.

[0860] "Text data" refers to data in which voice data is converted into text information.

[0861] "Proofreading" refers to correcting mistranslations and grammatical errors in converted text data.

[0862] "Nuance tagging" refers to adding tags that indicate the emotion or intention of a statement based on text data.

[0863] "Analysis" refers to analyzing data based on tagged text data and evaluating the degree to which the meeting's objectives were achieved.

[0864] "Feedback" refers to evaluations and improvement suggestions generated based on the analysis results and provided to the meeting organizer.

[0865] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[0866] System flow overview

[0867] Step 1: Acquire audio data

[0868] User: When the conference host starts the conference, the user enables the recording function on the device and records the audio data until the end of the conference. After the conference ends, the user uploads the recorded audio file to the server. Uploading is done using a dedicated uploader application or a browser.

[0869] Step 2: Speech to text conversion

[0870] Server: After receiving the uploaded audio files, the server uses a speech recognition system to convert the audio data into text data. Google Cloud Speech-to-Text API is commonly used as the speech recognition system. This system analyzes the received audio data and generates text data for each speaker. For example, the format might be "Tanaka-san (09:00:00): Thank you for your time today."

[0871] Step 3: Proofread your text

[0872] Server: Use the Content Moderation API to proofread the generated text data. This process corrects text translation errors and grammatical errors. After proofreading, the text data becomes more accurate and easier to read. It is recommended to use the Grammarly API as the Content Moderation API.

[0873] Step 4: Nuance tagging

[0874] Server: The proofread text data is sent to a sentiment analysis engine, which tags the nuances of the utterances. This analysis generates tags that indicate the sentiment and intent of the utterances. IBM Watson Tone Analyzer is commonly used as the sentiment analysis engine. For example, the format is something like "Tanaka-san (09:00:00): Thank you for your time today. [Positive]."

[0875] Step 5: Data analysis

[0876] Server: To analyze the tagged text data, data analysis libraries such as Python's Pandas and NumPy are used. This allows for evaluation of the progress of the meeting, balance of speeches, time management, etc. As a result of the analysis, the degree to which the meeting's objectives were achieved is scored.

[0877] Step 6: Provide feedback

[0878] Server: The server generates specific feedback based on the analysis results and provides it to the organizer. The feedback includes an overall evaluation of the conference and suggestions for improvement, making it useful for the organizer. The feedback is provided in the form of a report, which the organizer can receive by email or on the platform.

[0879] Specific examples

[0880] Acquiring and converting audio data

[0881] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[0882] Server: The server receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[0883] Proofreading text and tagging nuances

[0884] Server: Proofreads the generated text data and corrects typos. Then, a sentiment analysis engine is used to tag the corrected text with nuances such as "positive" or "negative." For example, a tag is added like this: "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Positive]."

[0885] Analyzing data and providing feedback

[0886] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[0887] Prompt Sentence Examples

[0888] Below are examples of specific prompts used at each step:

[0889] Voice data conversion prompt

[0890] Example of a prompt to convert voice data to text:

[0891] Audio Data: <Recorded audio file>

[0892] Settings: Encoding = "LINEAR16", Sample rate = 16000, Language code = "ja-JP"

[0893] Proofreading prompt

[0894] Example of a proofreading prompt:

[0895] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0896] Nuanced tagging prompts

[0897] Nuance tagging prompt example:

[0898] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[0899] Model: Sentiment Analysis Model 2023

[0900] The above is an embodiment of the present invention, which can improve the efficiency of online meetings and provide useful feedback to the organizer.

[0901] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0902] Step 1: Acquire audio data

[0903] User: When the meeting host starts a meeting, the recording function of the device is enabled. The recording function uses the device's built-in microphone or an external microphone to record the meeting audio in real time. After the meeting ends, the recorded audio file is uploaded to the server.

[0904] Input: Audio recorded during the meeting.

[0905] Output: Audio file uploaded to the server.

[0906] Specific operation: After the meeting ends, the user clicks the "upload" button on the online platform to upload the recorded file to the server.

[0907] Step 2: Speech to text conversion

[0908] Server: The server receives the uploaded audio files. Once the files are received, it calls a speech recognition system (e.g., Google Cloud Speech-to-Text API) to convert the audio files into text data. The speech recognition system analyzes the audio data and generates text data for each speaker.

[0909] Input: An audio file.

[0910] Output: Text data containing utterances by time.

[0911] Specific operation: The server sends the audio data to the speech recognition API and receives the converted text data, which includes a timestamp and the speaker's name.

[0912] Step 3: Proofread your text

[0913] Server: Use a Content Moderation API (e.g., Grammarly API) to proofread the generated text data. The proofreading process corrects spelling errors and grammatical errors.

[0914] Input: The converted text data.

[0915] Output: Text data with mistranslations and grammatical errors corrected.

[0916] What happens: The server sends the text data to the proofreading API and receives the corrected text along with the plain text. The corrected items include typos and grammatical errors.

[0917] Step 4: Nuance tagging

[0918] Server: Analyzes the nuances of the speech based on the proofread text data. This analysis is performed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) and tags are added to indicate the sentiment and intent of the speech.

[0919] Input: Proofread text data.

[0920] Output: Nuance-tagged text data.

[0921] How it works: The server sends the proofread text data to the sentiment analysis engine and receives sentiment tags such as positive, negative, or neutral.

[0922] Step 5: Data analysis

[0923] Server: Analyzes the tagged text data using a data analysis system (e.g., Python's Pandas, NumPy, and machine learning algorithms) and evaluates the achievement of the meeting's objectives, including the progress of the meeting, balance of speeches, and time management.

[0924] Input: Tagged text data.

[0925] Output: Score and feedback data to evaluate the achievement of the meeting objectives.

[0926] Specific operation: The server inputs the tagged text data into a data analysis algorithm, evaluates the frequency of comments and time allocation, and outputs a score indicating the degree to which the meeting's objectives were achieved.

[0927] Step 6: Provide feedback

[0928] Server: The server generates feedback based on the analysis results and provides it to the organizer. This feedback includes an overall evaluation of the conference and suggestions for improvement, and is provided in the form of a report.

[0929] User: The user receives the feedback provided by the server and uses it as a reference for the next meeting.

[0930] Input: Analysis results.

[0931] Output: Feedback report provided to organizer.

[0932] Specific operation: The server compiles the analysis results into a feedback report and sends it to the organizer by email. Users can also check the report on the platform.

[0933] (Application example 1)

[0934] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0935] Autonomous vehicles require accurate understanding of passenger satisfaction and the status of requests, and provide specific feedback to improve service quality. However, currently, there is a lack of effective means to analyze communication between passengers and vehicles and generate feedback. It is necessary to solve this problem and improve passenger satisfaction.

[0936] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0937] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a means for proofreading the text data. This makes it possible to accurately grasp the content of communication between passengers and autonomous vehicles. The server also includes a means for tagging the nuances of speech based on the text data, a means for analyzing the tagged text data to evaluate passenger satisfaction and response to requests in the autonomous vehicle, and a means for generating and providing feedback based on the evaluation. This makes it possible to improve the service quality of autonomous vehicles, increase passenger satisfaction, and provide a comfortable riding experience.

[0938] "Voice data" refers to data that records audio information, such as conversations and sounds with passengers inside an autonomous vehicle.

[0939] "Text data" is audio data converted into text information, and is data that records the content in an abstract manner.

[0940] "Proofreading" is the process of correcting typographical and grammatical errors in text data to make it accurate and easy to read.

[0941] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[0942] "Analysis" refers to the process of evaluating tagged text data and analyzing passenger satisfaction and requests within the autonomous vehicle.

[0943] "Evaluation" is the act of determining the degree of achievement of specific standards or objectives based on the results of analysis.

[0944] "Feedback" refers to information such as specific comments and suggestions that are generated based on the evaluation results.

[0945] "Service Quality" refers to the overall quality and performance of the service provided by an automated vehicle.

[0946] "Passenger satisfaction" is an indicator that shows the level of satisfaction felt by passengers using autonomous vehicles.

[0947] "Request response status" refers to the status of how appropriately passenger requests and demands are handled.

[0948] System Overview

[0949] This invention relates to a system for analyzing communication between passengers and the vehicle in an autonomous vehicle and providing feedback. This system includes a series of processes from acquiring voice data to evaluating it and generating feedback.

[0950] Acquiring and converting audio data

[0951] Server: To obtain voice data, collect voice data from microphones installed inside the autonomous vehicle. The collected voice data is converted into text data using a voice recognition system. For this voice recognition system, for example, the speech_recognition library is used.

[0952] Proofreading text data

[0953] Server: The converted text data is proofread to correct typos and grammatical errors. This proofreading process uses, for example, the TextBlob library.

[0954] Nuance tagging of statements

[0955] Server: Based on the proofread text data, tagging is performed to identify nuances that indicate the sentiment and intent of the speech. For tagging, the server uses the nltk library, for example, to classify the sentiment of words.

[0956] Analyzing data and providing feedback

[0957] Server: Analyzes the tagged text data and evaluates passenger satisfaction and request responsiveness. This evaluation includes determining the degree of achievement against specific criteria and objectives. Feedback generated based on the analysis results is provided to the operation manager. This feedback generation process allows for the improvement of the quality of service of the autonomous vehicle.

[0958] The specific hardware and software used

[0959] Hardware: Smartphones, microphones in self-driving vehicles

[0960] Software: Python program, speech_recognition library, TextBlob library, nltk library

[0961] Specific examples

[0962] For example, suppose a passenger reports a problem in an autonomous vehicle. The audio data is picked up by a microphone and converted into text data using a speech recognition system. This text data is then proofread using the TextBlob library and tagged with emotional nuances using the nltk library. Finally, the tagged data is analyzed to evaluate the passenger's level of dissatisfaction and how well the request was handled, and feedback is generated for the operation manager, such as "Passengers are very satisfied. We are providing good service."

[0963] Prompt Sentence Examples

[0964] "Analyze audio files recorded inside the autonomous vehicle to generate feedback that assesses passenger satisfaction and how well requests are handled."

[0965] In this way, by properly analyzing communication with passengers inside autonomous vehicles and providing feedback, operation managers can improve service quality and increase passenger satisfaction.

[0966] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0967] Step 1:

[0968] Acquiring voice data and converting it to text

[0969] The server acquires voice data from microphones installed in the autonomous vehicle. The acquired voice data is converted into text data using a voice recognition system. Specifically, the voice data is converted into character string information using the Python speech_recognition library. The input is an audio file, and the output is text data.

[0970] Step 2:

[0971] Proofreading text data

[0972] The server receives the converted text data as input and corrects typos and grammatical errors. This process uses the TextBlob library. Specifically, TextBlob is used to automatically correct grammatical errors in the text data. The input is the converted text data, and the output is the proofread text data.

[0973] Step 3:

[0974] Nuance tagging

[0975] The server receives the proofread text data and tags it with nuances that indicate the sentiment and intent of the utterance. This is done using the nltk library. Specifically, nltk is used to classify the sentiment of each word and assign tags such as positive, negative, and neutral. The input is the proofread text data, and the output is text data with nuance tags.

[0976] Step 4:

[0977] Data analysis

[0978] The server analyzes the tagged text data to evaluate passenger satisfaction and the response to their requests. This analysis determines the degree of achievement of specific criteria or objectives. Specifically, it calculates the number of positive, negative, and neutral tags and evaluates satisfaction based on that. The input is text data with nuanced tags, and the output is the analysis results.

[0979] Step 5:

[0980] Generating and Providing Feedback

[0981] The server generates feedback based on the analysis results and provides that feedback to the operation manager. Specifically, it generates specific suggestions and points to note for improving the service as text based on the analysis results. The input is the analysis results, and the output is the feedback text.

[0982] Specific examples of processing

[0983] 1. Voice data is acquired from the microphone saying, "The road is congested, please take a detour."

[0984] 2. The acquired voice data is converted into text data.

[0985] 3. The text data is proofread and typos are corrected, such as "The road is congested, please take a detour."

[0986] 4. The proofread text is tagged with nuances such as "The road is congested, so please take a detour [negative]."

[0987] 5. Analyze the tagged text data and evaluate the passenger dissatisfaction level.

[0988] 6. Based on the analysis results, feedback such as "Passengers are highly dissatisfied, so it is recommended that you consider proposing a detour route" is generated.

[0989] In this way, after going through each processing step, the autonomous vehicle operation manager can obtain specific feedback to improve the quality of service.

[0990] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0991] The present invention relates to a system that acquires audio data from an online conference, converts it into text data, proofreads it, tags it with nuances, analyzes it, and combines it with an emotion engine to recognize the user's emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[0992] System flow overview

[0993] System configuration

[0994] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[0995] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[0996] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text data.

[0997] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[0998] 5. Emotion engine: Recognizes the user's emotions from voice data and generates emotional information.

[0999] 6. Data analysis method: Analyze tagged text data and sentiment data to evaluate the achievement of the meeting objectives.

[1000] 7. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[1001] Program processing overview

[1002] Acquiring voice data and converting voice to text

[1003] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[1004] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[1005] Text Proofreading

[1006] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[1007] Nuance tagging of statements

[1008] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[1009] Use of emotion engine

[1010] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, speed, etc. of the voice to recognize the user's emotion. For example, it generates emotion information such as "relaxed," "tense," or "angry."

[1011] Data analysis and feedback

[1012] Server: Analyzes the nuance-tagged text data and the emotion data generated by the emotion engine. The analysis system evaluates the progress of the meeting and the degree to which the objectives were achieved, and generates more specific feedback taking into account the emotion information. The evaluation results are provided to the organizer as specific feedback.

[1013] Specific examples

[1014] Acquiring and converting audio data

[1015] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[1016] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[1017] Proofreading text and tagging nuances

[1018] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[1019] Use of emotion engine

[1020] Server: Sends the voice data to the emotion engine and analyzes it. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, it assigns emotional information in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[1021] Analyzing data and providing feedback

[1022] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1023] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[1024] The processing flow will be explained below.

[1025] Step 1:

[1026] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[1027] Step 2:

[1028] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[1029] Step 3:

[1030] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[1031] Step 4:

[1032] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[1033] Step 5:

[1034] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[1035] Step 6:

[1036] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions and generate emotion data. For example, emotional information such as "relaxed," "tense," or "angry" is generated.

[1037] Step 7:

[1038] Server: Sends the nuance-tagged text data and the emotion data generated by the emotion engine to the analysis system, which evaluates the progress of the meeting and the degree to which its objectives have been achieved based on the text data and emotion data.

[1039] Step 8:

[1040] Server: Generates feedback based on the evaluation results received from the analysis system, including an overall evaluation of the meeting and specific improvements.

[1041] Step 9:

[1042] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[1043] Step 10:

[1044] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[1045] The above is the specific processing flow of this system.

[1046] Example 2

[1047] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] With conventional online meeting systems, recording and reviewing meeting content is often done manually, making it difficult to evaluate the efficiency of the meeting and the degree to which the purpose was achieved.In addition, it is not possible to accurately grasp the nuances of what is being said or the emotions of the speaker, which leads to a problem of reduced quality of feedback.

[1049] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for recognizing a user's emotion from the voice data and generating emotion information, means for analyzing the tagged text data and emotion information and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to record online meetings, analyze nuances and emotions, evaluate the degree of goal achievement, and provide high-quality feedback.

[1050] "Audio data" refers to audio information stored in digital format, and is used to record the remarks of conference participants.

[1051] "Text data" refers to data obtained by converting voice data into character information, and expressing the content of a speech in a sentence format.

[1052] "Proofreading" is the process of detecting and correcting typographical and grammatical errors in text data.

[1053] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[1054] "Emotion information" is information generated by analyzing the user's emotions from voice data, and indicates an emotional state such as relaxation, tension, or anger.

[1055] "Evaluation of the degree of achievement of objectives" is the process of analyzing the progress and content of a meeting and evaluating the extent to which the set objectives were achieved.

[1056] "Providing feedback" refers to the task of generating improvements and overall evaluations of the conference based on the analysis results and conveying this information to the organizer.

[1057] The present invention is a system that converts voice data in online conferences into text data, then proofreads, nuance tags, and analyzes the text data, combines it with an emotion engine to recognize user emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[1058] System configuration

[1059] 1. How to acquire audio data: When users hold online meetings, they acquire audio data using the built-in recording function of their devices. The audio data is saved on the device in formats such as WAV or MP3.

[1060] 2. Audio file uploading method: After the conference ends, the user uploads the acquired audio data to the server. The device sends the audio file to the specified server's upload API.

[1061] 3. Speech recognition and text conversion: The server receives the uploaded audio files and converts them into text using a speech recognition system (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text). Analysis generates text data for each speaker.

[1062] 4. Text Proofreading: The server sends the generated text data to a proofreading system (e.g., Grammarly API or Microsoft Text Analytics API) to correct mistranslations and grammatical errors. The proofread text data is then sent back to the server.

[1063] 5. Nuance tagging: Based on the proofread text data, the server uses a nuance tagging system (e.g., a Natural Language Processing (NLP) model) to tag the utterance with a tag that indicates its sentiment or intent. For example, tags such as "positive," "negative," and "question" are applied.

[1064] 6. Emotion Engine: The server sends the voice data to an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer), which recognizes the user's emotion from the tone, pitch, and speed of the voice and generates emotion information. The emotion information includes emotional states such as relaxed, tense, and angry.

[1065] 7. Data analysis method: The server analyzes the nuance-tagged text data and emotional information using an analysis system (e.g., a Python analysis library) to evaluate the achievement of the meeting objectives.

[1066] 8. Feedback provision method: Based on the analysis results, the server generates feedback including an overall evaluation of the conference and suggestions for improvement, and provides it to the organizer. The feedback is sent to the organizer via email or the dashboard.

[1067] Specific examples

[1068] Acquiring and converting audio data

[1069] User: Start an online meeting, enable the recording function, and upload the recorded audio file to the server after the meeting ends.

[1070] Server: The server receives the audio file and converts it into text using a speech recognition system. This generates text such as "Tanaka-san (09:05:30): Let's discuss the progress of this project."

[1071] Proofreading text and tagging nuances

[1072] Server: Proofreads the generated text data, corrects grammatical errors and mistranslations, and then assigns nuances such as "positive" or "negative" to the corrected text.

[1073] Use of emotion engine

[1074] Server: The voice data is sent to the emotion engine and analyzed. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, emotional information is added in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[1075] Analyzing data and providing feedback

[1076] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1077] Prompt Sentence Examples

[1078] "I would like to develop a system that acquires audio data from online meetings, converts it to text data, proofreads it, tags it with nuances, uses an emotion engine to recognize users' emotions, and evaluates the degree to which the meeting achieved its objectives. Specifically, I would like to record the audio of the meeting, convert it to text using a speech recognition system, correct any errors in the text using a proofreading system, tag the text with nuances and emotions, and generate emotion data using an emotion engine. Finally, I would like a system that evaluates the meeting using a data analysis system and provides feedback."

[1079] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1080] Step 1: Acquire audio data

[1081] User: When starting an online meeting, enable the recording function on the device. After the meeting ends, the audio data will be saved on the device.

[1082] Input: Audio data during the meeting

[1083] Output: Recorded audio file (WAV or MP3 format)

[1084] How it works: The device collects audio during the meeting through the microphone and records it in real time. When the meeting ends, the audio file is saved on the device.

[1085] Step 2: Upload your audio file

[1086] User: After the meeting ends, upload the recorded audio file to the server.

[1087] Input: Saved audio file

[1088] Output: Audio file uploaded to the server

[1089] Specific operation: The user selects an audio file on the device and uploads it to the server. The audio file is then transferred to the specified server using the upload API.

[1090] Step 3: Speech recognition and text conversion

[1091] Server: After receiving the audio file, convert the audio data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API).

[1092] Input: Audio file uploaded to the server

[1093] Output: Text data converted from audio

[1094] Specific operation: The server sends the audio file to a speech recognition system, which analyzes the audio and generates text data that includes the speaker's name, the time of speech, and the content of the speech.

[1095] Step 4: Proofread the text data

[1096] Server: Sends the generated text data to a proofreading system (e.g., Grammarly API) to correct mistranslations and grammatical errors.

[1097] Input: Text data generated by a speech recognition system

[1098] Output: Proofread text data

[1099] Specific operation: The server sends the text data to the proofreading system, which detects and automatically corrects typos and grammatical errors. The corrected text data is then returned to the server.

[1100] Step 5: Nuance tagging

[1101] Server: Based on the proofread text data, a nuance tagging system (NLP model) is used to assign tags that indicate the sentiment and intent of the utterances.

[1102] Input: Proofread text data

[1103] Output: Text data with nuance tags

[1104] How it works: The server passes the proofread text data to an NLP model, which generates tags such as "positive" or "negative" based on the content of the speech and assigns them to the text.

[1105] Step 6: Emotion Recognition with the Emotion Engine

[1106] Server: Sends voice data to an emotion engine (e.g., Affectiva), which recognizes the user's emotions from the tone, pitch, speed, etc. of the voice and generates emotion information.

[1107] Input: Audio data

[1108] Output: Data containing emotional information

[1109] Specific operation: The server sends voice data to the emotion engine, which analyzes the voice characteristics to recognize the user's emotional state (relaxed, nervous, angry, etc.) and generates information about the user's emotional state. The generated emotion information is returned to the server.

[1110] Step 7: Data analysis

[1111] Server: Analyzes the nuance-tagged text data and emotional information using an analysis system (data analysis platform) to evaluate the degree to which the meeting's objectives were achieved.

[1112] Input: Nuance-tagged text data, emotion information

[1113] Output: Analysis results

[1114] Specific operation: The server passes the text data and emotion data to the analysis system, which evaluates the frequency of comments, emotional fluctuations, progress, etc. The analysis then calculates the overall achievement of the meeting's objectives.

[1115] Step 8: Generate and provide feedback

[1116] Server: Based on the analysis results, it generates feedback including an overall evaluation of the meeting and suggestions for improvement, and provides it to the organizer.

[1117] Input: Analysis results

[1118] Output: Feedback message

[1119] Specific operation: The server compiles the analysis results and generates feedback such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the following improvements would make the meeting better: - Strengthen time management - Balance speeches." This feedback is provided to the organizer via email or dashboard.

[1120] (Application example 2)

[1121] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1122] In conventional online meetings and instruction sharing meetings at logistics centers, even though it is possible to capture and convert audio data into text, there is a lack of means to effectively recognize the nuances of what is being said and the emotions of participants, and to properly analyze and evaluate the progress of the meeting and the degree to which its objectives have been achieved. This makes it difficult to understand the overall content of the meeting or identify areas for improvement, which can lead to a decline in the efficiency and quality of the meeting.

[1123] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for generating emotion information from the tagged text data and voice data, means for analyzing the emotion information and tagged text data and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to evaluate the degree of goal achievement of the meeting and provide feedback.

[1124] "Audio data" refers to audio information from meetings or communications recorded in digital format.

[1125] "Text data" is character information obtained by analyzing voice data.

[1126] "Proofreading" is the act of correcting mistranslations and grammatical errors in converted text data.

[1127] "Nuance tagging" refers to assigning labels that indicate the sentiment or intent of a statement based on text data.

[1128] "Emotion information" is data that indicates the user's emotion recognized from voice data or text data.

[1129] "Analysis" is the process of using tagged text data and sentiment information to evaluate the progress of the meeting and the degree to which its objectives were achieved.

[1130] "Evaluation" means analyzing the degree to which the meeting's objectives were achieved and presenting the results quantitatively or qualitatively.

[1131] "Feedback" is information about improvements and progress of the meeting that is generated based on the evaluation results.

[1132] A "system" is a combination of devices and software that executes a series of processes, from acquiring audio data to providing feedback.

[1133] A system for implementing this invention performs a series of processes including acquiring voice data, converting the voice data into text, proofreading the text data, tagging nuances, generating emotional information, analyzing the data, and generating feedback.

[1134] Hardware and software used

[1135] Hardware: Smartphone for primary processing, server for final data processing

[1136] software:

[1137] Speech recognition: Google Cloud Speech-to-Text API

[1138] Emotion recognition: IBM Watson Tone Analyzer

[1139] Data Analysis: Python + Pandas

[1140] Feedback Generation: OpenAI GPT-3

[1141] Processing flow

[1142] Step 1: Acquire audio data

[1143] The server uses smartphones to record audio from meetings at the logistics center, and the recorded audio data is stored in digital format.

[1144] Step 2: Speech to text conversion

[1145] The server receives the recorded audio data and converts it into text using the Google Cloud Speech-to-Text API, which also includes speaker information and speaking time information.

[1146] Step 3: Proofread the text data

[1147] The server proofreads the generated text data, using an automatic proofreading engine to correct grammatical errors and mistranslations.

[1148] Step 4: Nuance tagging

[1149] Based on the proofread text data, the server adds tags that indicate the emotion and intention of the statement as nuances.

[1150] Step 5: Generating emotional information

[1151] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information.

[1152] Step 6: Data analysis

[1153] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree to which the goal was achieved. The evaluation is performed using Python and Pandas.

[1154] Step 7: Generate feedback

[1155] The server generates feedback based on the evaluation results and provides it to the organizer. The feedback is generated using OpenAI's GPT-3 model.

[1156] Specific examples

[1157] At a meeting held at a logistics center, the organizer records the audio of the meeting using a smartphone. After recording is complete, the audio data is uploaded to a server. The server then converts the audio to text using the Google Cloud Speech-to-Text API, correcting grammatical errors and mistranslations using an automatic proofreading engine. Nuances are tagged in the corrected text data, and emotional information about participants is generated using IBM Watson Tone Analyzer. The generated text data and emotional information are analyzed using Python and Pandas, and the degree to which the meeting's objectives were achieved is evaluated. Finally, feedback is generated using OpenAI GPT-3 based on the evaluation results and provided to the organizer.

[1158] Prompt Sentence Examples

[1159] The meeting achieved a goal of 0.75. Please give us your suggestions for improvement.

[1160] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1161] Step 1:

[1162] A user uses a smartphone to record the audio of a meeting at a logistics center. The audio data is stored digitally on the smartphone. When the recording is complete, the user uploads the audio data to a server.

[1163] Step 2:

[1164] The server receives the uploaded audio data and converts it into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data containing speaker information and speech time information. The server sends the audio data to the API and generates corresponding text for each utterance.

[1165] Step 3:

[1166] The server sends the generated text data to an automatic proofreading engine, which corrects any mistranslations or grammatical errors. The input is the generated text data, and the output is the corrected text data. The automatic proofreading engine analyzes the text and makes appropriate corrections.

[1167] Step 4:

[1168] The server performs nuance tagging based on the corrected text data. The input is the corrected text data, and the output is the nuance-tagged text data. The server analyzes the text data and assigns tags based on its sentiment and intent.

[1169] Step 5:

[1170] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information. The input is voice data, and the output is emotional information. The server analyzes the voice data, and the emotional engine generates the emotional information.

[1171] Step 6:

[1172] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree of goal achievement. The input is the tagged text data and emotion information, and the output is the evaluation result of the degree of goal achievement of the meeting. The server analyzes the data using Python and Pandas and performs a quantitative evaluation.

[1173] Step 7:

[1174] The server generates feedback based on the evaluation results and provides it to the organizer. The input is the evaluation result of the meeting's goal achievement, and the output is feedback including areas for improvement. The feedback is generated using OpenAI's GPT-3 model, which generates prompts based on the evaluation results and creates feedback.

[1175] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1176] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1177] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1178] [Fourth embodiment]

[1179] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1180] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1181] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1182] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1183] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1184] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1185] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1186] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1187] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1188] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1189] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1190] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1191] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1192] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[1193] System flow overview

[1194] System configuration

[1195] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[1196] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[1197] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text.

[1198] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[1199] 5. Data analysis method: Analyze the tagged text data and evaluate the achievement of the meeting objectives.

[1200] 6. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[1201] Program processing overview

[1202] Acquiring voice data and converting voice to text

[1203] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[1204] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[1205] Text Proofreading

[1206] Server: Proofread the generated text data to ensure there are no mistranslations or grammatical errors. This proofreading process is very important to ensure the accuracy of the text information. After proofreading, the text is easier to read and the content is more accurate.

[1207] Nuance tagging of statements

[1208] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[1209] Data analysis and feedback

[1210] Server: Analyzes the tagged text data and evaluates the achievement of the meeting's objectives. This evaluation includes the progress of the meeting, balance of comments, time management, etc. The evaluation results are generated as specific feedback and provided to the organizer.

[1211] Specific examples

[1212] Acquiring and converting audio data

[1213] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[1214] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[1215] Proofreading text and tagging nuances

[1216] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[1217] Analyzing data and providing feedback

[1218] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1219] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[1220] The processing flow will be explained below.

[1221] Step 1:

[1222] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[1223] Step 2:

[1224] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[1225] Step 3:

[1226] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[1227] Step 4:

[1228] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[1229] Step 5:

[1230] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[1231] Step 6:

[1232] Server: Sends the nuance-tagged text data to the analysis system, which evaluates the progress of the meeting and the degree to which the objectives were achieved. Evaluation points include time management and balance of speech.

[1233] Step 7:

[1234] Server: Generates feedback based on the evaluation results received from the analysis system. This feedback provides an overall assessment of the meeting and specific areas for improvement.

[1235] Step 8:

[1236] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[1237] Step 9:

[1238] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[1239] The above is the specific processing flow of this system.

[1240] Example 1

[1241] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1242] Current online meeting systems lack a means to objectively evaluate the progress of meetings and the degree to which their objectives were achieved. This makes it difficult for organizers to obtain specific feedback to evaluate and improve the results of meetings. Furthermore, automated processes for accurately recording what is said during meetings and understanding the nuances of what is said are also uncommon. This makes it difficult to improve meeting efficiency.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1244] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the proofread text data, means for analyzing the tagged text data and evaluating the degree of achievement of the conference goal, and means for generating feedback based on the evaluation and providing it to the organizer. This makes it possible to objectively evaluate the progress of the conference and the degree of achievement of the goal, and to provide specific feedback to the organizer.

[1245] "Audio Data" refers to audio information recorded during a conference.

[1246] "Text data" refers to data in which voice data is converted into text information.

[1247] "Proofreading" refers to correcting mistranslations and grammatical errors in converted text data.

[1248] "Nuance tagging" refers to adding tags that indicate the emotion or intention of a statement based on text data.

[1249] "Analysis" refers to analyzing data based on tagged text data and evaluating the degree to which the meeting's objectives were achieved.

[1250] "Feedback" refers to evaluations and improvement suggestions generated based on the analysis results and provided to the meeting organizer.

[1251] The present invention relates to a system for analyzing audio data from an online conference, evaluating the achievement of the conference objectives, and providing feedback to the organizer. The system includes steps for converting the audio data into text, proofreading the text, adding nuance tags to the speech content, and analyzing the data to generate feedback.

[1252] System flow overview

[1253] Step 1: Acquire audio data

[1254] User: When the conference host starts the conference, the user enables the recording function on the device and records the audio data until the end of the conference. After the conference ends, the user uploads the recorded audio file to the server. Uploading is done using a dedicated uploader application or a browser.

[1255] Step 2: Speech to text conversion

[1256] Server: After receiving the uploaded audio files, the server uses a speech recognition system to convert the audio data into text data. Google Cloud Speech-to-Text API is commonly used as the speech recognition system. This system analyzes the received audio data and generates text data for each speaker. For example, the format might be "Tanaka-san (09:00:00): Thank you for your time today."

[1257] Step 3: Proofread your text

[1258] Server: Use the Content Moderation API to proofread the generated text data. This process corrects text translation errors and grammatical errors. After proofreading, the text data becomes more accurate and easier to read. It is recommended to use the Grammarly API as the Content Moderation API.

[1259] Step 4: Nuance tagging

[1260] Server: The proofread text data is sent to a sentiment analysis engine, which tags the nuances of the utterances. This analysis generates tags that indicate the sentiment and intent of the utterances. IBM Watson Tone Analyzer is commonly used as the sentiment analysis engine. For example, the format is something like "Tanaka-san (09:00:00): Thank you for your time today. [Positive]."

[1261] Step 5: Data analysis

[1262] Server: To analyze the tagged text data, data analysis libraries such as Python's Pandas and NumPy are used. This allows for evaluation of the progress of the meeting, balance of speeches, time management, etc. As a result of the analysis, the degree to which the meeting's objectives were achieved is scored.

[1263] Step 6: Provide feedback

[1264] Server: The server generates specific feedback based on the analysis results and provides it to the organizer. The feedback includes an overall evaluation of the conference and suggestions for improvement, making it useful for the organizer. The feedback is provided in the form of a report, which the organizer can receive by email or on the platform.

[1265] Specific examples

[1266] Acquiring and converting audio data

[1267] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[1268] Server: The server receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[1269] Proofreading text and tagging nuances

[1270] Server: Proofreads the generated text data and corrects typos. Then, a sentiment analysis engine is used to tag the corrected text with nuances such as "positive" or "negative." For example, a tag is added like this: "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Positive]."

[1271] Analyzing data and providing feedback

[1272] Server: Analyzes the completed text data and evaluates the progress of the meeting and the degree to which the objectives were achieved. Based on the results of this evaluation, the server provides feedback to the organizer, such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1273] Prompt Sentence Examples

[1274] Below are examples of specific prompts used at each step:

[1275] Voice data conversion prompt

[1276] Example of a prompt to convert voice data to text:

[1277] Audio Data: <Recorded audio file>

[1278] Settings: Encoding = "LINEAR16", Sample rate = 16000, Language code = "ja-JP"

[1279] Proofreading prompt

[1280] Example of a proofreading prompt:

[1281] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[1282] Nuanced tagging prompts

[1283] Nuance tagging prompt example:

[1284] Text data: Tanaka-san (09:05:30): Let's discuss the progress of this project.

[1285] Model: Sentiment Analysis Model 2023

[1286] The above is an embodiment of the present invention, which can improve the efficiency of online meetings and provide useful feedback to the organizer.

[1287] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1288] Step 1: Acquire audio data

[1289] User: When the meeting host starts a meeting, the recording function of the device is enabled. The recording function uses the device's built-in microphone or an external microphone to record the meeting audio in real time. After the meeting ends, the recorded audio file is uploaded to the server.

[1290] Input: Audio recorded during the meeting.

[1291] Output: Audio file uploaded to the server.

[1292] Specific operation: After the meeting ends, the user clicks the "upload" button on the online platform to upload the recorded file to the server.

[1293] Step 2: Speech to text conversion

[1294] Server: The server receives the uploaded audio files. Once the files are received, it calls a speech recognition system (e.g., Google Cloud Speech-to-Text API) to convert the audio files into text data. The speech recognition system analyzes the audio data and generates text data for each speaker.

[1295] Input: An audio file.

[1296] Output: Text data containing utterances by time.

[1297] Specific operation: The server sends the audio data to the speech recognition API and receives the converted text data, which includes a timestamp and the speaker's name.

[1298] Step 3: Proofread your text

[1299] Server: Use a Content Moderation API (e.g., Grammarly API) to proofread the generated text data. The proofreading process corrects spelling errors and grammatical errors.

[1300] Input: The converted text data.

[1301] Output: Text data with mistranslations and grammatical errors corrected.

[1302] What happens: The server sends the text data to the proofreading API and receives the corrected text along with the plain text. The corrected items include typos and grammatical errors.

[1303] Step 4: Nuance tagging

[1304] Server: Analyzes the nuances of the speech based on the proofread text data. This analysis is performed using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) and tags are added to indicate the sentiment and intent of the speech.

[1305] Input: Proofread text data.

[1306] Output: Nuance-tagged text data.

[1307] How it works: The server sends the proofread text data to the sentiment analysis engine and receives sentiment tags such as positive, negative, or neutral.

[1308] Step 5: Data analysis

[1309] Server: Analyzes the tagged text data using a data analysis system (e.g., Python's Pandas, NumPy, and machine learning algorithms) and evaluates the achievement of the meeting's objectives, including the progress of the meeting, balance of speeches, and time management.

[1310] Input: Tagged text data.

[1311] Output: Score and feedback data to evaluate the achievement of the meeting objectives.

[1312] Specific operation: The server inputs the tagged text data into a data analysis algorithm, evaluates the frequency of comments and time allocation, and outputs a score indicating the degree to which the meeting's objectives were achieved.

[1313] Step 6: Provide feedback

[1314] Server: The server generates feedback based on the analysis results and provides it to the organizer. This feedback includes an overall evaluation of the conference and suggestions for improvement, and is provided in the form of a report.

[1315] User: The user receives the feedback provided by the server and uses it as a reference for the next meeting.

[1316] Input: Analysis results.

[1317] Output: Feedback report provided to organizer.

[1318] Specific operation: The server compiles the analysis results into a feedback report and sends it to the organizer by email. Users can also check the report on the platform.

[1319] (Application example 1)

[1320] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1321] Autonomous vehicles require accurate understanding of passenger satisfaction and the status of requests, and provide specific feedback to improve service quality. However, currently, there is a lack of effective means to analyze communication between passengers and vehicles and generate feedback. It is necessary to solve this problem and improve passenger satisfaction.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1323] In this invention, the server includes a means for acquiring voice data, a means for converting the voice data into text data, and a means for proofreading the text data. This makes it possible to accurately grasp the content of communication between passengers and autonomous vehicles. The server also includes a means for tagging the nuances of speech based on the text data, a means for analyzing the tagged text data to evaluate passenger satisfaction and response to requests in the autonomous vehicle, and a means for generating and providing feedback based on the evaluation. This makes it possible to improve the service quality of autonomous vehicles, increase passenger satisfaction, and provide a comfortable riding experience.

[1324] "Voice data" refers to data that records audio information, such as conversations and sounds with passengers inside an autonomous vehicle.

[1325] "Text data" is audio data converted into text information, and is data that records the content in an abstract manner.

[1326] "Proofreading" is the process of correcting typographical and grammatical errors in text data to make it accurate and easy to read.

[1327] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[1328] "Analysis" refers to the process of evaluating tagged text data and analyzing passenger satisfaction and requests within the autonomous vehicle.

[1329] "Evaluation" is the act of determining the degree of achievement of specific standards or objectives based on the results of analysis.

[1330] "Feedback" refers to information such as specific comments and suggestions that are generated based on the evaluation results.

[1331] "Service Quality" refers to the overall quality and performance of the service provided by an automated vehicle.

[1332] "Passenger satisfaction" is an indicator that shows the level of satisfaction felt by passengers using autonomous vehicles.

[1333] "Request response status" refers to the status of how appropriately passenger requests and demands are handled.

[1334] System Overview

[1335] This invention relates to a system for analyzing communication between passengers and the vehicle in an autonomous vehicle and providing feedback. This system includes a series of processes from acquiring voice data to evaluating it and generating feedback.

[1336] Acquiring and converting audio data

[1337] Server: To obtain voice data, collect voice data from microphones installed inside the autonomous vehicle. The collected voice data is converted into text data using a voice recognition system. For this voice recognition system, for example, the speech_recognition library is used.

[1338] Proofreading text data

[1339] Server: The converted text data is proofread to correct typos and grammatical errors. This proofreading process uses, for example, the TextBlob library.

[1340] Nuance tagging of statements

[1341] Server: Based on the proofread text data, tagging is performed to identify nuances that indicate the sentiment and intent of the speech. For tagging, the server uses the nltk library, for example, to classify the sentiment of words.

[1342] Analyzing data and providing feedback

[1343] Server: Analyzes the tagged text data and evaluates passenger satisfaction and request responsiveness. This evaluation includes determining the degree of achievement against specific criteria and objectives. Feedback generated based on the analysis results is provided to the operation manager. This feedback generation process allows for the improvement of the quality of service of the autonomous vehicle.

[1344] The specific hardware and software used

[1345] Hardware: Smartphones, microphones in self-driving vehicles

[1346] Software: Python program, speech_recognition library, TextBlob library, nltk library

[1347] Specific examples

[1348] For example, suppose a passenger reports a problem in an autonomous vehicle. The audio data is picked up by a microphone and converted into text data using a speech recognition system. This text data is then proofread using the TextBlob library and tagged with emotional nuances using the nltk library. Finally, the tagged data is analyzed to evaluate the passenger's level of dissatisfaction and how well the request was handled, and feedback is generated for the operation manager, such as "Passengers are very satisfied. We are providing good service."

[1349] Prompt Sentence Examples

[1350] "Analyze audio files recorded inside the autonomous vehicle to generate feedback that assesses passenger satisfaction and how well requests are handled."

[1351] In this way, by properly analyzing communication with passengers inside autonomous vehicles and providing feedback, operation managers can improve service quality and increase passenger satisfaction.

[1352] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1353] Step 1:

[1354] Acquiring voice data and converting it to text

[1355] The server acquires voice data from microphones installed in the autonomous vehicle. The acquired voice data is converted into text data using a voice recognition system. Specifically, the voice data is converted into character string information using the Python speech_recognition library. The input is an audio file, and the output is text data.

[1356] Step 2:

[1357] Proofreading text data

[1358] The server receives the converted text data as input and corrects typos and grammatical errors. This process uses the TextBlob library. Specifically, TextBlob is used to automatically correct grammatical errors in the text data. The input is the converted text data, and the output is the proofread text data.

[1359] Step 3:

[1360] Nuance tagging

[1361] The server receives the proofread text data and tags it with nuances that indicate the sentiment and intent of the utterance. This is done using the nltk library. Specifically, nltk is used to classify the sentiment of each word and assign tags such as positive, negative, and neutral. The input is the proofread text data, and the output is text data with nuance tags.

[1362] Step 4:

[1363] Data analysis

[1364] The server analyzes the tagged text data to evaluate passenger satisfaction and the response to their requests. This analysis determines the degree of achievement of specific criteria or objectives. Specifically, it calculates the number of positive, negative, and neutral tags and evaluates satisfaction based on that. The input is text data with nuanced tags, and the output is the analysis results.

[1365] Step 5:

[1366] Generating and Providing Feedback

[1367] The server generates feedback based on the analysis results and provides that feedback to the operation manager. Specifically, it generates specific suggestions and points to note for improving the service as text based on the analysis results. The input is the analysis results, and the output is the feedback text.

[1368] Specific examples of processing

[1369] 1. Voice data is acquired from the microphone saying, "The road is congested, please take a detour."

[1370] 2. The acquired voice data is converted into text data.

[1371] 3. The text data is proofread and typos are corrected, such as "The road is congested, please take a detour."

[1372] 4. The proofread text is tagged with nuances such as "The road is congested, so please take a detour [negative]."

[1373] 5. Analyze the tagged text data and evaluate the passenger dissatisfaction level.

[1374] 6. Based on the analysis results, feedback such as "Passengers are highly dissatisfied, so it is recommended that you consider proposing a detour route" is generated.

[1375] In this way, after going through each processing step, the autonomous vehicle operation manager can obtain specific feedback to improve the quality of service.

[1376] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1377] The present invention relates to a system that acquires audio data from an online conference, converts it into text data, proofreads it, tags it with nuances, analyzes it, and combines it with an emotion engine to recognize the user's emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[1378] System flow overview

[1379] System configuration

[1380] 1. Means for acquiring audio data: Equipped with a recording function for recording audio data during a meeting.

[1381] 2. Voice to text conversion means: A voice recognition system is used to convert the acquired voice data into text information.

[1382] 3. Text proofreading: Correcting mistranslations and grammatical errors in the converted text data.

[1383] 4. Nuance tagging: Tagging nuances that indicate the sentiment and intent of a statement based on the corrected text.

[1384] 5. Emotion engine: Recognizes the user's emotions from voice data and generates emotional information.

[1385] 6. Data analysis method: Analyze tagged text data and sentiment data to evaluate the achievement of the meeting objectives.

[1386] 7. Feedback provision method: Feedback will be generated based on the analysis results and provided to the organizer.

[1387] Program processing overview

[1388] Acquiring voice data and converting voice to text

[1389] User: The conference organizer records the audio data on their device and uploads it to the server after the conference ends. This allows all audio from the conference to be recorded.

[1390] Server: After receiving the uploaded audio files, it starts processing them to convert them into text using a speech recognition system. This speech recognition system analyzes the received audio data and generates text data for each speaker. For example, a text containing the speaker's name, speaking time, and spoken content is generated, such as "Mr. Sato (09:00:00): Good morning, let's start the meeting."

[1391] Text Proofreading

[1392] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[1393] Nuance tagging of statements

[1394] Server: Analyzes the nuances of statements based on the proofread text data. This analysis is the process of assigning tags that indicate the sentiment and intention of the statement. For example, a tag such as "positive" is assigned to a specific statement, such as "Mr. Sato (09:00:00): Good morning. Let's start the meeting. [Positive]."

[1395] Use of emotion engine

[1396] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, speed, etc. of the voice to recognize the user's emotion. For example, it generates emotion information such as "relaxed," "tense," or "angry."

[1397] Data analysis and feedback

[1398] Server: Analyzes the nuance-tagged text data and the emotion data generated by the emotion engine. The analysis system evaluates the progress of the meeting and the degree to which the objectives were achieved, and generates more specific feedback taking into account the emotion information. The evaluation results are provided to the organizer as specific feedback.

[1399] Specific examples

[1400] Acquiring and converting audio data

[1401] User: A meeting host starts an online meeting, enables the recording function during the meeting, and uploads the recorded audio file to the server after the meeting ends.

[1402] Server: Receives the audio file and converts it into text data using a speech recognition system. Text data such as "Tanaka-san (09:05:30): Let's discuss the progress of this project." is generated.

[1403] Proofreading text and tagging nuances

[1404] Server: Proofreads the generated text data and corrects typos. Then, the corrected text is given nuances such as "positive" or "negative."

[1405] Use of emotion engine

[1406] Server: Sends the voice data to the emotion engine and analyzes it. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, it assigns emotional information in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[1407] Analyzing data and providing feedback

[1408] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1409] The above is an embodiment of the present invention, and is a system that can improve the efficiency of online meetings and provide useful feedback to the organizer.

[1410] The processing flow will be explained below.

[1411] Step 1:

[1412] User: Starts a meeting and enables the recording function on the device. Records audio data throughout the meeting. After the meeting ends, uploads the recorded audio file to the server.

[1413] Step 2:

[1414] Server: Receives the uploaded audio file. Checks the file format and quality. If the file is not suitable, generates an error message to notify the user. If the format is suitable, proceeds to the next step.

[1415] Step 3:

[1416] Server: The voice data is sent to the speech recognition system. This system converts the voice data into text and generates text information. For example, text data in the form of "Mr. Sato (09:00:00): Good morning, let's start the meeting." is generated.

[1417] Step 4:

[1418] Server: The generated text data is sent to the proofreading system for proofreading. The proofreading system automatically detects and corrects grammatical errors and mistranslations. The corrected text data is then returned to the server.

[1419] Step 5:

[1420] Server: The proofread text data is sent to the nuance analysis system, which analyzes the sentiment and intent of the utterance and assigns appropriate nuance tags. For example, tags such as "positive" or "negative" are assigned to the utterance.

[1421] Step 6:

[1422] Server: Sends voice data to the emotion engine. The emotion engine analyzes the tone, pitch, and speed of the voice to recognize the user's emotions and generate emotion data. For example, emotional information such as "relaxed," "tense," or "angry" is generated.

[1423] Step 7:

[1424] Server: Sends the nuance-tagged text data and the emotion data generated by the emotion engine to the analysis system, which evaluates the progress of the meeting and the degree to which its objectives have been achieved based on the text data and emotion data.

[1425] Step 8:

[1426] Server: Generates feedback based on the evaluation results received from the analysis system, including an overall evaluation of the meeting and specific improvements.

[1427] Step 9:

[1428] Server: Provides the generated feedback to the organizer, mainly via email or display on a dedicated dashboard.

[1429] Step 10:

[1430] User: Checks the feedback provided by the server and uses it to manage the next meeting. This feedback is expected to improve the quality of the meeting.

[1431] The above is the specific processing flow of this system.

[1432] Example 2

[1433] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1434] With conventional online meeting systems, recording and reviewing meeting content is often done manually, making it difficult to evaluate the efficiency of the meeting and the degree to which the purpose was achieved.In addition, it is not possible to accurately grasp the nuances of what is being said or the emotions of the speaker, which leads to a problem of reduced quality of feedback.

[1435] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for recognizing a user's emotion from the voice data and generating emotion information, means for analyzing the tagged text data and emotion information and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to record online meetings, analyze nuances and emotions, evaluate the degree of goal achievement, and provide high-quality feedback.

[1436] "Audio data" refers to audio information stored in digital format, and is used to record the remarks of conference participants.

[1437] "Text data" refers to data obtained by converting voice data into character information, and expressing the content of a speech in a sentence format.

[1438] "Proofreading" is the process of detecting and correcting typographical and grammatical errors in text data.

[1439] "Nuance tagging" is the process of assigning tags that indicate the emotion or intention of a statement based on text data.

[1440] "Emotion information" is information generated by analyzing the user's emotions from voice data, and indicates an emotional state such as relaxation, tension, or anger.

[1441] "Evaluation of the degree of achievement of objectives" is the process of analyzing the progress and content of a meeting and evaluating the extent to which the set objectives were achieved.

[1442] "Providing feedback" refers to the task of generating improvements and overall evaluations of the conference based on the analysis results and conveying this information to the organizer.

[1443] The present invention is a system that converts voice data in online conferences into text data, then proofreads, nuance tags, and analyzes the text data, combines it with an emotion engine to recognize user emotions, evaluates the degree to which the conference's objectives have been achieved, and provides feedback to the organizer.

[1444] System configuration

[1445] 1. How to acquire audio data: When users hold online meetings, they acquire audio data using the built-in recording function of their devices. The audio data is saved on the device in formats such as WAV or MP3.

[1446] 2. Audio file uploading method: After the conference ends, the user uploads the acquired audio data to the server. The device sends the audio file to the specified server's upload API.

[1447] 3. Speech recognition and text conversion: The server receives the uploaded audio files and converts them into text using a speech recognition system (e.g., Google Cloud Speech-to-Text API or IBM Watson Speech to Text). Analysis generates text data for each speaker.

[1448] 4. Text Proofreading: The server sends the generated text data to a proofreading system (e.g., Grammarly API or Microsoft Text Analytics API) to correct mistranslations and grammatical errors. The proofread text data is then sent back to the server.

[1449] 5. Nuance tagging: Based on the proofread text data, the server uses a nuance tagging system (e.g., a Natural Language Processing (NLP) model) to tag the utterance with a tag that indicates its sentiment or intent. For example, tags such as "positive," "negative," and "question" are applied.

[1450] 6. Emotion Engine: The server sends the voice data to an emotion engine (e.g., Affectiva or IBM Watson Tone Analyzer), which recognizes the user's emotion from the tone, pitch, and speed of the voice and generates emotion information. The emotion information includes emotional states such as relaxed, tense, and angry.

[1451] 7. Data analysis method: The server analyzes the nuance-tagged text data and emotional information using an analysis system (e.g., a Python analysis library) to evaluate the achievement of the meeting objectives.

[1452] 8. Feedback provision method: Based on the analysis results, the server generates feedback including an overall evaluation of the conference and suggestions for improvement, and provides it to the organizer. The feedback is sent to the organizer via email or the dashboard.

[1453] Specific examples

[1454] Acquiring and converting audio data

[1455] User: Start an online meeting, enable the recording function, and upload the recorded audio file to the server after the meeting ends.

[1456] Server: The server receives the audio file and converts it into text using a speech recognition system. This generates text such as "Tanaka-san (09:05:30): Let's discuss the progress of this project."

[1457] Proofreading text and tagging nuances

[1458] Server: Proofreads the generated text data, corrects grammatical errors and mistranslations, and then assigns nuances such as "positive" or "negative" to the corrected text.

[1459] Use of emotion engine

[1460] Server: The voice data is sent to the emotion engine and analyzed. The emotion engine recognizes emotions such as "relaxed," "tense," and "anger" from the voice data. For example, emotional information is added in the form of "Tanaka-san (09:05:30): Let's discuss the progress of this project. [Tense]."

[1461] Analyzing data and providing feedback

[1462] Server: Analyzes the completed text data and emotion data to evaluate the progress of the meeting and the degree to which the objectives were achieved. Based on the evaluation results, the server provides feedback to the organizer, such as "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the meeting could be improved by improving the following points: - Strengthen time management - Balance speech."

[1463] Prompt Sentence Examples

[1464] "I would like to develop a system that acquires audio data from online meetings, converts it to text data, proofreads it, tags it with nuances, uses an emotion engine to recognize users' emotions, and evaluates the degree to which the meeting achieved its objectives. Specifically, I would like to record the audio of the meeting, convert it to text using a speech recognition system, correct any errors in the text using a proofreading system, tag the text with nuances and emotions, and generate emotion data using an emotion engine. Finally, I would like a system that evaluates the meeting using a data analysis system and provides feedback."

[1465] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1466] Step 1: Acquire audio data

[1467] User: When starting an online meeting, enable the recording function on the device. After the meeting ends, the audio data will be saved on the device.

[1468] Input: Audio data during the meeting

[1469] Output: Recorded audio file (WAV or MP3 format)

[1470] How it works: The device collects audio during the meeting through the microphone and records it in real time. When the meeting ends, the audio file is saved on the device.

[1471] Step 2: Upload your audio file

[1472] User: After the meeting ends, upload the recorded audio file to the server.

[1473] Input: Saved audio file

[1474] Output: Audio file uploaded to the server

[1475] Specific operation: The user selects an audio file on the device and uploads it to the server. The audio file is then transferred to the specified server using the upload API.

[1476] Step 3: Speech recognition and text conversion

[1477] Server: After receiving the audio file, convert the audio data into text data using a speech recognition system (e.g., Google Cloud Speech-to-Text API).

[1478] Input: Audio file uploaded to the server

[1479] Output: Text data converted from audio

[1480] Specific operation: The server sends the audio file to a speech recognition system, which analyzes the audio and generates text data that includes the speaker's name, the time of speech, and the content of the speech.

[1481] Step 4: Proofread the text data

[1482] Server: Sends the generated text data to a proofreading system (e.g., Grammarly API) to correct mistranslations and grammatical errors.

[1483] Input: Text data generated by a speech recognition system

[1484] Output: Proofread text data

[1485] Specific operation: The server sends the text data to the proofreading system, which detects and automatically corrects typos and grammatical errors. The corrected text data is then returned to the server.

[1486] Step 5: Nuance tagging

[1487] Server: Based on the proofread text data, a nuance tagging system (NLP model) is used to assign tags that indicate the sentiment and intent of the utterances.

[1488] Input: Proofread text data

[1489] Output: Text data with nuance tags

[1490] How it works: The server passes the proofread text data to an NLP model, which generates tags such as "positive" or "negative" based on the content of the speech and assigns them to the text.

[1491] Step 6: Emotion Recognition with the Emotion Engine

[1492] Server: Sends voice data to an emotion engine (e.g., Affectiva), which recognizes the user's emotions from the tone, pitch, speed, etc. of the voice and generates emotion information.

[1493] Input: Audio data

[1494] Output: Data containing emotional information

[1495] Specific operation: The server sends voice data to the emotion engine, which analyzes the voice characteristics to recognize the user's emotional state (relaxed, nervous, angry, etc.) and generates information about the user's emotional state. The generated emotion information is returned to the server.

[1496] Step 7: Data analysis

[1497] Server: Analyzes the nuance-tagged text data and emotional information using an analysis system (data analysis platform) to evaluate the degree to which the meeting's objectives were achieved.

[1498] Input: Nuance-tagged text data, emotion information

[1499] Output: Analysis results

[1500] Specific operation: The server passes the text data and emotion data to the analysis system, which evaluates the frequency of comments, emotional fluctuations, progress, etc. The analysis then calculates the overall achievement of the meeting's objectives.

[1501] Step 8: Generate and provide feedback

[1502] Server: Based on the analysis results, it generates feedback including an overall evaluation of the meeting and suggestions for improvement, and provides it to the organizer.

[1503] Input: Analysis results

[1504] Output: Feedback message

[1505] Specific operation: The server compiles the analysis results and generates feedback such as, "Overall evaluation of the meeting: The objectives of the meeting were largely achieved, but the following improvements would make the meeting better: - Strengthen time management - Balance speeches." This feedback is provided to the organizer via email or dashboard.

[1506] (Application example 2)

[1507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1508] In conventional online meetings and instruction sharing meetings at logistics centers, even though it is possible to capture and convert audio data into text, there is a lack of means to effectively recognize the nuances of what is being said and the emotions of participants, and to properly analyze and evaluate the progress of the meeting and the degree to which its objectives have been achieved. This makes it difficult to understand the overall content of the meeting or identify areas for improvement, which can lead to a decline in the efficiency and quality of the meeting.

[1509] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data, means for converting voice data into text data, means for proofreading the text data, means for tagging the nuances of utterances based on the text data, means for generating emotion information from the tagged text data and voice data, means for analyzing the emotion information and tagged text data and evaluating the degree of goal achievement of the meeting, and means for generating and providing feedback based on the evaluation. This makes it possible to evaluate the degree of goal achievement of the meeting and provide feedback.

[1510] "Audio data" refers to audio information from meetings or communications recorded in digital format.

[1511] "Text data" is character information obtained by analyzing voice data.

[1512] "Proofreading" is the act of correcting mistranslations and grammatical errors in converted text data.

[1513] "Nuance tagging" refers to assigning labels that indicate the sentiment or intent of a statement based on text data.

[1514] "Emotion information" is data that indicates the user's emotion recognized from voice data or text data.

[1515] "Analysis" is the process of using tagged text data and sentiment information to evaluate the progress of the meeting and the degree to which its objectives were achieved.

[1516] "Evaluation" means analyzing the degree to which the meeting's objectives were achieved and presenting the results quantitatively or qualitatively.

[1517] "Feedback" is information about improvements and progress of the meeting that is generated based on the evaluation results.

[1518] A "system" is a combination of devices and software that executes a series of processes, from acquiring audio data to providing feedback.

[1519] A system for implementing this invention performs a series of processes including acquiring voice data, converting the voice data into text, proofreading the text data, tagging nuances, generating emotional information, analyzing the data, and generating feedback.

[1520] Hardware and software used

[1521] Hardware: Smartphone for primary processing, server for final data processing

[1522] software:

[1523] Speech recognition: Google Cloud Speech-to-Text API

[1524] Emotion recognition: IBM Watson Tone Analyzer

[1525] Data Analysis: Python + Pandas

[1526] Feedback Generation: OpenAI GPT-3

[1527] Processing flow

[1528] Step 1: Acquire audio data

[1529] The server uses smartphones to record audio from meetings at the logistics center, and the recorded audio data is stored in digital format.

[1530] Step 2: Speech to text conversion

[1531] The server receives the recorded audio data and converts it into text using the Google Cloud Speech-to-Text API, which also includes speaker information and speaking time information.

[1532] Step 3: Proofread the text data

[1533] The server proofreads the generated text data, using an automatic proofreading engine to correct grammatical errors and mistranslations.

[1534] Step 4: Nuance tagging

[1535] Based on the proofread text data, the server adds tags that indicate the emotion and intention of the statement as nuances.

[1536] Step 5: Generating emotional information

[1537] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information.

[1538] Step 6: Data analysis

[1539] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree to which the goal was achieved. The evaluation is performed using Python and Pandas.

[1540] Step 7: Generate feedback

[1541] The server generates feedback based on the evaluation results and provides it to the organizer. The feedback is generated using OpenAI's GPT-3 model.

[1542] Specific examples

[1543] At a meeting held at a logistics center, the organizer records the audio of the meeting using a smartphone. After recording is complete, the audio data is uploaded to a server. The server then converts the audio to text using the Google Cloud Speech-to-Text API, correcting grammatical errors and mistranslations using an automatic proofreading engine. Nuances are tagged in the corrected text data, and emotional information about participants is generated using IBM Watson Tone Analyzer. The generated text data and emotional information are analyzed using Python and Pandas, and the degree to which the meeting's objectives were achieved is evaluated. Finally, feedback is generated using OpenAI GPT-3 based on the evaluation results and provided to the organizer.

[1544] Prompt Sentence Examples

[1545] The meeting achieved a goal of 0.75. Please give us your suggestions for improvement.

[1546] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1547] Step 1:

[1548] A user uses a smartphone to record the audio of a meeting at a logistics center. The audio data is stored digitally on the smartphone. When the recording is complete, the user uploads the audio data to a server.

[1549] Step 2:

[1550] The server receives the uploaded audio data and converts it into text using the Google Cloud Speech-to-Text API. The input is audio data, and the output is text data containing speaker information and speech time information. The server sends the audio data to the API and generates corresponding text for each utterance.

[1551] Step 3:

[1552] The server sends the generated text data to an automatic proofreading engine, which corrects any mistranslations or grammatical errors. The input is the generated text data, and the output is the corrected text data. The automatic proofreading engine analyzes the text and makes appropriate corrections.

[1553] Step 4:

[1554] The server performs nuance tagging based on the corrected text data. The input is the corrected text data, and the output is the nuance-tagged text data. The server analyzes the text data and assigns tags based on its sentiment and intent.

[1555] Step 5:

[1556] The server sends the voice data to the IBM Watson Tone Analyzer, which analyzes the tone, pitch, speed, etc. of the voice to generate emotional information. The input is voice data, and the output is emotional information. The server analyzes the voice data, and the emotional engine generates the emotional information.

[1557] Step 6:

[1558] The server analyzes the tagged text data and generated emotion information to evaluate the progress of the meeting and the degree of goal achievement. The input is the tagged text data and emotion information, and the output is the evaluation result of the degree of goal achievement of the meeting. The server analyzes the data using Python and Pandas and performs a quantitative evaluation.

[1559] Step 7:

[1560] The server generates feedback based on the evaluation results and provides it to the organizer. The input is the evaluation result of the meeting's goal achievement, and the output is feedback including areas for improvement. The feedback is generated using OpenAI's GPT-3 model, which generates prompts based on the evaluation results and creates feedback.

[1561] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1562] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1563] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1564] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1565] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1566] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1567] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1568] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1569] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1570] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1571] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1572] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1573] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1574] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1575] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1576] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1577] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1578] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1579] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1580] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1581] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1582] The following is further disclosed regarding the above embodiment.

[1583] (Claim 1)

[1584] means for acquiring audio data;

[1585] means for converting the voice data into text data;

[1586] means for proofreading the text data;

[1587] means for tagging the nuance of a statement based on the text data;

[1588] means for analyzing the tagged text data and evaluating the degree of purpose achievement of the meeting;

[1589] means for generating and providing feedback based on said evaluation;

[1590] A system including:

[1591] (Claim 2)

[1592] 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to the voice data.

[1593] (Claim 3)

[1594] 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors.

[1595] "Example 1"

[1596] (Claim 1)

[1597] means for acquiring audio data;

[1598] means for converting the voice data into text data;

[1599] means for proofreading the text data;

[1600] means for tagging nuances of utterances based on the proofread text data;

[1601] means for analyzing the tagged text data and evaluating the degree of purpose achievement of the meeting;

[1602] means for generating and providing feedback to the organizer based on said evaluation;

[1603] A system including:

[1604] (Claim 2)

[1605] 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to the voice data.

[1606] (Claim 3)

[1607] 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors.

[1608] "Application Example 1"

[1609] (Claim 1)

[1610] means for acquiring audio data;

[1611] means for converting the voice data into text data;

[1612] means for proofreading the text data;

[1613] means for tagging the nuance of a statement based on the text data;

[1614] means for analyzing the tagged text data and evaluating passenger satisfaction and request response status in the autonomous vehicle;

[1615] means for generating and providing feedback based on said evaluation;

[1616] A system including:

[1617] (Claim 2)

[1618] 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to the voice data.

[1619] (Claim 3)

[1620] 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors.

[1621] "Example 2: Combining Emotion Engines"

[1622] (Claim 1)

[1623] means for acquiring audio data;

[1624] means for converting the voice data into text data;

[1625] means for proofreading the text data;

[1626] means for tagging the nuance of a statement based on the text data;

[1627] means for recognizing a user's emotion from the voice data and generating emotion information;

[1628] a means for analyzing the tagged text data and emotion information and evaluating the degree of purpose achievement of the meeting;

[1629] means for generating and providing feedback based on said evaluation;

[1630] A system including:

[1631] (Claim 2)

[1632] 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to the voice data.

[1633] (Claim 3)

[1634] 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors.

[1635] "Application example 2 when combining emotion engines"

[1636] (Claim 1)

[1637] means for acquiring audio data;

[1638] means for converting the voice data into text data;

[1639] means for proofreading the text data;

[1640] means for tagging the nuance of a statement based on the text data;

[1641] means for generating emotion information from the tagged text data and audio data;

[1642] a means for analyzing the emotion information and tagged text data and evaluating the degree of purpose achievement of the meeting;

[1643] means for generating and providing feedback based on said evaluation;

[1644] A system including:

[1645] (Claim 2)

[1646] 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to the voice data.

[1647] (Claim 3)

[1648] 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors. [Explanation of symbols]

[1649] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring audio data; means for converting the voice data into text data; means for proofreading the text data; means for tagging the nuance of a statement based on the text data; means for analyzing the tagged text data and evaluating the degree of purpose achievement of the meeting; means for generating and providing feedback based on said evaluation; A system including:

2. 2. The system according to claim 1, further comprising means for adding speaker information and speech time information to said voice data.

3. 2. The system according to claim 1, wherein the means for proofreading the text data includes means for correcting mistranslations and grammatical errors.

Images