system

Abstract

We provide the system. [Solution] A means of converting speech into text data, A means of analyzing text data using natural language processing technology to identify the communication style of individual users, A means for generating customized feedback for the user based on identified communication styles, A means of proposing a training program suitable for the user based on the generated feedback, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern corporate activities, there is a problem that individual employees lack opportunities to accurately understand their own communication methods and utilize them for improvement. In addition, with the spread of remote work, improving communication skills in an online environment has been emphasized, but there is a lack of practical and effective means of providing training. In order to improve such a situation, there is a demand for a system that promotes self-awareness among employees and continuously improves communication skills.

Means for Solving the Problems

[0005] This invention first provides a means for converting audio data from meetings and online conferences into text data in real time. Using this converted text data, natural language processing technology is used to analyze the content and tone of the user's speech, thereby identifying the user's communication style. Next, based on this analysis, a means is used to customize and generate feedback for each user. This feedback not only indicates specific areas for improvement and strengths, but also includes a means for proposing an individually optimized training program based on that feedback, and further includes a means for recording the user's training progress and enabling tracking of progress. Through the above means, users can understand their own communication characteristics and continuously improve their skills.

[0006] "Speech" refers to the sound signals generated from human speech in meetings, online conferences, and other similar situations.

[0007] "Text data" refers to data in text format that is generated by converting speech.

[0008] "Natural language processing technology" refers to the technology that enables computers to understand and analyze human language.

[0009] "Communication style" refers to the characteristics of each user's speech and response patterns.

[0010] "Feedback" refers to advice and information about the user's strengths and areas for improvement, provided based on an analyzed communication style.

[0011] A "training program" refers to a series of exercises and learning activities designed to improve users' communication skills.

[0012] "Progress" refers to the process or results that show the extent to which a user has achieved their set goals. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system that utilizes speech recognition and natural language processing technologies to improve the communication skills of individual users. This system converts speech generated during meetings and online conferences into text data in real time, and then analyzes that data to identify each user's communication style. Based on the analysis results, it provides accurate feedback to the user and recommends an individualized training program.

[0035] Server: The server receives audio data streamed from the meeting and converts it into text data in real time. This conversion is performed using an advanced speech recognition engine, ensuring fast and accurate text conversion. The converted text data is passed to a natural language processing module, where grammar, sentiment, and tone analysis is performed. Based on the information collected through the analysis, a machine learning algorithm classifies and identifies the communication style. Based on the identified style, generative AI is used to create pinpoint feedback and generate a suitable training program.

[0036] Terminal: The terminal visualizes the feedback and training program provided by the server and displays it in a way that is easy for the user to understand. The terminal also implements support functions so that users can check the feedback and reflect it in their daily actions. Furthermore, it records the progress of the training program, contributing to future pacing.

[0037] User: Users improve their speaking style and expression by carefully reviewing and understanding the feedback received through their devices. They strive to improve their skills at their own pace by following the training program provided by the server. Progress in this program is recorded, allowing users to track their achievements in a compatible format and visually see which speaking points they have improved and which goals they have achieved.

[0038] Specific example: For instance, suppose a user's comments in a meeting don't align with those around them, resulting in a misunderstanding of their intended meaning. In this case, the system converts the comments into text data and performs sentiment analysis to analyze communication style, including tone and emphasis. Based on this, it provides the user with specific feedback for improvement, such as "adding clearer preambles before and after comments" and "inserting positive feedback as appropriate," and suggests related training such as internal workshops or peer review sessions. By repeating this process, the user's communication skills improve, and the system is designed to allow for reassessment.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server receives audio data streamed from the conferencing application and divides this data into small chunks in real time. The speech recognition engine then converts these divided audio chunks into text data.

[0042] Step 2:

[0043] The server passes the converted text data to a natural language processing module. Here, grammatical analysis, sentiment analysis, and tonal analysis are performed to extract the content and tone characteristics of the utterance.

[0044] Step 3:

[0045] The server uses NLP analysis results to apply machine learning algorithms and identify each user's communication style. This style includes factors such as frequency of speech, emotional intensity, and tone diversity.

[0046] Step 4:

[0047] Based on identified communication styles, the server utilizes generative AI to create personalized feedback for each user. This feedback includes specific areas for improvement and details of strengths.

[0048] Step 5:

[0049] The server generates individually optimized training programs based on feedback. These programs include specific training sessions and self-study suggestions.

[0050] Step 6:

[0051] The terminal receives feedback and training programs sent from the server. The terminal visualizes this information through a user interface, making it easy for the user to understand.

[0052] Step 7:

[0053] Users review the feedback content via their device and understand their own communication style. Following the provided training program, they apply the feedback to their daily activities.

[0054] Step 8:

[0055] The device records the user's progress through the training program. By accumulating progress data, users can visually track their skill improvement.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In modern meetings and online conferences, many users face the problem of their communication skills not being effectively utilized, resulting in their intended message not being conveyed to others. Furthermore, it is usually difficult for users to accurately understand and improve their own communication patterns. Given this situation, there is a need to improve communication skills through the provision of appropriate and customized feedback and training programs tailored to each user.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information using natural language processing technology to identify the communication patterns of individual users, and means for generating a customized response for the user based on the identified communication patterns. This enables users to understand their own communication style and take concrete actions to improve it.

[0061] "Auditory information" refers to data obtained through sound vibrations, and in particular, recordings of human speech.

[0062] "Text information" refers to data composed of electronically represented strings of characters or sentences.

[0063] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language.

[0064] A "communication pattern" is a distinctive style in which individual users communicate using their own unique verbal and nonverbal means.

[0065] A "response" is an automatic or planned reaction from a system to a specific situation or input.

[0066] A "learning program" is a planned series of activities and training aimed at improving a user's abilities and acquiring knowledge.

[0067] A "prompt statement" is an input statement used to guide a generative AI model to a specific output.

[0068] To implement this invention, the server, terminal, and user each provide system components that play specific roles.

[0069] server

[0070] The server receives "audio information" streamed from meetings and online conferences and converts it into "text information" in real time. A general-purpose cloud service is used as the speech recognition engine for this conversion. Specifically, a speech recognition API is utilized to convert "audio information" into "text information." The converted "text information" is then analyzed using "natural language processing technology." Natural language processing libraries (e.g., NLTK or spaCy) are used for the analysis to identify "communication patterns." After identification, a generative AI model is used to generate "responses" based on the identified patterns. "Prompt statements" are used in this process. Example of a prompt statement: Please suggest improvements if a user's speech is not understood in a meeting.

[0071] terminal

[0072] The terminal's role is to visualize the "responses" and "learning programs" provided by the server. The user interface is designed to be intuitively easy for users to understand. Specifically, it uses a web application platform (e.g., React or Vue.js) to display the "responses" and help users understand their own areas for improvement based on them. The terminal also records the progress of the "learning program," allowing users to visually check their own progress.

[0073] User

[0074] Users receive feedback from the server through their devices and take action to improve their own "communication patterns." For example, if a user makes a statement that is difficult to understand during a meeting, they can adjust their speaking style based on the "response" provided by the server. Furthermore, it is expected that they will gradually improve their skills by regularly conducting self-assessments through the progress management function provided by their devices.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server receives audio information streamed in real time from conferences and meetings. The audio information is input via microphones and communication networks. The server converts this audio information into text using a speech recognition API. The converted text information is then output.

[0078] Step 2:

[0079] The server analyzes the obtained text information using natural language processing techniques. Specifically, it uses libraries such as NLTK and spaCy to analyze the grammatical structure, sentiment, and tone of the text. Based on this input text, it detects grammatical errors and extracts attributes such as positive / negative sentiment, outputting them as analysis results.

[0080] Step 3:

[0081] The server identifies user communication patterns based on the analysis results. This process uses machine learning algorithms to compare past and current data, extracting unique features. The analysis results are used as input, and the identified communication patterns are output.

[0082] Step 4:

[0083] The server utilizes a generative AI model to generate responses based on identified communication patterns. It uses prompt statements as input, instructing the generative AI with phrases like, "Please suggest improvements for situations where a user's message isn't understood in a meeting." This results in a customized response tailored to the user.

[0084] Step 5:

[0085] The terminal visualizes the responses received from the server and the learning program, and presents them to the user. It uses the response information from the server as input and displays the visualized response information as output via a web application. The user then uses this visual information to identify areas for improvement.

[0086] Step 6:

[0087] Based on the responses received on the device, users take action to improve their communication methods. They use the information displayed on the device screen to take specific improvement actions and reflect them in their daily communication.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In modern society, improving individual communication skills is considered important. However, the means of providing personalized, real-time feedback to individual users in home and educational settings are limited. In particular, there is a growing need for systems that utilize speech recognition and natural language processing technologies, but such technologies are generally complex and expensive, posing a challenge to widespread adoption. This invention aims to solve these problems and provide affordable and effective support for improving communication.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for converting speech into digital data, means for analyzing the digital data using natural language processing technology to identify the individual user's communication style, and means for creating a response adapted to the user based on the identified communication style. This makes it possible for robots used in home and educational environments to provide real-time, effective, and low-cost feedback through interaction with users.

[0093] "Converting audio to digital data" is the process of converting audio signals into a text format that can be processed by a computer.

[0094] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0095] "Individual user communication styles" refer to the unique ways of speaking and expressing oneself that each user possesses.

[0096] "Generating a response" refers to the process of generating appropriate feedback or instructions for the user based on analyzed information.

[0097] "Interacting with users in a home or educational setting" refers to situations in which a robot communicates with an individual or a group.

[0098] "Providing real-time feedback" refers to a process of responding immediately to user actions and comments.

[0099] In the system that implements this application, the server uses a high-precision speech recognition engine to convert speech into digital data. This engine uses existing speech recognition technologies, such as Google® Speech-to-Text API, to quickly and accurately convert the input speech into text format. The converted text is then analyzed using natural language processing technologies, specifically Python's NLTK and spaCy libraries. This analysis reveals the individual user's communication style, and appropriate responses are generated based on the user's utterances and tone of voice.

[0100] The terminal displays real-time feedback from the server to the user. This could potentially utilize single-board computers such as Raspberry Pi or Jetson Nano, allowing for direct and easy information transmission to the user through a user interface.

[0101] Users can improve their communication skills in their daily lives by following the feedback and suggested guidance programs provided by the robot. Progress is continuously recorded and tracked in a visualized format, supporting users' self-improvement over the long term.

[0102] For example, if a child wants to improve their conversations with friends, the robot can analyze the child's speaking style at home and provide specific advice such as "softening your facial expressions when speaking" or "asking questions at the right time." Another example of a prompt when using a generative AI model would be: "What improvements are needed for the user to communicate more clearly and effectively? Please suggest specific advice and training menus."

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] The server receives audio data from the user. This input is audio streamed in real time, and the server converts this data into text using the Google Speech-to-Text API. The output is recognized text data. Specifically, certain keywords or phrases are picked from the audio signal and formatted into text.

[0106] Step 2:

[0107] The server analyzes the generated text data using natural language processing engines such as NLTK and spaCy. The input is the text data obtained in step 1. As a result of the analysis, information such as grammatical structure, sentiment analysis, and tone is extracted. The output is data about the user's communication style based on the analyzed information.

[0108] Step 3:

[0109] The server uses the data obtained in Step 2 to generate feedback for the user using a generative AI model. The input is analytical data, and the output is responses and advice adapted to the user's specific communication style. At this stage, specific feedback messages are generated, including suggestions for training programs.

[0110] Step 4:

[0111] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand format through the user interface. The input is the feedback data generated in step 3. The output is the information displayed on the screen that the user sees. Specifically, the feedback is displayed in text or audio format, making it easy for the user to immediately translate it into action.

[0112] Step 5:

[0113] Users improve their daily communication style based on feedback and training programs provided on the device. Input includes feedback heard on screen and via audio, as well as how the user puts it into practice. Output is the visible change as an improvement in the user's skills. In terms of concrete actions, users take specific actions based on the feedback and track their own progress.

[0114] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0115] This invention is a system developed to improve the communication skills of individual users by utilizing speech recognition technology, natural language processing technology, and emotion recognition technology. This system processes speech during meetings and online conferences into text data in real time, and by analyzing this text data, it can identify each user's communication style and emotional state. Based on the analysis results, it provides individually optimized feedback to the user and proposes a training program based on that feedback.

[0116] The server first receives the streaming audio data and converts it into text data in real time using a high-precision speech recognition engine. This text data is then analyzed using natural language processing technology to obtain the content and tone of speech, as well as the emotional state using an emotion engine. This emotion engine instantly determines the user's emotions based on the audio and text data and incorporates the result into the evaluation of the communication style. Based on this evaluation, a machine learning algorithm creates feedback optimized for each individual user and adjusts the content as needed. It also considers how specific emotional states may change advice and action guidelines for skill improvement when generating a training program.

[0117] The device receives feedback and training programs supplied from the server and displays them in an easy-to-understand format for the user. The feedback includes not only strengths and areas for improvement, but also specific, emotion-based approaches. The device provides interactive support tailored to the user's usage scenario, enabling users to apply the feedback to their daily activities. It also automatically records user behavior to monitor progress and help improve skills in the future.

[0118] Users review the displayed feedback to deepen their understanding of their own communication skills and emotional states. Based on this understanding, they work to improve their communication in their daily work. Furthermore, by implementing the provided training program and utilizing real-time feedback from the emotion engine to continuously improve their skills, they can learn effective communication methods that respond immediately to their own changes.

[0119] For example, if a user lacks confidence in expressing themselves, the emotional engine recognizes this anxiety, and the server uses this information to provide specific feedback such as "practices for speaking with confidence" and "emphasize positive feedback." Based on this feedback, the system also recommends effective ways to respond to positive feedback and mental techniques for gaining confidence as part of a training program. This invention promotes the improvement of users' self-awareness and skills through such methods.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The server receives audio data streamed from the conferencing application. This audio data is converted into text data in real time using a speech recognition engine. The audio signal is divided into short chunks to improve accuracy.

[0123] Step 2:

[0124] The server passes the converted text data to a natural language processing module. This module analyzes the text data to determine the context, grammatical structure, and intent of the utterance, thereby identifying the overall meaning of the statement. Simultaneously, the emotion engine determines the user's emotional state from the audio and text.

[0125] Step 3:

[0126] Based on the analyzed communication style and emotions, the server uses machine learning algorithms to generate personalized feedback for each user. Emotional information is used to adjust the emphasis and suggestions in the feedback. This feedback includes identifying strengths and providing specific advice on areas for improvement.

[0127] Step 4:

[0128] Based on feedback, the server creates a training program optimized for the user. The training program suggests specific action plans and practice tasks to manage emotions that affect communication skills.

[0129] Step 5:

[0130] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand visual format. The user interface is designed to allow users to intuitively understand the feedback and apply it to their daily work.

[0131] Step 6:

[0132] Users review the feedback provided on their devices. Based on information about their communication style and emotional state, they understand areas for improvement in their daily communication and put those improvements into practice.

[0133] Step 7:

[0134] Users participate in a feedback-based training program. Program progress is tracked via a device, allowing users to monitor their own growth. The training includes managing emotional responses and learning effective communication techniques.

[0135] Step 8:

[0136] The server stores user training results and progress data, which is then used for feedback and to improve future training programs. This feedback loop allows the content to evolve as the user grows, supporting continuous skill development.

[0137] (Example 2)

[0138] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0139] In today's communication environment, users are expected to appropriately understand and improve their own statements and expressions. However, there is a lack of means for individual users to improve themselves in real time based on their emotional state and communication style in the moment. To solve this problem, a system is needed that can instantly analyze a user's emotional state and communication style and provide specific feedback and training methods based on that analysis.

[0140] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0141] In this invention, the server includes means for converting voice information into text data, means for analyzing the text information using language analysis technology to identify the communication method of each user, and means for determining the emotional state from the voice information and text information. This makes it possible to provide users with appropriate feedback and suggestions for training methods in real time.

[0142] "Audio information" refers to data obtained by converting audio signals into a digital format.

[0143] "Text data" refers to information in text format obtained after audio information has been converted.

[0144] "Linguistic analysis technology" refers to techniques for analyzing text information and interpreting its grammatical and contextual meaning.

[0145] "Communication method" refers to the style and techniques of communication used by each user.

[0146] "Response" refers to the feedback that the system generates for the user based on the analyzed information.

[0147] "Training methods" refer to programs proposed to users to improve their skills and communication abilities.

[0148] "Emotional state" refers to the internal emotions judged from the user's statements and voice.

[0149] "Activity logging" refers to the process of saving user interactions and actions as logs and tracking progress.

[0150] To implement this invention, it is important to use various engines that perform speech recognition, natural language processing, and emotion recognition. Specifically, the server first uses a "speech recognition engine" to convert the received speech information into text data. Here, a general cloud-based service that boasts high accuracy can be used as the speech recognition engine.

[0151] Next, the server analyzes this text data using a "language analysis engine." This technology performs grammatical analysis and keyword extraction to identify the user's communication method. Furthermore, it uses an "emotion recognition engine" to evaluate the user's emotional state from the audio information and text data. This process provides the foundational data for generating situation-appropriate feedback.

[0152] The server applies machine learning algorithms to this data to generate user-customized responses and training methods. Common machine learning frameworks such as "TENSORFLOW®" can be used, which can improve the accuracy of the feedback and training programs.

[0153] The device retrieves responses and training methods sent from the server and displays them in a user-friendly format. Through this information provided by the device, users can improve their communication skills. The device also provides an interface that records user activity and allows for progress tracking.

[0154] For example, if a user makes a negative comment during an online meeting, emotion recognition technology could identify that emotion, and the server could generate responses such as "how to receive positive feedback" or "how to offer constructive opinions." This would provide users with an efficient means to improve their communication style.

[0155] Furthermore, an example of a prompt to be input to a generative AI model is: "Design a system that analyzes user statements in real time in a communication environment and provides emotion-based feedback. Explain how to generate a training method that takes into account the emotional state of individual users." This prompt clearly communicates a specific objective and serves as a foundation for effectively utilizing the calculations of the generative AI model.

[0156] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0157] Step 1:

[0158] The server receives audio information obtained from meetings and online conferences. This audio information is input into the "speech recognition engine" and converted into text data. The converted text becomes the basis for the next process. Based on the input audio data, the server outputs the spoken content as text information.

[0159] Step 2:

[0160] The server inputs the generated text data into a "language analysis engine." This engine analyzes the grammatical structure of the text and identifies important keywords and the user's communication method. The analysis results obtained from the input text data include information about the intent and style of the utterance.

[0161] Step 3:

[0162] The server inputs text data and analysis results into the "emotion recognition engine." This engine evaluates the user's emotional state and determines what emotion was behind their statements. Based on the input of voice and text data, the server outputs the emotional state in real time, which is used to generate responses in the next step.

[0163] Step 4:

[0164] The server utilizes the previously analyzed results and uses machine learning algorithms to generate responses and training methods for the user. In this case, it creates content tailored to the user's unique communication style and emotional state. The server's input includes the analyzed data, and the output generates customized responses and training programs.

[0165] Step 5:

[0166] The terminal receives responses and training methods sent from the server and displays them to the user on an interface. The terminal provides information in an intuitive and easy-to-understand way, making it easy for the user to learn. Input to the terminal is data from the server, and output is feedback displayed to the user.

[0167] Step 6:

[0168] Users review the feedback provided by the device and take action to improve their communication skills. At this stage, they can take action and practice based on the feedback. The user's input is feedback, and the output is actual improvement actions.

[0169] (Application Example 2)

[0170] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0171] In modern family environments, improving individual communication skills is crucial, and there is a need to enhance the quality of communication within families. However, it is difficult for individuals to objectively understand and improve their own communication style and emotional state. Furthermore, there is a lack of means to receive regular feedback on their speaking style and appropriate learning programs. Therefore, there is a need for an effective system that can easily improve communication skills within the family.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0173] In this invention, the server includes means for converting speech into linguistic data, means for analyzing the linguistic data using natural language processing technology to identify the information exchange style of each user, means for generating a customized response for the user based on the identified information exchange style, means for the home machine to analyze the user's daily conversation in real time and provide individualized feedback and improvement suggestions, and means for displaying the generated feedback and improvement suggestions in audible and visual form. This enables users in a home environment to objectively evaluate their own information exchange skills and receive individually optimized feedback and learning programs.

[0174] "Means of converting speech into linguistic data" refers to technologies that convert speech information spoken by a user into text data that can be understood by a machine.

[0175] "Natural language processing technology" is a technique that uses computers to analyze linguistic data and understand its meaning and context.

[0176] "Information exchange style" refers to the unique methods of communication and patterns that individual users express in conversations and written texts.

[0177] "Means for generating customized responses" refers to technologies that create feedback and instructions tailored to each user based on their individual characteristics and circumstances.

[0178] "A means by which home devices analyze users' everyday conversations in real time and provide personalized feedback and improvement suggestions" refers to a system in which devices used in the home instantly analyze users' conversations and suggest appropriate advice and improvement methods.

[0179] "Means of displaying generated feedback and improvement measures in audible or visual form" refers to a method of generating feedback and improvement suggestions from analysis results either by emitting sound or displaying them on a screen or similar device.

[0180] In the system implementing this invention, a server acts as the central point for receiving voice data and converting it into text data. This process uses the "Google Speech-to-Text API" as the speech recognition technology. This API has the function of converting the user's voice input into text data with high accuracy.

[0181] The converted text data is analyzed through a natural language processing (NLP) process. The Google Language API is used to analyze the meaning and context of the text data and identify the information exchange style. Based on this analysis, a customized response tailored to the user is generated. This process utilizes a pre-trained generative AI model using TensorFlow to construct an appropriate response.

[0182] Furthermore, home devices, or terminals, monitor the user's daily conversations in real time. Using tools such as "IBM Watson® Tone Analyzer," the system analyzes the emotions expressed during conversations and provides feedback. The generated feedback and improvement suggestions are presented to the user in auditory and visual formats, enhancing the quality of information exchange within the home.

[0183] As a concrete example, consider a situation during a family conversation where the father seeks to communicate with his child in a less intimidating way. In this case, the device analyzes the father's statements in real time, offers suggestions to alleviate his emotions, and presents them via voice or on the screen. This facilitates smoother communication within the family.

[0184] An example prompt could be, "During a family meeting, analyze participants' speaking styles and emotions in real time and provide feedback to facilitate positive conversation." This prompt allows the system to provide specific analysis and feedback, supporting the user in improving their communication skills.

[0185] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0186] Step 1:

[0187] The server receives audio data transmitted from home appliances. The input is audio data, and the output is text data generated by a speech recognition engine. This audio data is converted into text data in a linguistic format using the "Google Speech-to-Text API". In this conversion process, the user's voice is recognized in real time to generate highly accurate text data.

[0188] Step 2:

[0189] The server receives the converted text data and performs natural language processing using the Google Language API. The input is the text data generated in step 1, and the output is the parsed content information. Here, the content and context of the text data are analyzed to identify the user's information exchange style. This process involves grammatical analysis and keyword extraction to understand the user's communication style.

[0190] Step 3:

[0191] The server generates customized feedback based on the analysis results. This process uses a generative AI model trained with TensorFlow. The input is the analysis results from step 2, and the output is the customized feedback data. The AI ​​model constructs appropriate responses based on past data, creating feedback and improvement suggestions tailored to the user.

[0192] Step 4:

[0193] The terminal presents the customized feedback received from the server to the user audibly or visually. The input is the feedback data generated in step 3, and the output is the format in which it is presented to the user. This process uses audio output and display functions to provide information in a way that is easy for the user to understand.

[0194] Step 5:

[0195] Based on the feedback provided, users begin taking action to improve their communication skills. The input is the feedback presented in step 4, and the output is the change in the user's behavior. Users implement the suggested improvement measures and training programs in their daily lives to enhance their skills.

[0196] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0197] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0198] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0199] [Second Embodiment]

[0200] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0201] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0202] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0203] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0204] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0205] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0206] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0207] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0208] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0209] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0210] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0211] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0212] This invention is a system that utilizes speech recognition and natural language processing technologies to improve the communication skills of individual users. This system converts speech generated during meetings and online conferences into text data in real time, and then analyzes that data to identify each user's communication style. Based on the analysis results, it provides accurate feedback to the user and recommends an individualized training program.

[0213] Server: The server receives audio data streamed from the meeting and converts it into text data in real time. This conversion is performed using an advanced speech recognition engine, ensuring fast and accurate text conversion. The converted text data is passed to a natural language processing module, where grammar, sentiment, and tone analysis is performed. Based on the information collected through the analysis, a machine learning algorithm classifies and identifies the communication style. Based on the identified style, generative AI is used to create pinpoint feedback and generate a suitable training program.

[0214] Terminal: The terminal visualizes the feedback and training program provided by the server and displays it in a way that is easy for the user to understand. The terminal also implements support functions so that users can check the feedback and reflect it in their daily actions. Furthermore, it records the progress of the training program, contributing to future pacing.

[0215] User: Users improve their speaking style and expression by carefully reviewing and understanding the feedback received through their devices. They strive to improve their skills at their own pace by following the training program provided by the server. Progress in this program is recorded, allowing users to track their achievements in a compatible format and visually see which speaking points they have improved and which goals they have achieved.

[0216] Specific example: For instance, suppose a user's comments in a meeting don't align with those around them, resulting in a misunderstanding of their intended meaning. In this case, the system converts the comments into text data and performs sentiment analysis to analyze communication style, including tone and emphasis. Based on this, it provides the user with specific feedback for improvement, such as "adding clearer preambles before and after comments" and "inserting positive feedback as appropriate," and suggests related training such as internal workshops or peer review sessions. By repeating this process, the user's communication skills improve, and the system is designed to allow for reassessment.

[0217] The following describes the processing flow.

[0218] Step 1:

[0219] The server receives audio data streamed from the conferencing application and divides this data into small chunks in real time. The speech recognition engine then converts these divided audio chunks into text data.

[0220] Step 2:

[0221] The server passes the converted text data to a natural language processing module. Here, grammatical analysis, sentiment analysis, and tonal analysis are performed to extract the content and tone characteristics of the utterance.

[0222] Step 3:

[0223] The server uses NLP analysis results to apply machine learning algorithms and identify each user's communication style. This style includes factors such as frequency of speech, emotional intensity, and tone diversity.

[0224] Step 4:

[0225] Based on identified communication styles, the server utilizes generative AI to create personalized feedback for each user. This feedback includes specific areas for improvement and details of strengths.

[0226] Step 5:

[0227] The server generates individually optimized training programs based on feedback. These programs include specific training sessions and self-study suggestions.

[0228] Step 6:

[0229] The terminal receives feedback and training programs sent from the server. The terminal visualizes this information through a user interface, making it easy for the user to understand.

[0230] Step 7:

[0231] Users review the feedback content via their device and understand their own communication style. Following the provided training program, they apply the feedback to their daily activities.

[0232] Step 8:

[0233] The device records the user's progress through the training program. By accumulating progress data, users can visually track their skill improvement.

[0234] (Example 1)

[0235] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0236] In modern meetings and online conferences, many users face the problem of their communication skills not being effectively utilized, resulting in their intended message not being conveyed to others. Furthermore, it is usually difficult for users to accurately understand and improve their own communication patterns. Given this situation, there is a need to improve communication skills through the provision of appropriate and customized feedback and training programs tailored to each user.

[0237] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0238] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information using natural language processing technology to identify the communication patterns of individual users, and means for generating a customized response for the user based on the identified communication patterns. This enables users to understand their own communication style and take concrete actions to improve it.

[0239] "Auditory information" refers to data obtained through sound vibrations, and in particular, recordings of human speech.

[0240] "Text information" refers to data composed of electronically represented strings of characters or sentences.

[0241] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language.

[0242] A "communication pattern" is a distinctive style in which individual users communicate using their own unique verbal and nonverbal means.

[0243] A "response" is an automatic or planned reaction from a system to a specific situation or input.

[0244] A "learning program" is a planned series of activities and training aimed at improving a user's abilities and acquiring knowledge.

[0245] A "prompt statement" is an input statement used to guide a generative AI model to a specific output.

[0246] To implement this invention, the server, terminal, and user each provide system components that play specific roles.

[0247] server

[0248] The server receives "audio information" streamed from meetings and online conferences and converts it into "text information" in real time. A general-purpose cloud service is used as the speech recognition engine for this conversion. Specifically, a speech recognition API is utilized to convert "audio information" into "text information." The converted "text information" is then analyzed using "natural language processing technology." Natural language processing libraries (e.g., NLTK or spaCy) are used for the analysis to identify "communication patterns." After identification, a generative AI model is used to generate "responses" based on the identified patterns. "Prompt statements" are used in this process. Example of a prompt statement: Please suggest improvements if a user's speech is not understood in a meeting.

[0249] terminal

[0250] The terminal's role is to visualize the "responses" and "learning programs" provided by the server. The user interface is designed to be intuitively easy for users to understand. Specifically, it uses a web application platform (e.g., React or Vue.js) to display the "responses" and help users understand their own areas for improvement based on them. The terminal also records the progress of the "learning program," allowing users to visually check their own progress.

[0251] User

[0252] Users receive feedback from the server through their devices and take action to improve their own "communication patterns." For example, if a user makes a statement that is difficult to understand during a meeting, they can adjust their speaking style based on the "response" provided by the server. Furthermore, it is expected that they will gradually improve their skills by regularly conducting self-assessments through the progress management function provided by their devices.

[0253] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0254] Step 1:

[0255] The server receives audio information streamed in real time from conferences and meetings. The audio information is input via microphones and communication networks. The server converts this audio information into text using a speech recognition API. The converted text information is then output.

[0256] Step 2:

[0257] The server analyzes the obtained text information using natural language processing techniques. Specifically, it uses libraries such as NLTK and spaCy to analyze the grammatical structure, sentiment, and tone of the text. Based on this input text, it detects grammatical errors and extracts attributes such as positive / negative sentiment, outputting them as analysis results.

[0258] Step 3:

[0259] The server identifies user communication patterns based on the analysis results. This process uses machine learning algorithms to compare past and current data, extracting unique features. The analysis results are used as input, and the identified communication patterns are output.

[0260] Step 4:

[0261] The server utilizes a generative AI model to generate responses based on identified communication patterns. It uses prompt statements as input, instructing the generative AI with phrases like, "Please suggest improvements for situations where a user's message isn't understood in a meeting." This results in a customized response tailored to the user.

[0262] Step 5:

[0263] The terminal visualizes the responses received from the server and the learning program, and presents them to the user. It uses the response information from the server as input and displays the visualized response information as output via a web application. The user then uses this visual information to identify areas for improvement.

[0264] Step 6:

[0265] Based on the responses received on the device, users take action to improve their communication methods. They use the information displayed on the device screen to take specific improvement actions and reflect them in their daily communication.

[0266] (Application Example 1)

[0267] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0268] In modern society, improving individual communication skills is considered important. However, the means of providing personalized, real-time feedback to individual users in home and educational settings are limited. In particular, there is a growing need for systems that utilize speech recognition and natural language processing technologies, but such technologies are generally complex and expensive, posing a challenge to widespread adoption. This invention aims to solve these problems and provide affordable and effective support for improving communication.

[0269] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0270] In this invention, the server includes means for converting speech into digital data, means for analyzing the digital data using natural language processing technology to identify the individual user's communication style, and means for creating a response adapted to the user based on the identified communication style. This makes it possible for robots used in home and educational environments to provide real-time, effective, and low-cost feedback through interaction with users.

[0271] "Converting audio to digital data" is the process of converting audio signals into a text format that can be processed by a computer.

[0272] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0273] "Individual user communication styles" refer to the unique ways of speaking and expressing oneself that each user possesses.

[0274] "Generating a response" refers to the process of generating appropriate feedback or instructions for the user based on analyzed information.

[0275] "Interacting with users in a home or educational setting" refers to situations in which a robot communicates with an individual or a group.

[0276] "Providing real-time feedback" refers to a process of responding immediately to user actions and comments.

[0277] In the system that implements this application, the server uses a high-precision speech recognition engine to convert speech into digital data. This engine uses existing speech recognition technologies, such as the Google Speech-to-Text API, to quickly and accurately convert the input speech into text format. The converted text is then analyzed using natural language processing technologies, specifically Python's NLTK and spaCy libraries. This analysis reveals the individual user's communication style, and appropriate responses are generated based on the user's utterances and tone of voice.

[0278] The terminal displays real-time feedback from the server to the user. This could potentially utilize single-board computers such as Raspberry Pi or Jetson Nano, allowing for direct and easy information transmission to the user through a user interface.

[0279] Users can improve their communication skills in their daily lives by following the feedback and suggested guidance programs provided by the robot. Progress is continuously recorded and tracked in a visualized format, supporting users' self-improvement over the long term.

[0280] For example, if a child wants to improve their conversations with friends, the robot can analyze the child's speaking style at home and provide specific advice such as "softening your facial expressions when speaking" or "asking questions at the right time." Another example of a prompt when using a generative AI model would be: "What improvements are needed for the user to communicate more clearly and effectively? Please suggest specific advice and training menus."

[0281] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0282] Step 1:

[0283] The server receives voice data from the user. This input is voice streamed in real time, and the server converts this data into text using the Google Speech-to-Text API. The output is the recognized text data. As a specific operation, specific keywords and phrases are picked up from the voice signal and formatted into text form.

[0284] Step 2:

[0285] The server analyzes the generated text data using natural language analysis engines such as NLTK and spaCy. The input is the text data obtained in Step 1. As a result of the analysis, information such as grammatical structure, sentiment analysis, and tone is extracted. The output is data regarding the user's information transmission style based on the analyzed information.

[0286] Step 3:

[0287] The server uses a generative AI model with the data obtained in Step 2 to generate feedback for the user. The input is the analysis data, and the output is a response or advice adapted to the user's specific communication style. At this stage, specific feedback messages are generated, including proposals for training programs.

[0288] Step 4:

[0289] The terminal receives the feedback and training program sent from the server and displays them in an easy-to-understand form for the user through the user interface. The input is the feedback data generated in Step 3. The output is the information on the screen that the user sees. As a specific operation, the feedback is displayed in the form of text or voice, making it easy for the user to immediately reflect it in their actions.

[0290] Step 5:

[0291] Users improve their daily communication style based on feedback and training programs provided on the device. Input includes feedback heard on screen and via audio, as well as how the user puts it into practice. Output is the visible change as an improvement in the user's skills. In terms of concrete actions, users take specific actions based on the feedback and track their own progress.

[0292] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0293] This invention is a system developed to improve the communication skills of individual users by utilizing speech recognition technology, natural language processing technology, and emotion recognition technology. This system processes speech during meetings and online conferences into text data in real time, and by analyzing this text data, it can identify each user's communication style and emotional state. Based on the analysis results, it provides individually optimized feedback to the user and proposes a training program based on that feedback.

[0294] The server first receives the streaming audio data and converts it into text data in real time using a high-precision speech recognition engine. This text data is then analyzed using natural language processing technology to obtain the content and tone of speech, as well as the emotional state using an emotion engine. This emotion engine instantly determines the user's emotions based on the audio and text data and incorporates the result into the evaluation of the communication style. Based on this evaluation, a machine learning algorithm creates feedback optimized for each individual user and adjusts the content as needed. It also considers how specific emotional states may change advice and action guidelines for skill improvement when generating a training program.

[0295] The device receives feedback and training programs supplied from the server and displays them in an easy-to-understand format for the user. The feedback includes not only strengths and areas for improvement, but also specific, emotion-based approaches. The device provides interactive support tailored to the user's usage scenario, enabling users to apply the feedback to their daily activities. It also automatically records user behavior to monitor progress and help improve skills in the future.

[0296] Users review the displayed feedback to deepen their understanding of their own communication skills and emotional states. Based on this understanding, they work to improve their communication in their daily work. Furthermore, by implementing the provided training program and utilizing real-time feedback from the emotion engine to continuously improve their skills, they can learn effective communication methods that respond immediately to their own changes.

[0297] For example, if a user lacks confidence in expressing themselves, the emotional engine recognizes this anxiety, and the server uses this information to provide specific feedback such as "practices for speaking with confidence" and "emphasize positive feedback." Based on this feedback, the system also recommends effective ways to respond to positive feedback and mental techniques for gaining confidence as part of a training program. This invention promotes the improvement of users' self-awareness and skills through such methods.

[0298] The following describes the processing flow.

[0299] Step 1:

[0300] The server receives audio data streamed from the conferencing application. This audio data is converted into text data in real time using a speech recognition engine. The audio signal is divided into short chunks to improve accuracy.

[0301] Step 2:

[0302] The server passes the converted character data to the natural language processing module. This module analyzes the context, grammar structure, intention, etc. of the speech from the character data to identify the meaning of the entire speech. At the same time, the emotion engine determines the user's emotional state from the voice and text.

[0303] Step 3:

[0304] Based on the analyzed communication style and emotion, the server uses machine learning algorithms to generate customized feedback for each user. The emotion information is utilized as a factor to adjust the emphasis points and proposed content of the feedback. This feedback includes specific advice on highlighting strengths and areas for improvement.

[0305] Step 4:

[0306] Based on the feedback, the server creates a training program optimized for the user. The training program proposes specific action plans and practice tasks for managing emotions that affect communication skills.

[0307] Step 5:

[0308] The terminal receives the feedback and training program sent from the server and displays them in a visually understandable way. The user interface is designed so that the user can intuitively understand the feedback and apply it to daily work.

[0309] Step 6:

[0310] The user checks the feedback provided on the terminal. Based on the information about their own communication style and emotional state, they understand the areas for improvement in daily communication and put them into practice.

[0311] Step 7:

[0312] Users participate in a feedback-based training program. Program progress is tracked via a device, allowing users to monitor their own growth. The training includes managing emotional responses and learning effective communication techniques.

[0313] Step 8:

[0314] The server stores user training results and progress data, which is then used for feedback and to improve future training programs. This feedback loop allows the content to evolve as the user grows, supporting continuous skill development.

[0315] (Example 2)

[0316] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0317] In today's communication environment, users are expected to appropriately understand and improve their own statements and expressions. However, there is a lack of means for individual users to improve themselves in real time based on their emotional state and communication style in the moment. To solve this problem, a system is needed that can instantly analyze a user's emotional state and communication style and provide specific feedback and training methods based on that analysis.

[0318] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0319] In this invention, the server includes means for converting voice information into text data, means for analyzing the text information using language analysis technology to identify the communication method of each user, and means for determining the emotional state from the voice information and text information. This makes it possible to provide users with appropriate feedback and suggestions for training methods in real time.

[0320] "Audio information" refers to data obtained by converting audio signals into a digital format.

[0321] "Text data" refers to information in text format obtained after audio information has been converted.

[0322] "Linguistic analysis technology" refers to techniques for analyzing text information and interpreting its grammatical and contextual meaning.

[0323] "Communication method" refers to the style and techniques of communication used by each user.

[0324] "Response" refers to the feedback that the system generates for the user based on the analyzed information.

[0325] "Training methods" refer to programs proposed to users to improve their skills and communication abilities.

[0326] "Emotional state" refers to the internal emotions judged from the user's statements and voice.

[0327] "Activity logging" refers to the process of saving user interactions and actions as logs and tracking progress.

[0328] To implement this invention, it is important to use various engines that perform speech recognition, natural language processing, and emotion recognition. Specifically, the server first uses a "speech recognition engine" to convert the received speech information into text data. Here, a general cloud-based service that boasts high accuracy can be used as the speech recognition engine.

[0329] Next, the server analyzes this text data using a "language analysis engine." This technology performs grammatical analysis and keyword extraction to identify the user's communication method. Furthermore, it uses an "emotion recognition engine" to evaluate the user's emotional state from the audio information and text data. This process provides the foundational data for generating situation-appropriate feedback.

[0330] The server applies machine learning algorithms to this data to generate customized responses and training methods for the user. Common machine learning frameworks such as TensorFlow can be used, which can improve the accuracy of the feedback and training programs.

[0331] The device retrieves responses and training methods sent from the server and displays them in a user-friendly format. Through this information provided by the device, users can improve their communication skills. The device also provides an interface that records user activity and allows for progress tracking.

[0332] For example, if a user makes a negative comment during an online meeting, emotion recognition technology could identify that emotion, and the server could generate responses such as "how to receive positive feedback" or "how to offer constructive opinions." This would provide users with an efficient means to improve their communication style.

[0333] Furthermore, an example of a prompt to be input to a generative AI model is: "Design a system that analyzes user statements in real time in a communication environment and provides emotion-based feedback. Explain how to generate a training method that takes into account the emotional state of individual users." This prompt clearly communicates a specific objective and serves as a foundation for effectively utilizing the calculations of the generative AI model.

[0334] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0335] Step 1:

[0336] The server receives audio information obtained from meetings and online conferences. This audio information is input into the "speech recognition engine" and converted into text data. The converted text becomes the basis for the next process. Based on the input audio data, the server outputs the spoken content as text information.

[0337] Step 2:

[0338] The server inputs the generated text data into a "language analysis engine." This engine analyzes the grammatical structure of the text and identifies important keywords and the user's communication method. The analysis results obtained from the input text data include information about the intent and style of the utterance.

[0339] Step 3:

[0340] The server inputs text data and analysis results into the "emotion recognition engine." This engine evaluates the user's emotional state and determines what emotion was behind their statements. Based on the input of voice and text data, the server outputs the emotional state in real time, which is used to generate responses in the next step.

[0341] Step 4:

[0342] The server utilizes the previously analyzed results and uses machine learning algorithms to generate responses and training methods for the user. In this case, it creates content tailored to the user's unique communication style and emotional state. The server's input includes the analyzed data, and the output generates customized responses and training programs.

[0343] Step 5:

[0344] The terminal receives responses and training methods sent from the server and displays them to the user on an interface. The terminal provides information in an intuitive and easy-to-understand way, making it easy for the user to learn. Input to the terminal is data from the server, and output is feedback displayed to the user.

[0345] Step 6:

[0346] Users review the feedback provided by the device and take action to improve their communication skills. At this stage, they can take action and practice based on the feedback. The user's input is feedback, and the output is actual improvement actions.

[0347] (Application Example 2)

[0348] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0349] In modern family environments, improving individual communication skills is crucial, and there is a need to enhance the quality of communication within families. However, it is difficult for individuals to objectively understand and improve their own communication style and emotional state. Furthermore, there is a lack of means to receive regular feedback on their speaking style and appropriate learning programs. Therefore, there is a need for an effective system that can easily improve communication skills within the family.

[0350] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0351] In this invention, the server includes means for converting speech into linguistic data, means for analyzing the linguistic data using natural language processing technology to identify the information exchange style of each user, means for generating a customized response for the user based on the identified information exchange style, means for the home machine to analyze the user's daily conversation in real time and provide individualized feedback and improvement suggestions, and means for displaying the generated feedback and improvement suggestions in audible and visual form. This enables users in a home environment to objectively evaluate their own information exchange skills and receive individually optimized feedback and learning programs.

[0352] "Means of converting speech into linguistic data" refers to technologies that convert speech information spoken by a user into text data that can be understood by a machine.

[0353] "Natural language processing technology" is a technique that uses computers to analyze linguistic data and understand its meaning and context.

[0354] "Information exchange style" refers to the unique methods of communication and patterns that individual users express in conversations and written texts.

[0355] "Means for generating customized responses" refers to technologies that create feedback and instructions tailored to each user based on their individual characteristics and circumstances.

[0356] "A means by which home devices analyze users' everyday conversations in real time and provide personalized feedback and improvement suggestions" refers to a system in which devices used in the home instantly analyze users' conversations and suggest appropriate advice and improvement methods.

[0357] "Means of displaying generated feedback and improvement measures in audible or visual form" refers to a method of generating feedback and improvement suggestions from analysis results either by emitting sound or displaying them on a screen or similar device.

[0358] In the system implementing this invention, a server acts as the central point for receiving voice data and converting it into text data. This process uses the "Google Speech-to-Text API" as the speech recognition technology. This API has the function of converting the user's voice input into text data with high accuracy.

[0359] The converted text data is analyzed through a natural language processing (NLP) process. The Google Language API is used to analyze the meaning and context of the text data and identify the information exchange style. Based on this analysis, a customized response tailored to the user is generated. This process utilizes a pre-trained generative AI model using TensorFlow to construct an appropriate response.

[0360] Furthermore, home devices, or terminals, monitor the user's daily conversations in real time. Using tools such as "IBM Watson Tone Analyzer," the device analyzes the emotions expressed during conversations and provides feedback. The generated feedback and improvement suggestions are presented to the user in auditory and visual formats, enhancing the quality of information exchange within the home.

[0361] As a concrete example, consider a situation during a family conversation where the father seeks to communicate with his child in a less intimidating way. In this case, the device analyzes the father's statements in real time, offers suggestions to alleviate his emotions, and presents them via voice or on the screen. This facilitates smoother communication within the family.

[0362] An example prompt could be, "During a family meeting, analyze participants' speaking styles and emotions in real time and provide feedback to facilitate positive conversation." This prompt allows the system to provide specific analysis and feedback, supporting the user in improving their communication skills.

[0363] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0364] Step 1:

[0365] The server receives audio data transmitted from home appliances. The input is audio data, and the output is text data generated by a speech recognition engine. This audio data is converted into text data in a linguistic format using the "Google Speech-to-Text API". In this conversion process, the user's voice is recognized in real time to generate highly accurate text data.

[0366] Step 2:

[0367] The server receives the converted text data and performs natural language processing using the Google Language API. The input is the text data generated in step 1, and the output is the parsed content information. Here, the content and context of the text data are analyzed to identify the user's information exchange style. This process involves grammatical analysis and keyword extraction to understand the user's communication style.

[0368] Step 3:

[0369] The server generates customized feedback based on the analysis results. This process uses a generative AI model trained with TensorFlow. The input is the analysis results from step 2, and the output is the customized feedback data. The AI ​​model constructs appropriate responses based on past data, creating feedback and improvement suggestions tailored to the user.

[0370] Step 4:

[0371] The terminal presents the customized feedback received from the server to the user audibly or visually. The input is the feedback data generated in step 3, and the output is the format in which it is presented to the user. This process uses audio output and display functions to provide information in a way that is easy for the user to understand.

[0372] Step 5:

[0373] Based on the feedback provided, users begin taking action to improve their communication skills. The input is the feedback presented in step 4, and the output is the change in the user's behavior. Users implement the suggested improvement measures and training programs in their daily lives to enhance their skills.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0375] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0377] [Third Embodiment]

[0378] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0379] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0380] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0382] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0384] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0385] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0386] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0389] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0390] This invention is a system that utilizes speech recognition and natural language processing technologies to improve the communication skills of individual users. This system converts speech generated during meetings and online conferences into text data in real time, and then analyzes that data to identify each user's communication style. Based on the analysis results, it provides accurate feedback to the user and recommends an individualized training program.

[0391] Server: The server receives audio data streamed from the meeting and converts it into text data in real time. This conversion is performed using an advanced speech recognition engine, ensuring fast and accurate text conversion. The converted text data is passed to a natural language processing module, where grammar, sentiment, and tone analysis is performed. Based on the information collected through the analysis, a machine learning algorithm classifies and identifies the communication style. Based on the identified style, generative AI is used to create pinpoint feedback and generate a suitable training program.

[0392] Terminal: The terminal visualizes the feedback and training program provided by the server and displays it in a way that is easy for the user to understand. The terminal also implements support functions so that users can check the feedback and reflect it in their daily actions. Furthermore, it records the progress of the training program, contributing to future pacing.

[0393] User: Users improve their speaking style and expression by carefully reviewing and understanding the feedback received through their devices. They strive to improve their skills at their own pace by following the training program provided by the server. Progress in this program is recorded, allowing users to track their achievements in a compatible format and visually see which speaking points they have improved and which goals they have achieved.

[0394] Specific example: For instance, suppose a user's comments in a meeting don't align with those around them, resulting in a misunderstanding of their intended meaning. In this case, the system converts the comments into text data and performs sentiment analysis to analyze communication style, including tone and emphasis. Based on this, it provides the user with specific feedback for improvement, such as "adding clearer preambles before and after comments" and "inserting positive feedback as appropriate," and suggests related training such as internal workshops or peer review sessions. By repeating this process, the user's communication skills improve, and the system is designed to allow for reassessment.

[0395] The following describes the processing flow.

[0396] Step 1:

[0397] The server receives audio data streamed from the conferencing application and divides this data into small chunks in real time. The speech recognition engine then converts these divided audio chunks into text data.

[0398] Step 2:

[0399] The server passes the converted text data to a natural language processing module. Here, grammatical analysis, sentiment analysis, and tonal analysis are performed to extract the content and tone characteristics of the utterance.

[0400] Step 3:

[0401] The server uses NLP analysis results to apply machine learning algorithms and identify each user's communication style. This style includes factors such as frequency of speech, emotional intensity, and tone diversity.

[0402] Step 4:

[0403] Based on identified communication styles, the server utilizes generative AI to create personalized feedback for each user. This feedback includes specific areas for improvement and details of strengths.

[0404] Step 5:

[0405] The server generates individually optimized training programs based on feedback. These programs include specific training sessions and self-study suggestions.

[0406] Step 6:

[0407] The terminal receives feedback and training programs sent from the server. The terminal visualizes this information through a user interface, making it easy for the user to understand.

[0408] Step 7:

[0409] Users review the feedback content via their device and understand their own communication style. Following the provided training program, they apply the feedback to their daily activities.

[0410] Step 8:

[0411] The device records the user's progress through the training program. By accumulating progress data, users can visually track their skill improvement.

[0412] (Example 1)

[0413] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0414] In modern meetings and online conferences, many users face the problem of their communication skills not being effectively utilized, resulting in their intended message not being conveyed to others. Furthermore, it is usually difficult for users to accurately understand and improve their own communication patterns. Given this situation, there is a need to improve communication skills through the provision of appropriate and customized feedback and training programs tailored to each user.

[0415] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0416] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information using natural language processing technology to identify the communication patterns of individual users, and means for generating a customized response for the user based on the identified communication patterns. This enables users to understand their own communication style and take concrete actions to improve it.

[0417] "Auditory information" refers to data obtained through sound vibrations, and in particular, recordings of human speech.

[0418] "Text information" refers to data composed of electronically represented strings of characters or sentences.

[0419] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language.

[0420] A "communication pattern" is a distinctive style in which individual users communicate using their own unique verbal and nonverbal means.

[0421] A "response" is an automatic or planned reaction from a system to a specific situation or input.

[0422] A "learning program" is a planned series of activities and training aimed at improving a user's abilities and acquiring knowledge.

[0423] A "prompt statement" is an input statement used to guide a generative AI model to a specific output.

[0424] To implement this invention, the server, terminal, and user each provide system components that play specific roles.

[0425] server

[0426] The server receives "audio information" streamed from meetings and online conferences and converts it into "text information" in real time. A general-purpose cloud service is used as the speech recognition engine for this conversion. Specifically, a speech recognition API is utilized to convert "audio information" into "text information." The converted "text information" is then analyzed using "natural language processing technology." Natural language processing libraries (e.g., NLTK or spaCy) are used for the analysis to identify "communication patterns." After identification, a generative AI model is used to generate "responses" based on the identified patterns. "Prompt statements" are used in this process. Example of a prompt statement: Please suggest improvements if a user's speech is not understood in a meeting.

[0427] terminal

[0428] The terminal's role is to visualize the "responses" and "learning programs" provided by the server. The user interface is designed to be intuitively easy for users to understand. Specifically, it uses a web application platform (e.g., React or Vue.js) to display the "responses" and help users understand their own areas for improvement based on them. The terminal also records the progress of the "learning program," allowing users to visually check their own progress.

[0429] User

[0430] Users receive feedback from the server through their devices and take action to improve their own "communication patterns." For example, if a user makes a statement that is difficult to understand during a meeting, they can adjust their speaking style based on the "response" provided by the server. Furthermore, it is expected that they will gradually improve their skills by regularly conducting self-assessments through the progress management function provided by their devices.

[0431] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0432] Step 1:

[0433] The server receives audio information streamed in real time from conferences and meetings. The audio information is input via microphones and communication networks. The server converts this audio information into text using a speech recognition API. The converted text information is then output.

[0434] Step 2:

[0435] The server analyzes the obtained text information using natural language processing techniques. Specifically, it uses libraries such as NLTK and spaCy to analyze the grammatical structure, sentiment, and tone of the text. Based on this input text, it detects grammatical errors and extracts attributes such as positive / negative sentiment, outputting them as analysis results.

[0436] Step 3:

[0437] The server identifies user communication patterns based on the analysis results. This process uses machine learning algorithms to compare past and current data, extracting unique features. The analysis results are used as input, and the identified communication patterns are output.

[0438] Step 4:

[0439] The server utilizes a generative AI model to generate responses based on identified communication patterns. It uses prompt statements as input, instructing the generative AI with phrases like, "Please suggest improvements for situations where a user's message isn't understood in a meeting." This results in a customized response tailored to the user.

[0440] Step 5:

[0441] The terminal visualizes the responses received from the server and the learning program, and presents them to the user. It uses the response information from the server as input and displays the visualized response information as output via a web application. The user then uses this visual information to identify areas for improvement.

[0442] Step 6:

[0443] Based on the responses received on the device, users take action to improve their communication methods. They use the information displayed on the device screen to take specific improvement actions and reflect them in their daily communication.

[0444] (Application Example 1)

[0445] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] In modern society, improving individual communication skills is considered important. However, the means of providing personalized, real-time feedback to individual users in home and educational settings are limited. In particular, there is a growing need for systems that utilize speech recognition and natural language processing technologies, but such technologies are generally complex and expensive, posing a challenge to widespread adoption. This invention aims to solve these problems and provide affordable and effective support for improving communication.

[0447] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0448] In this invention, the server includes means for converting speech into digital data, means for analyzing the digital data using natural language processing technology to identify the individual user's communication style, and means for creating a response adapted to the user based on the identified communication style. This makes it possible for robots used in home and educational environments to provide real-time, effective, and low-cost feedback through interaction with users.

[0449] "Converting audio to digital data" is the process of converting audio signals into a text format that can be processed by a computer.

[0450] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0451] "Individual user communication styles" refer to the unique ways of speaking and expressing oneself that each user possesses.

[0452] "Generating a response" refers to the process of generating appropriate feedback or instructions for the user based on analyzed information.

[0453] "Interacting with users in a home or educational setting" refers to situations in which a robot communicates with an individual or a group.

[0454] "Providing real-time feedback" refers to a process of responding immediately to user actions and comments.

[0455] In the system that implements this application, the server uses a high-precision speech recognition engine to convert speech into digital data. This engine uses existing speech recognition technologies, such as the Google Speech-to-Text API, to quickly and accurately convert the input speech into text format. The converted text is then analyzed using natural language processing technologies, specifically Python's NLTK and spaCy libraries. This analysis reveals the individual user's communication style, and appropriate responses are generated based on the user's utterances and tone of voice.

[0456] The terminal displays real-time feedback from the server to the user. This could potentially utilize single-board computers such as Raspberry Pi or Jetson Nano, allowing for direct and easy information transmission to the user through a user interface.

[0457] Users can improve their communication skills in their daily lives by following the feedback and suggested guidance programs provided by the robot. Progress is continuously recorded and tracked in a visualized format, supporting users' self-improvement over the long term.

[0458] For example, if a child wants to improve their conversations with friends, the robot can analyze the child's speaking style at home and provide specific advice such as "softening your facial expressions when speaking" or "asking questions at the right time." Another example of a prompt when using a generative AI model would be: "What improvements are needed for the user to communicate more clearly and effectively? Please suggest specific advice and training menus."

[0459] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0460] Step 1:

[0461] The server receives audio data from the user. This input is audio streamed in real time, and the server converts this data into text using the Google Speech-to-Text API. The output is recognized text data. Specifically, certain keywords or phrases are picked from the audio signal and formatted into text.

[0462] Step 2:

[0463] The server analyzes the generated text data using natural language processing engines such as NLTK and spaCy. The input is the text data obtained in step 1. As a result of the analysis, information such as grammatical structure, sentiment analysis, and tone is extracted. The output is data about the user's communication style based on the analyzed information.

[0464] Step 3:

[0465] The server uses the data obtained in Step 2 to generate feedback for the user using a generative AI model. The input is analytical data, and the output is responses and advice adapted to the user's specific communication style. At this stage, specific feedback messages are generated, including suggestions for training programs.

[0466] Step 4:

[0467] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand format through the user interface. The input is the feedback data generated in step 3. The output is the information displayed on the screen that the user sees. Specifically, the feedback is displayed in text or audio format, making it easy for the user to immediately translate it into action.

[0468] Step 5:

[0469] Users improve their daily communication style based on feedback and training programs provided on the device. Input includes feedback heard on screen and via audio, as well as how the user puts it into practice. Output is the visible change as an improvement in the user's skills. In terms of concrete actions, users take specific actions based on the feedback and track their own progress.

[0470] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0471] This invention is a system developed to improve the communication skills of individual users by utilizing speech recognition technology, natural language processing technology, and emotion recognition technology. This system processes speech during meetings and online conferences into text data in real time, and by analyzing this text data, it can identify each user's communication style and emotional state. Based on the analysis results, it provides individually optimized feedback to the user and proposes a training program based on that feedback.

[0472] The server first receives the streaming audio data and converts it into text data in real time using a high-precision speech recognition engine. This text data is then analyzed using natural language processing technology to obtain the content and tone of speech, as well as the emotional state using an emotion engine. This emotion engine instantly determines the user's emotions based on the audio and text data and incorporates the result into the evaluation of the communication style. Based on this evaluation, a machine learning algorithm creates feedback optimized for each individual user and adjusts the content as needed. It also considers how specific emotional states may change advice and action guidelines for skill improvement when generating a training program.

[0473] The device receives feedback and training programs supplied from the server and displays them in an easy-to-understand format for the user. The feedback includes not only strengths and areas for improvement, but also specific, emotion-based approaches. The device provides interactive support tailored to the user's usage scenario, enabling users to apply the feedback to their daily activities. It also automatically records user behavior to monitor progress and help improve skills in the future.

[0474] Users review the displayed feedback to deepen their understanding of their own communication skills and emotional states. Based on this understanding, they work to improve their communication in their daily work. Furthermore, by implementing the provided training program and utilizing real-time feedback from the emotion engine to continuously improve their skills, they can learn effective communication methods that respond immediately to their own changes.

[0475] For example, if a user lacks confidence in expressing themselves, the emotional engine recognizes this anxiety, and the server uses this information to provide specific feedback such as "practices for speaking with confidence" and "emphasize positive feedback." Based on this feedback, the system also recommends effective ways to respond to positive feedback and mental techniques for gaining confidence as part of a training program. This invention promotes the improvement of users' self-awareness and skills through such methods.

[0476] The following describes the processing flow.

[0477] Step 1:

[0478] The server receives audio data streamed from the conferencing application. This audio data is converted into text data in real time using a speech recognition engine. The audio signal is divided into short chunks to improve accuracy.

[0479] Step 2:

[0480] The server passes the converted text data to a natural language processing module. This module analyzes the text data to determine the context, grammatical structure, and intent of the utterance, thereby identifying the overall meaning of the statement. Simultaneously, the emotion engine determines the user's emotional state from the audio and text.

[0481] Step 3:

[0482] Based on the analyzed communication style and emotions, the server uses machine learning algorithms to generate personalized feedback for each user. Emotional information is used to adjust the emphasis and suggestions in the feedback. This feedback includes identifying strengths and providing specific advice on areas for improvement.

[0483] Step 4:

[0484] Based on feedback, the server creates a training program optimized for the user. The training program suggests specific action plans and practice tasks to manage emotions that affect communication skills.

[0485] Step 5:

[0486] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand visual format. The user interface is designed to allow users to intuitively understand the feedback and apply it to their daily work.

[0487] Step 6:

[0488] Users review the feedback provided on their devices. Based on information about their communication style and emotional state, they understand areas for improvement in their daily communication and put those improvements into practice.

[0489] Step 7:

[0490] Users participate in a feedback-based training program. Program progress is tracked via a device, allowing users to monitor their own growth. The training includes managing emotional responses and learning effective communication techniques.

[0491] Step 8:

[0492] The server stores user training results and progress data, which is then used for feedback and to improve future training programs. This feedback loop allows the content to evolve as the user grows, supporting continuous skill development.

[0493] (Example 2)

[0494] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] In today's communication environment, users are expected to appropriately understand and improve their own statements and expressions. However, there is a lack of means for individual users to improve themselves in real time based on their emotional state and communication style in the moment. To solve this problem, a system is needed that can instantly analyze a user's emotional state and communication style and provide specific feedback and training methods based on that analysis.

[0496] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0497] In this invention, the server includes means for converting voice information into text data, means for analyzing the text information using language analysis technology to identify the communication method of each user, and means for determining the emotional state from the voice information and text information. This makes it possible to provide users with appropriate feedback and suggestions for training methods in real time.

[0498] "Audio information" refers to data obtained by converting audio signals into a digital format.

[0499] "Text data" refers to information in text format obtained after audio information has been converted.

[0500] "Linguistic analysis technology" refers to techniques for analyzing text information and interpreting its grammatical and contextual meaning.

[0501] "Communication method" refers to the style and techniques of communication used by each user.

[0502] "Response" refers to the feedback that the system generates for the user based on the analyzed information.

[0503] "Training methods" refer to programs proposed to users to improve their skills and communication abilities.

[0504] "Emotional state" refers to the internal emotions judged from the user's statements and voice.

[0505] "Activity logging" refers to the process of saving user interactions and actions as logs and tracking progress.

[0506] To implement this invention, it is important to use various engines that perform speech recognition, natural language processing, and emotion recognition. Specifically, the server first uses a "speech recognition engine" to convert the received speech information into text data. Here, a general cloud-based service that boasts high accuracy can be used as the speech recognition engine.

[0507] Next, the server analyzes this text data using a "language analysis engine." This technology performs grammatical analysis and keyword extraction to identify the user's communication method. Furthermore, it uses an "emotion recognition engine" to evaluate the user's emotional state from the audio information and text data. This process provides the foundational data for generating situation-appropriate feedback.

[0508] The server applies machine learning algorithms to this data to generate customized responses and training methods for the user. Common machine learning frameworks such as TensorFlow can be used, which can improve the accuracy of the feedback and training programs.

[0509] The device retrieves responses and training methods sent from the server and displays them in a user-friendly format. Through this information provided by the device, users can improve their communication skills. The device also provides an interface that records user activity and allows for progress tracking.

[0510] For example, if a user makes a negative comment during an online meeting, emotion recognition technology could identify that emotion, and the server could generate responses such as "how to receive positive feedback" or "how to offer constructive opinions." This would provide users with an efficient means to improve their communication style.

[0511] Furthermore, an example of a prompt to be input to a generative AI model is: "Design a system that analyzes user statements in real time in a communication environment and provides emotion-based feedback. Explain how to generate a training method that takes into account the emotional state of individual users." This prompt clearly communicates a specific objective and serves as a foundation for effectively utilizing the calculations of the generative AI model.

[0512] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0513] Step 1:

[0514] The server receives audio information obtained from meetings and online conferences. This audio information is input into the "speech recognition engine" and converted into text data. The converted text becomes the basis for the next process. Based on the input audio data, the server outputs the spoken content as text information.

[0515] Step 2:

[0516] The server inputs the generated text data into a "language analysis engine." This engine analyzes the grammatical structure of the text and identifies important keywords and the user's communication method. The analysis results obtained from the input text data include information about the intent and style of the utterance.

[0517] Step 3:

[0518] The server inputs text data and analysis results into the "emotion recognition engine." This engine evaluates the user's emotional state and determines what emotion was behind their statements. Based on the input of voice and text data, the server outputs the emotional state in real time, which is used to generate responses in the next step.

[0519] Step 4:

[0520] The server utilizes the previously analyzed results and uses machine learning algorithms to generate responses and training methods for the user. In this case, it creates content tailored to the user's unique communication style and emotional state. The server's input includes the analyzed data, and the output generates customized responses and training programs.

[0521] Step 5:

[0522] The terminal receives responses and training methods sent from the server and displays them to the user on an interface. The terminal provides information in an intuitive and easy-to-understand way, making it easy for the user to learn. Input to the terminal is data from the server, and output is feedback displayed to the user.

[0523] Step 6:

[0524] Users review the feedback provided by the device and take action to improve their communication skills. At this stage, they can take action and practice based on the feedback. The user's input is feedback, and the output is actual improvement actions.

[0525] (Application Example 2)

[0526] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] In modern family environments, improving individual communication skills is crucial, and there is a need to enhance the quality of communication within families. However, it is difficult for individuals to objectively understand and improve their own communication style and emotional state. Furthermore, there is a lack of means to receive regular feedback on their speaking style and appropriate learning programs. Therefore, there is a need for an effective system that can easily improve communication skills within the family.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0529] In this invention, the server includes means for converting speech into linguistic data, means for analyzing the linguistic data using natural language processing technology to identify the information exchange style of each user, means for generating a customized response for the user based on the identified information exchange style, means for the home machine to analyze the user's daily conversation in real time and provide individualized feedback and improvement suggestions, and means for displaying the generated feedback and improvement suggestions in audible and visual form. This enables users in a home environment to objectively evaluate their own information exchange skills and receive individually optimized feedback and learning programs.

[0530] "Means of converting speech into linguistic data" refers to technologies that convert speech information spoken by a user into text data that can be understood by a machine.

[0531] "Natural language processing technology" is a technique that uses computers to analyze linguistic data and understand its meaning and context.

[0532] "Information exchange style" refers to the unique methods of communication and patterns that individual users express in conversations and written texts.

[0533] "Means for generating customized responses" refers to technologies that create feedback and instructions tailored to each user based on their individual characteristics and circumstances.

[0534] "A means by which home devices analyze users' everyday conversations in real time and provide personalized feedback and improvement suggestions" refers to a system in which devices used in the home instantly analyze users' conversations and suggest appropriate advice and improvement methods.

[0535] "Means of displaying generated feedback and improvement measures in audible or visual form" refers to a method of generating feedback and improvement suggestions from analysis results either by emitting sound or displaying them on a screen or similar device.

[0536] In the system implementing this invention, a server acts as the central point for receiving voice data and converting it into text data. This process uses the "Google Speech-to-Text API" as the speech recognition technology. This API has the function of converting the user's voice input into text data with high accuracy.

[0537] The converted text data is analyzed through a natural language processing (NLP) process. The Google Language API is used to analyze the meaning and context of the text data and identify the information exchange style. Based on this analysis, a customized response tailored to the user is generated. This process utilizes a pre-trained generative AI model using TensorFlow to construct an appropriate response.

[0538] Furthermore, home devices, or terminals, monitor the user's daily conversations in real time. Using tools such as "IBM Watson Tone Analyzer," the device analyzes the emotions expressed during conversations and provides feedback. The generated feedback and improvement suggestions are presented to the user in auditory and visual formats, enhancing the quality of information exchange within the home.

[0539] As a concrete example, consider a situation during a family conversation where the father seeks to communicate with his child in a less intimidating way. In this case, the device analyzes the father's statements in real time, offers suggestions to alleviate his emotions, and presents them via voice or on the screen. This facilitates smoother communication within the family.

[0540] An example prompt could be, "During a family meeting, analyze participants' speaking styles and emotions in real time and provide feedback to facilitate positive conversation." This prompt allows the system to provide specific analysis and feedback, supporting the user in improving their communication skills.

[0541] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0542] Step 1:

[0543] The server receives audio data transmitted from home appliances. The input is audio data, and the output is text data generated by a speech recognition engine. This audio data is converted into text data in a linguistic format using the "Google Speech-to-Text API". In this conversion process, the user's voice is recognized in real time to generate highly accurate text data.

[0544] Step 2:

[0545] The server receives the converted text data and performs natural language processing using the Google Language API. The input is the text data generated in step 1, and the output is the parsed content information. Here, the content and context of the text data are analyzed to identify the user's information exchange style. This process involves grammatical analysis and keyword extraction to understand the user's communication style.

[0546] Step 3:

[0547] The server generates customized feedback based on the analysis results. This process uses a generative AI model trained with TensorFlow. The input is the analysis results from step 2, and the output is the customized feedback data. The AI ​​model constructs appropriate responses based on past data, creating feedback and improvement suggestions tailored to the user.

[0548] Step 4:

[0549] The terminal presents the customized feedback received from the server to the user audibly or visually. The input is the feedback data generated in step 3, and the output is the format in which it is presented to the user. This process uses audio output and display functions to provide information in a way that is easy for the user to understand.

[0550] Step 5:

[0551] Based on the feedback provided, users begin taking action to improve their communication skills. The input is the feedback presented in step 4, and the output is the change in the user's behavior. Users implement the suggested improvement measures and training programs in their daily lives to enhance their skills.

[0552] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0553] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0554] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0555] [Fourth Embodiment]

[0556] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0557] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0558] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0559] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0560] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0561] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0562] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0563] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0564] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0565] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0568] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0569] This invention is a system that utilizes speech recognition and natural language processing technologies to improve the communication skills of individual users. This system converts speech generated during meetings and online conferences into text data in real time, and then analyzes that data to identify each user's communication style. Based on the analysis results, it provides accurate feedback to the user and recommends an individualized training program.

[0570] Server: The server receives audio data streamed from the meeting and converts it into text data in real time. This conversion is performed using an advanced speech recognition engine, ensuring fast and accurate text conversion. The converted text data is passed to a natural language processing module, where grammar, sentiment, and tone analysis is performed. Based on the information collected through the analysis, a machine learning algorithm classifies and identifies the communication style. Based on the identified style, generative AI is used to create pinpoint feedback and generate a suitable training program.

[0571] Terminal: The terminal visualizes the feedback and training program provided by the server and displays it in a way that is easy for the user to understand. The terminal also implements support functions so that users can check the feedback and reflect it in their daily actions. Furthermore, it records the progress of the training program, contributing to future pacing.

[0572] User: Users improve their speaking style and expression by carefully reviewing and understanding the feedback received through their devices. They strive to improve their skills at their own pace by following the training program provided by the server. Progress in this program is recorded, allowing users to track their achievements in a compatible format and visually see which speaking points they have improved and which goals they have achieved.

[0573] Specific example: For instance, suppose a user's comments in a meeting don't align with those around them, resulting in a misunderstanding of their intended meaning. In this case, the system converts the comments into text data and performs sentiment analysis to analyze communication style, including tone and emphasis. Based on this, it provides the user with specific feedback for improvement, such as "adding clearer preambles before and after comments" and "inserting positive feedback as appropriate," and suggests related training such as internal workshops or peer review sessions. By repeating this process, the user's communication skills improve, and the system is designed to allow for reassessment.

[0574] The following describes the processing flow.

[0575] Step 1:

[0576] The server receives audio data streamed from the conferencing application and divides this data into small chunks in real time. The speech recognition engine then converts these divided audio chunks into text data.

[0577] Step 2:

[0578] The server passes the converted text data to a natural language processing module. Here, grammatical analysis, sentiment analysis, and tonal analysis are performed to extract the content and tone characteristics of the utterance.

[0579] Step 3:

[0580] The server uses NLP analysis results to apply machine learning algorithms and identify each user's communication style. This style includes factors such as frequency of speech, emotional intensity, and tone diversity.

[0581] Step 4:

[0582] Based on identified communication styles, the server utilizes generative AI to create personalized feedback for each user. This feedback includes specific areas for improvement and details of strengths.

[0583] Step 5:

[0584] The server generates individually optimized training programs based on feedback. These programs include specific training sessions and self-study suggestions.

[0585] Step 6:

[0586] The terminal receives feedback and training programs sent from the server. The terminal visualizes this information through a user interface, making it easy for the user to understand.

[0587] Step 7:

[0588] Users review the feedback content via their device and understand their own communication style. Following the provided training program, they apply the feedback to their daily activities.

[0589] Step 8:

[0590] The device records the user's progress through the training program. By accumulating progress data, users can visually track their skill improvement.

[0591] (Example 1)

[0592] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0593] In modern meetings and online conferences, many users face the problem of their communication skills not being effectively utilized, resulting in their intended message not being conveyed to others. Furthermore, it is usually difficult for users to accurately understand and improve their own communication patterns. Given this situation, there is a need to improve communication skills through the provision of appropriate and customized feedback and training programs tailored to each user.

[0594] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0595] In this invention, the server includes means for converting voice information into text information, means for analyzing the text information using natural language processing technology to identify the communication patterns of individual users, and means for generating a customized response for the user based on the identified communication patterns. This enables users to understand their own communication style and take concrete actions to improve it.

[0596] "Auditory information" refers to data obtained through sound vibrations, and in particular, recordings of human speech.

[0597] "Text information" refers to data composed of electronically represented strings of characters or sentences.

[0598] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language.

[0599] A "communication pattern" is a distinctive style in which individual users communicate using their own unique verbal and nonverbal means.

[0600] A "response" is an automatic or planned reaction from a system to a specific situation or input.

[0601] A "learning program" is a planned series of activities and training aimed at improving a user's abilities and acquiring knowledge.

[0602] A "prompt statement" is an input statement used to guide a generative AI model to a specific output.

[0603] To implement this invention, the server, terminal, and user each provide system components that play specific roles.

[0604] server

[0605] The server receives "audio information" streamed from meetings and online conferences and converts it into "text information" in real time. A general-purpose cloud service is used as the speech recognition engine for this conversion. Specifically, a speech recognition API is utilized to convert "audio information" into "text information." The converted "text information" is then analyzed using "natural language processing technology." Natural language processing libraries (e.g., NLTK or spaCy) are used for the analysis to identify "communication patterns." After identification, a generative AI model is used to generate "responses" based on the identified patterns. "Prompt statements" are used in this process. Example of a prompt statement: Please suggest improvements if a user's speech is not understood in a meeting.

[0606] terminal

[0607] The terminal's role is to visualize the "responses" and "learning programs" provided by the server. The user interface is designed to be intuitively easy for users to understand. Specifically, it uses a web application platform (e.g., React or Vue.js) to display the "responses" and help users understand their own areas for improvement based on them. The terminal also records the progress of the "learning program," allowing users to visually check their own progress.

[0608] User

[0609] Users receive feedback from the server through their devices and take action to improve their own "communication patterns." For example, if a user makes a statement that is difficult to understand during a meeting, they can adjust their speaking style based on the "response" provided by the server. Furthermore, it is expected that they will gradually improve their skills by regularly conducting self-assessments through the progress management function provided by their devices.

[0610] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0611] Step 1:

[0612] The server receives audio information streamed in real time from conferences and meetings. The audio information is input via microphones and communication networks. The server converts this audio information into text using a speech recognition API. The converted text information is then output.

[0613] Step 2:

[0614] The server analyzes the obtained text information using natural language processing techniques. Specifically, it uses libraries such as NLTK and spaCy to analyze the grammatical structure, sentiment, and tone of the text. Based on this input text, it detects grammatical errors and extracts attributes such as positive / negative sentiment, outputting them as analysis results.

[0615] Step 3:

[0616] The server identifies user communication patterns based on the analysis results. This process uses machine learning algorithms to compare past and current data, extracting unique features. The analysis results are used as input, and the identified communication patterns are output.

[0617] Step 4:

[0618] The server utilizes a generative AI model to generate responses based on identified communication patterns. It uses prompt statements as input, instructing the generative AI with phrases like, "Please suggest improvements for situations where a user's message isn't understood in a meeting." This results in a customized response tailored to the user.

[0619] Step 5:

[0620] The terminal visualizes the responses received from the server and the learning program, and presents them to the user. It uses the response information from the server as input and displays the visualized response information as output via a web application. The user then uses this visual information to identify areas for improvement.

[0621] Step 6:

[0622] Based on the responses received on the device, users take action to improve their communication methods. They use the information displayed on the device screen to take specific improvement actions and reflect them in their daily communication.

[0623] (Application Example 1)

[0624] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0625] In modern society, improving individual communication skills is considered important. However, the means of providing personalized, real-time feedback to individual users in home and educational settings are limited. In particular, there is a growing need for systems that utilize speech recognition and natural language processing technologies, but such technologies are generally complex and expensive, posing a challenge to widespread adoption. This invention aims to solve these problems and provide affordable and effective support for improving communication.

[0626] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0627] In this invention, the server includes means for converting speech into digital data, means for analyzing the digital data using natural language processing technology to identify the individual user's communication style, and means for creating a response adapted to the user based on the identified communication style. This makes it possible for robots used in home and educational environments to provide real-time, effective, and low-cost feedback through interaction with users.

[0628] "Converting audio to digital data" is the process of converting audio signals into a text format that can be processed by a computer.

[0629] "Natural language processing technology" is a technology that enables computers to understand and analyze human language.

[0630] "Individual user communication styles" refer to the unique ways of speaking and expressing oneself that each user possesses.

[0631] "Generating a response" refers to the process of generating appropriate feedback or instructions for the user based on analyzed information.

[0632] "Interacting with users in a home or educational setting" refers to situations in which a robot communicates with an individual or a group.

[0633] "Providing real-time feedback" refers to a process of responding immediately to user actions and comments.

[0634] In the system that implements this application, the server uses a high-precision speech recognition engine to convert speech into digital data. This engine uses existing speech recognition technologies, such as the Google Speech-to-Text API, to quickly and accurately convert the input speech into text format. The converted text is then analyzed using natural language processing technologies, specifically Python's NLTK and spaCy libraries. This analysis reveals the individual user's communication style, and appropriate responses are generated based on the user's utterances and tone of voice.

[0635] The terminal displays real-time feedback from the server to the user. This could potentially utilize single-board computers such as Raspberry Pi or Jetson Nano, allowing for direct and easy information transmission to the user through a user interface.

[0636] Users can improve their communication skills in their daily lives by following the feedback and suggested guidance programs provided by the robot. Progress is continuously recorded and tracked in a visualized format, supporting users' self-improvement over the long term.

[0637] For example, if a child wants to improve their conversations with friends, the robot can analyze the child's speaking style at home and provide specific advice such as "softening your facial expressions when speaking" or "asking questions at the right time." Another example of a prompt when using a generative AI model would be: "What improvements are needed for the user to communicate more clearly and effectively? Please suggest specific advice and training menus."

[0638] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0639] Step 1:

[0640] The server receives audio data from the user. This input is audio streamed in real time, and the server converts this data into text using the Google Speech-to-Text API. The output is recognized text data. Specifically, certain keywords or phrases are picked from the audio signal and formatted into text.

[0641] Step 2:

[0642] The server analyzes the generated text data using natural language processing engines such as NLTK and spaCy. The input is the text data obtained in step 1. As a result of the analysis, information such as grammatical structure, sentiment analysis, and tone is extracted. The output is data about the user's communication style based on the analyzed information.

[0643] Step 3:

[0644] The server uses the data obtained in Step 2 to generate feedback for the user using a generative AI model. The input is analytical data, and the output is responses and advice adapted to the user's specific communication style. At this stage, specific feedback messages are generated, including suggestions for training programs.

[0645] Step 4:

[0646] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand format through the user interface. The input is the feedback data generated in step 3. The output is the information displayed on the screen that the user sees. Specifically, the feedback is displayed in text or audio format, making it easy for the user to immediately translate it into action.

[0647] Step 5:

[0648] Users improve their daily communication style based on feedback and training programs provided on the device. Input includes feedback heard on screen and via audio, as well as how the user puts it into practice. Output is the visible change as an improvement in the user's skills. In terms of concrete actions, users take specific actions based on the feedback and track their own progress.

[0649] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0650] This invention is a system developed to improve the communication skills of individual users by utilizing speech recognition technology, natural language processing technology, and emotion recognition technology. This system processes speech during meetings and online conferences into text data in real time, and by analyzing this text data, it can identify each user's communication style and emotional state. Based on the analysis results, it provides individually optimized feedback to the user and proposes a training program based on that feedback.

[0651] The server first receives the streaming audio data and converts it into text data in real time using a high-precision speech recognition engine. This text data is then analyzed using natural language processing technology to obtain the content and tone of speech, as well as the emotional state using an emotion engine. This emotion engine instantly determines the user's emotions based on the audio and text data and incorporates the result into the evaluation of the communication style. Based on this evaluation, a machine learning algorithm creates feedback optimized for each individual user and adjusts the content as needed. It also considers how specific emotional states may change advice and action guidelines for skill improvement when generating a training program.

[0652] The device receives feedback and training programs supplied from the server and displays them in an easy-to-understand format for the user. The feedback includes not only strengths and areas for improvement, but also specific, emotion-based approaches. The device provides interactive support tailored to the user's usage scenario, enabling users to apply the feedback to their daily activities. It also automatically records user behavior to monitor progress and help improve skills in the future.

[0653] Users review the displayed feedback to deepen their understanding of their own communication skills and emotional states. Based on this understanding, they work to improve their communication in their daily work. Furthermore, by implementing the provided training program and utilizing real-time feedback from the emotion engine to continuously improve their skills, they can learn effective communication methods that respond immediately to their own changes.

[0654] For example, if a user lacks confidence in expressing themselves, the emotional engine recognizes this anxiety, and the server uses this information to provide specific feedback such as "practices for speaking with confidence" and "emphasize positive feedback." Based on this feedback, the system also recommends effective ways to respond to positive feedback and mental techniques for gaining confidence as part of a training program. This invention promotes the improvement of users' self-awareness and skills through such methods.

[0655] The following describes the processing flow.

[0656] Step 1:

[0657] The server receives audio data streamed from the conferencing application. This audio data is converted into text data in real time using a speech recognition engine. The audio signal is divided into short chunks to improve accuracy.

[0658] Step 2:

[0659] The server passes the converted text data to a natural language processing module. This module analyzes the text data to determine the context, grammatical structure, and intent of the utterance, thereby identifying the overall meaning of the statement. Simultaneously, the emotion engine determines the user's emotional state from the audio and text.

[0660] Step 3:

[0661] Based on the analyzed communication style and emotions, the server uses machine learning algorithms to generate personalized feedback for each user. Emotional information is used to adjust the emphasis and suggestions in the feedback. This feedback includes identifying strengths and providing specific advice on areas for improvement.

[0662] Step 4:

[0663] Based on feedback, the server creates a training program optimized for the user. The training program suggests specific action plans and practice tasks to manage emotions that affect communication skills.

[0664] Step 5:

[0665] The terminal receives feedback and training programs sent from the server and displays them in an easy-to-understand visual format. The user interface is designed to allow users to intuitively understand the feedback and apply it to their daily work.

[0666] Step 6:

[0667] Users review the feedback provided on their devices. Based on information about their communication style and emotional state, they understand areas for improvement in their daily communication and put those improvements into practice.

[0668] Step 7:

[0669] Users participate in a feedback-based training program. Program progress is tracked via a device, allowing users to monitor their own growth. The training includes managing emotional responses and learning effective communication techniques.

[0670] Step 8:

[0671] The server stores user training results and progress data, which is then used for feedback and to improve future training programs. This feedback loop allows the content to evolve as the user grows, supporting continuous skill development.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] In today's communication environment, users are expected to appropriately understand and improve their own statements and expressions. However, there is a lack of means for individual users to improve themselves in real time based on their emotional state and communication style in the moment. To solve this problem, a system is needed that can instantly analyze a user's emotional state and communication style and provide specific feedback and training methods based on that analysis.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0676] In this invention, the server includes means for converting voice information into text data, means for analyzing the text information using language analysis technology to identify the communication method of each user, and means for determining the emotional state from the voice information and text information. This makes it possible to provide users with appropriate feedback and suggestions for training methods in real time.

[0677] "Audio information" refers to data obtained by converting audio signals into a digital format.

[0678] "Text data" refers to information in text format obtained after audio information has been converted.

[0679] "Linguistic analysis technology" refers to techniques for analyzing text information and interpreting its grammatical and contextual meaning.

[0680] "Communication method" refers to the style and techniques of communication used by each user.

[0681] "Response" refers to the feedback that the system generates for the user based on the analyzed information.

[0682] "Training methods" refer to programs proposed to users to improve their skills and communication abilities.

[0683] "Emotional state" refers to the internal emotions judged from the user's statements and voice.

[0684] "Activity logging" refers to the process of saving user interactions and actions as logs and tracking progress.

[0685] To implement this invention, it is important to use various engines that perform speech recognition, natural language processing, and emotion recognition. Specifically, the server first uses a "speech recognition engine" to convert the received speech information into text data. Here, a general cloud-based service that boasts high accuracy can be used as the speech recognition engine.

[0686] Next, the server analyzes this text data using a "language analysis engine." This technology performs grammatical analysis and keyword extraction to identify the user's communication method. Furthermore, it uses an "emotion recognition engine" to evaluate the user's emotional state from the audio information and text data. This process provides the foundational data for generating situation-appropriate feedback.

[0687] The server applies machine learning algorithms to this data to generate customized responses and training methods for the user. Common machine learning frameworks such as TensorFlow can be used, which can improve the accuracy of the feedback and training programs.

[0688] The device retrieves responses and training methods sent from the server and displays them in a user-friendly format. Through this information provided by the device, users can improve their communication skills. The device also provides an interface that records user activity and allows for progress tracking.

[0689] For example, if a user makes a negative comment during an online meeting, emotion recognition technology could identify that emotion, and the server could generate responses such as "how to receive positive feedback" or "how to offer constructive opinions." This would provide users with an efficient means to improve their communication style.

[0690] Furthermore, an example of a prompt to be input to a generative AI model is: "Design a system that analyzes user statements in real time in a communication environment and provides emotion-based feedback. Explain how to generate a training method that takes into account the emotional state of individual users." This prompt clearly communicates a specific objective and serves as a foundation for effectively utilizing the calculations of the generative AI model.

[0691] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0692] Step 1:

[0693] The server receives audio information obtained from meetings and online conferences. This audio information is input into the "speech recognition engine" and converted into text data. The converted text becomes the basis for the next process. Based on the input audio data, the server outputs the spoken content as text information.

[0694] Step 2:

[0695] The server inputs the generated text data into a "language analysis engine." This engine analyzes the grammatical structure of the text and identifies important keywords and the user's communication method. The analysis results obtained from the input text data include information about the intent and style of the utterance.

[0696] Step 3:

[0697] The server inputs text data and analysis results into the "emotion recognition engine." This engine evaluates the user's emotional state and determines what emotion was behind their statements. Based on the input of voice and text data, the server outputs the emotional state in real time, which is used to generate responses in the next step.

[0698] Step 4:

[0699] The server utilizes the previously analyzed results and uses machine learning algorithms to generate responses and training methods for the user. In this case, it creates content tailored to the user's unique communication style and emotional state. The server's input includes the analyzed data, and the output generates customized responses and training programs.

[0700] Step 5:

[0701] The terminal receives responses and training methods sent from the server and displays them to the user on an interface. The terminal provides information in an intuitive and easy-to-understand way, making it easy for the user to learn. Input to the terminal is data from the server, and output is feedback displayed to the user.

[0702] Step 6:

[0703] Users review the feedback provided by the device and take action to improve their communication skills. At this stage, they can take action and practice based on the feedback. The user's input is feedback, and the output is actual improvement actions.

[0704] (Application Example 2)

[0705] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] In modern family environments, improving individual communication skills is crucial, and there is a need to enhance the quality of communication within families. However, it is difficult for individuals to objectively understand and improve their own communication style and emotional state. Furthermore, there is a lack of means to receive regular feedback on their speaking style and appropriate learning programs. Therefore, there is a need for an effective system that can easily improve communication skills within the family.

[0707] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0708] In this invention, the server includes means for converting speech into linguistic data, means for analyzing the linguistic data using natural language processing technology to identify the information exchange style of each user, means for generating a customized response for the user based on the identified information exchange style, means for the home machine to analyze the user's daily conversation in real time and provide individualized feedback and improvement suggestions, and means for displaying the generated feedback and improvement suggestions in audible and visual form. This enables users in a home environment to objectively evaluate their own information exchange skills and receive individually optimized feedback and learning programs.

[0709] "Means of converting speech into linguistic data" refers to technologies that convert speech information spoken by a user into text data that can be understood by a machine.

[0710] "Natural language processing technology" is a technique that uses computers to analyze linguistic data and understand its meaning and context.

[0711] "Information exchange style" refers to the unique methods of communication and patterns that individual users express in conversations and written texts.

[0712] "Means for generating customized responses" refers to technologies that create feedback and instructions tailored to each user based on their individual characteristics and circumstances.

[0713] "A means by which home devices analyze users' everyday conversations in real time and provide personalized feedback and improvement suggestions" refers to a system in which devices used in the home instantly analyze users' conversations and suggest appropriate advice and improvement methods.

[0714] "Means of displaying generated feedback and improvement measures in audible or visual form" refers to a method of generating feedback and improvement suggestions from analysis results either by emitting sound or displaying them on a screen or similar device.

[0715] In the system implementing this invention, a server acts as the central point for receiving voice data and converting it into text data. This process uses the "Google Speech-to-Text API" as the speech recognition technology. This API has the function of converting the user's voice input into text data with high accuracy.

[0716] The converted text data is analyzed through a natural language processing (NLP) process. The Google Language API is used to analyze the meaning and context of the text data and identify the information exchange style. Based on this analysis, a customized response tailored to the user is generated. This process utilizes a pre-trained generative AI model using TensorFlow to construct an appropriate response.

[0717] Furthermore, home devices, or terminals, monitor the user's daily conversations in real time. Using tools such as "IBM Watson Tone Analyzer," the device analyzes the emotions expressed during conversations and provides feedback. The generated feedback and improvement suggestions are presented to the user in auditory and visual formats, enhancing the quality of information exchange within the home.

[0718] As a concrete example, consider a situation during a family conversation where the father seeks to communicate with his child in a less intimidating way. In this case, the device analyzes the father's statements in real time, offers suggestions to alleviate his emotions, and presents them via voice or on the screen. This facilitates smoother communication within the family.

[0719] An example prompt could be, "During a family meeting, analyze participants' speaking styles and emotions in real time and provide feedback to facilitate positive conversation." This prompt allows the system to provide specific analysis and feedback, supporting the user in improving their communication skills.

[0720] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0721] Step 1:

[0722] The server receives audio data transmitted from home appliances. The input is audio data, and the output is text data generated by a speech recognition engine. This audio data is converted into text data in a linguistic format using the "Google Speech-to-Text API". In this conversion process, the user's voice is recognized in real time to generate highly accurate text data.

[0723] Step 2:

[0724] The server receives the converted text data and performs natural language processing using the Google Language API. The input is the text data generated in step 1, and the output is the parsed content information. Here, the content and context of the text data are analyzed to identify the user's information exchange style. This process involves grammatical analysis and keyword extraction to understand the user's communication style.

[0725] Step 3:

[0726] The server generates customized feedback based on the analysis results. This process uses a generative AI model trained with TensorFlow. The input is the analysis results from step 2, and the output is the customized feedback data. The AI ​​model constructs appropriate responses based on past data, creating feedback and improvement suggestions tailored to the user.

[0727] Step 4:

[0728] The terminal presents the customized feedback received from the server to the user audibly or visually. The input is the feedback data generated in step 3, and the output is the format in which it is presented to the user. This process uses audio output and display functions to provide information in a way that is easy for the user to understand.

[0729] Step 5:

[0730] Based on the feedback provided, users begin taking action to improve their communication skills. The input is the feedback presented in step 4, and the output is the change in the user's behavior. Users implement the suggested improvement measures and training programs in their daily lives to enhance their skills.

[0731] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0732] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0733] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0734] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0735] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0736] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0737] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0738] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0739] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0740] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0741] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0742] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0743] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0744] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0745] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0746] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0747] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0748] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0749] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0750] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0751] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0752] The following is further disclosed regarding the embodiments described above.

[0753] (Claim 1)

[0754] A means of converting speech into text data,

[0755] A means of analyzing text data using natural language processing technology to identify the communication style of individual users,

[0756] A means for generating customized feedback for the user based on identified communication styles,

[0757] A means of proposing a training program suitable for the user based on the generated feedback,

[0758] A system that includes this.

[0759] (Claim 2)

[0760] The system according to claim 1, characterized by analyzing the content and tone of the user's statements.

[0761] (Claim 3)

[0762] The system according to claim 1, characterized by having means for recording the progress of a training program and enabling users to track their own progress.

[0763] "Example 1"

[0764] (Claim 1)

[0765] A means of converting audio information into text information,

[0766] A means of analyzing text information using natural language processing technology to identify the communication patterns of individual users,

[0767] A means for generating a customized response for the user based on identified communication patterns,

[0768] A means of utilizing the generated response to recommend a learning program suitable for the user,

[0769] A means for generating a response using a prompt sentence based on the analysis results,

[0770] A system that includes this.

[0771] (Claim 2)

[0772] The system according to claim 1, characterized by analyzing the user's voice content and emotional tone.

[0773] (Claim 3)

[0774] The system according to claim 1, characterized by comprising means for recording the progress of a learning program and enabling the user to manage their own progress.

[0775] "Application Example 1"

[0776] (Claim 1)

[0777] A means of converting audio into digital data,

[0778] A means of analyzing digital data using natural language processing technology to identify the information transmission style of individual users,

[0779] A means for creating a user-adapted response based on an identified information transmission pattern,

[0780] A means of proposing the most suitable instructional program for the user based on the generated responses,

[0781] A means for robots to interact with users in a home or educational setting and provide real-time feedback,

[0782] A system that includes this.

[0783] (Claim 2)

[0784] The system according to claim 1, characterized by analyzing the content and tone of the user's statements and providing immediate feedback.

[0785] (Claim 3)

[0786] The system according to claim 1, characterized in that it includes means for recording the progress of the instruction program and enabling users to visualize their own improvement.

[0787] "Example 2 of combining an emotion engine"

[0788] (Claim 1)

[0789] A means of converting audio information into text data,

[0790] A means of analyzing textual information using language analysis technology to identify the communication method of each individual user,

[0791] Means for generating a customized response for the user based on an identified communication method,

[0792] A means of proposing a training method suitable for the user based on the generated response,

[0793] A means for determining emotional state from audio and text information,

[0794] A means of recording user activities and observing their progress,

[0795] A system that includes this.

[0796] (Claim 2)

[0797] The system according to claim 1, characterized by analyzing the user's voice expression and speaking style.

[0798] (Claim 3)

[0799] The system according to claim 1, characterized by comprising means for recording the progress of training methods and enabling users to track their own development.

[0800] "Application example 2 when combining with an emotional engine"

[0801] (Claim 1)

[0802] A means of converting audio into data in a language format,

[0803] A means of analyzing linguistic data using natural language processing technology to identify the information exchange style of individual users,

[0804] Means for generating a customized response for the user based on the identified information exchange style,

[0805] A means of proposing a learning program suitable for the user based on the generated response,

[0806] A means by which a home-use device analyzes the user's everyday conversations in real time and provides personalized feedback and improvement suggestions,

[0807] A means of displaying the generated feedback and improvement measures in the form of sound or visuals,

[0808] A system that includes this.

[0809] (Claim 2)

[0810] The system according to claim 1, characterized by analyzing the content of the user's statements and voice characteristics to promote a positive communication style during family meetings.

[0811] (Claim 3)

[0812] The system according to claim 1, characterized in that it includes means for recording the progress of a learning program and enabling the user to track their own improvement, and the home device monitors and provides the user's everyday conversation style. [Explanation of symbols]

[0813] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting speech into text data, A means of analyzing text data using natural language processing technology to identify the communication style of individual users, A means for generating customized feedback for the user based on identified communication styles, A means of proposing a training program suitable for the user based on the generated feedback, A system that includes this.

2. The system according to claim 1, characterized by analyzing the content and tone of the user's statements.

3. The system according to claim 1, characterized by comprising means for recording the progress of a training program and enabling users to track their own progress.

Images