System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving voice data of a user and converting the voice data into text data; means for analyzing an emotion of the user from the text data; means for providing appropriate feedback or advice when the emotion of the user changes based on the analyzed emotion; means for allowing the user to simulate a daily situation; means for forming a sympathy community with other users and matching users in the same situation; and means for connecting to a specialist or a medical institution as necessary.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, users suffer from a great deal of mental stress and feelings of isolation on a daily basis. Furthermore, lack of communication skills in business and everyday life often leads to high levels of stress and low self-esteem. However, conventional systems are unable to analyze users' emotions in real time and provide appropriate feedback, and there are insufficient environments for practicing communication skills through simulations. Furthermore, due to a lack of opportunities for users to connect with others in similar situations or to receive appropriate advice from experts, users' feelings of social isolation and mental burden are only increasing. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. This system includes means for receiving a user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, and means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions. The system also includes means for providing a function that allows users to simulate everyday situations, means for forming empathy communities with other users and matching users in similar situations, and means for connecting users to experts or medical institutions as needed. This allows users to receive real-time emotion analysis, improve their communication skills, reduce feelings of social isolation, and receive professional support.

[0006] "Voice Data" means audio information collected from a User's speech and stored in digital form.

[0007] "Text data" is voice data converted into text information, and is the data that forms the basis for natural language processing.

[0008] "Emotion analysis" refers to the process of identifying and assessing a user's emotional state (e.g., joy, sadness, anger, stress, etc.) from text data.

[0009] "Feedback" refers to advice or information provided based on a user's emotional state or behavior.

[0010] The "simulation function" is a function that provides an environment in which users can practice imagining real-life situations.

[0011] An "empathy community" is a platform where users in the same position or situation can connect with each other and support each other.

[0012] "Matching" is the process by which the system identifies and connects users with similar emotional states and needs.

[0013] "Means to connect with experts and medical institutions" refers to a function that refers users to appropriate experts and medical institutions to provide the specialized support or advice that users need.

[0014] "Converting to text in real time" refers to the process of instantly analyzing the user's voice and simultaneously converting it into text data.

[0015] "Natural language processing algorithms" are computer programs and methods for analyzing meaning and sentiment from text data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0038] Analyzing user sentiment and providing feedback

[0039] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert the voice data into text data. The server then analyzes the text data and applies a natural language processing algorithm to identify the user's emotional state. For example, if the user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice.

[0040] Improving skills through simulation of everyday situations

[0041] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0042] Building a community of empathy

[0043] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0044] Providing professional support

[0045] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0046] Specific examples

[0047] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0048] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] User voice input

[0052] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0053] Step 2:

[0054] Sending audio data

[0055] The device records the user's voice data and sends the data to the server.

[0056] Step 3:

[0057] Converting audio data to text

[0058] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[0059] Step 4:

[0060] Sentiment analysis of text data

[0061] The server uses natural language processing algorithms to analyze the user's emotional state from the text data, identifying emotions such as "fatigue" or "stress."

[0062] Step 5:

[0063] Feedback Generation

[0064] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[0065] Step 6:

[0066] Providing Feedback

[0067] The server provides the generated feedback to the user through the terminal.

[0068] Step 7:

[0069] Presenting a simulation scenario

[0070] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[0071] Step 8:

[0072] Collecting and sending simulated responses

[0073] The user responds according to the simulation, and the device records the user's voice response and sends it to the server.

[0074] Step 9:

[0075] Response text transformation and parsing

[0076] The server converts the simulation voice data into text data and analyzes the text data. For example, a response such as "I work for X Corporation" is analyzed.

[0077] Step 10:

[0078] Generate and provide improvement feedback

[0079] The server generates feedback on how to improve responses and speaking style, and provides it to the user via the device. For example, it may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[0080] Step 11:

[0081] Forming and matching communities of empathy

[0082] The server searches for and matches users with similar situations based on the user's emotional state and profile. For example, users who are suffering from work stress can be matched with each other.

[0083] Step 12:

[0084] Providing community features

[0085] The server provides a chat room or a place for video calls between successfully matched users, where users can share their experiences and feelings on the spot.

[0086] Step 13:

[0087] Collaboration with experts and medical institutions

[0088] The server provides a means to connect users to specialists or medical institutions based on their emotional state and concerns, as needed, for example by introducing them to a psychological counselor.

[0089] In this way, each step of the program concretely implements a series of steps, starting with user input and ending with analysis, feedback, simulation, community building, and professional support.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Conventional emotion analysis systems lack the ability to process users' voice data in real time or provide feedback through simulation functions, making it difficult for users to immediately identify areas for improvement and receive feedback. Furthermore, they lacked the functionality to match users with similar situations and form communities, making it difficult to prevent users from becoming isolated. Furthermore, it was difficult to appropriately refer users to experts or medical institutions when necessary, making it difficult for users to receive specialized support.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for recording and transmitting user voice data, means for converting the voice data into text data using a voice recognition engine, means for identifying the user's emotional state by applying a natural language processing algorithm, means for generating and providing feedback to the user based on the analysis results, and means for presenting a simulation scenario and collecting and analyzing the user's responses. This allows users to receive emotional analysis and feedback in real time, enabling them to improve their skills through simulations of everyday situations. In addition, by matching users in similar situations and forming empathetic communities, users can support each other and, by quickly connecting them to experts and medical institutions, make it easier for them to receive the support they need.

[0095] A "user" is an individual who uses the system to input voice and receive sentiment analysis and feedback.

[0096] A "terminal" is a device that allows a user to input voice, record the voice data, and transmit it to a server.

[0097] "Server" is a computer system that receives, analyzes, and provides feedback on voice data sent by users.

[0098] "Voice Data" means digital audio information uttered by a User and recorded by a Device.

[0099] "Text data" is character string information converted from voice data by a voice recognition engine.

[0100] A "voice recognition engine" is a software or hardware technology for converting voice data into text data.

[0101] A "natural language processing algorithm" is a computational method for analyzing text data and understanding its meaning and emotions.

[0102] "Sentiment analysis" is the process of identifying a user's emotional state from text data.

[0103] "Feedback" refers to advice and information generated based on the results of sentiment analysis.

[0104] A "simulation scenario" is a setting or situation in which users can virtually recreate everyday situations to practice and improve their skills.

[0105] An "empathy community" is a virtual group where users in the same position or situation interact and support each other.

[0106] "Matching" is the process of pairing users who share the same position or situation within a community of empathy.

[0107] An "expert" is a professional (e.g., a psychological counselor or medical institution staff) who provides the support that users need.

[0108] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0109] Analyzing user sentiment and providing feedback

[0110] This system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a voice recognition engine to convert this voice data into text data. Specifically, it uses the Google® Cloud Speech-to-Text API. The server then analyzes this text data and applies a natural language processing algorithm to identify the user's emotional state. Specifically, it uses IBM Watson® Natural Language Understanding. For example, if a user says, "Work has been tough and I'm tired lately," the server will derive "fatigue" or "stress" as the analysis result. Then, based on the analyzed emotions, the server provides appropriate feedback and advice to the user. For example, it may provide feedback such as, "Would you like to know more about how to relax?"

[0111] Improving skills through simulation of everyday situations

[0112] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0113] Building a community of empathy

[0114] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms or video calls. Specific software includes general-purpose software used for video calls and messaging applications used in chat rooms.

[0115] Providing professional support

[0116] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0117] Specific examples

[0118] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0119] Prompt Sentence Examples

[0120] "Your task will be to create a program that receives voice data from users, analyzes their emotions, and provides appropriate feedback. The speech recognition engine used will be the Google Cloud Speech-to-Text API, and the natural language processing engine will be IBM Watson Natural Language Understanding. Your task will also include the ability to present simulation scenarios, analyze responses, match users with empathetic communities, and provide professional support."

[0121] In this way, the system of the present invention can consistently provide users' sentiment analysis, simulation of everyday situations, building empathy communities, and professional support.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1:

[0124] The user speaks into the device. For example, if the user says, "Work has been hard lately and I'm tired," the device's microphone captures this voice. This voice data is temporarily stored on the device as a digital file. The input is the user's voice, and the output is a digital audio file.

[0125] Step 2:

[0126] The device sends the recorded audio data to the server using an HTTP Post request. The input is a digital audio file, and the output is an HTTP request sent to the server.

[0127] Step 3:

[0128] The server passes the received voice data to the voice recognition engine. Specifically, it converts the voice data into text data using the Google Cloud Speech-to-Text API. At this stage, the server sends the voice data to the API and receives the response as text data. The input is voice data, and the output is text data.

[0129] Step 4:

[0130] The server passes the generated text data to a natural language processing algorithm. Specifically, it uses IBM Watson Natural Language Understanding to perform sentiment analysis on the text data. Here, the input is the text data, and the output is the analyzed emotional state (e.g., "fatigue" or "stress").

[0131] Step 5:

[0132] The server generates appropriate feedback based on the analysis results. For example, if the result of the emotion analysis is "fatigue," the server generates advice on relaxation methods and stress management. At this stage, the server creates text-based feedback based on the emotion analysis results. The input is the analyzed emotional state, and the output is the feedback text.

[0133] Step 6:

[0134] The server sends the generated feedback to the device. Specifically, it returns the feedback text to the device as an HTTP response. The device can then display the received feedback to the user or give it to them as audio. The input is the feedback text, and the output is the display or audio output for the user.

[0135] Step 7:

[0136] When a user uses the simulation function, the terminal presents a simulation scenario. For example, if the user selects an interview simulation, the terminal presents questions based on the scenario. The input is the user's simulation scenario selection, and the output is the presented questions.

[0137] Step 8:

[0138] When the user answers questions in the simulation, the response is recorded by the device and sent to the server. The server receives this voice data, converts it into text using a speech recognition engine, and analyzes it using a natural language processing algorithm. The input is the user's response, and the output is the analyzed response.

[0139] Step 9:

[0140] The server generates feedback based on the content of the user's simulated responses and sends it to the device, including specific advice such as "You should speak more confidently." The input is the parsed response content, and the output is the feedback text.

[0141] Step 10:

[0142] The server searches for and matches other users based on the user's emotional state and profile information. This allows for the formation of empathetic communities. For example, users who suffer from work-related stress can be matched. The input is the user's emotional state and profile information, and the output is the matching results.

[0143] Step 11:

[0144] If necessary, the server provides a means to connect the user to an appropriate specialist or medical institution. For example, if psychological counseling is determined to be necessary, the server will refer the user to a counselor. The input is the emotion analysis result and the user's condition, and the output is the connection information for the specialist or medical institution.

[0145] In this way, the inventive system can seamlessly provide users' sentiment analysis, feedback provision, simulation, empathetic community building and professional support.

[0146] (Application example 1)

[0147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] Employees working at logistics centers are often exposed to mental stress and pressure in the course of their work, which has a negative impact on their work efficiency and health. Furthermore, due to a lack of communication between employees and limited opportunities to receive professional counseling and support, sufficient measures are not being taken to reduce stress and improve work efficiency. To solve these issues, a system is needed that can accurately analyze employees' emotional states and provide appropriate feedback and advice in real time.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0150] In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts or medical institutions as needed, and means for supporting the user's mental health and improving work efficiency, thereby making it possible to maintain the mental health of employees working at logistics centers and improve work efficiency.

[0151] "User" refers to any individual or employee who uses the System.

[0152] "Voice data" means electronic data that is a recording of what a user says.

[0153] "Text data" refers to data obtained by converting voice data into character information.

[0154] "Sentiment analysis" is a technique that applies natural language processing algorithms to text data to identify a user's emotional state.

[0155] "Feedback" refers to advice or comments provided to users based on the results of sentiment analysis.

[0156] "Everyday scene simulation" is a function that allows users to simulate real-life situations.

[0157] An "empathy community" is a community where users in the same position or circumstances can support each other.

[0158] "Matching" refers to connecting appropriate users within a community of like-minded people.

[0159] "Means for connecting with experts and medical institutions" refers to means for providing the necessary professional support based on the results of the user's sentiment analysis.

[0160] "Mental health" refers to a user's state of psychological and emotional well-being.

[0161] "Work efficiency" refers to the degree to which logistics center employees perform their work effectively and efficiently.

[0162] This invention relates to a system for supporting the mental health of logistics center employees and improving their work efficiency. Specifically, the system receives and analyzes users' voice data and provides appropriate feedback and advice. It also provides a simulation function, builds a community of empathy among employees, and connects them to experts and medical institutions as needed.

[0163] Hardware and Software

[0164] The system uses hardware such as a smartphone, microphone, and robot, as well as Python, the SpeechRecognition library, the transformers library, and Google's speech recognition API.

[0165] Data processing and calculation

[0166] First, when a user speaks to a smartphone or robot, the voice data is recorded. For example, suppose the user says, "Work has been tough lately and I'm feeling tired." An application on the smartphone receives this voice data and converts it into text data via Google's speech recognition API. Next, this text data undergoes sentiment analysis using a natural language processing algorithm. Specifically, emotional states such as "fatigue" and "stress" are identified using the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library.

[0167] Based on the results of this sentiment analysis, the server then provides appropriate feedback and advice to the user, such as "Take better care of yourself and take a short break when it comes to relaxation methods."

[0168] Specific examples

[0169] User A says into his smartphone, "Work has been tough lately and I'm feeling tired." The smartphone records this voice data and converts it into text data using Google's speech recognition API. This text data is sent to a server, where sentiment analysis is performed using the BERT model in the Transformers library. As a result of the analysis, "fatigue" and "stress" are identified, and feedback advice is provided, such as "Regarding relaxation methods, you should take better care of yourself. I recommend taking a short break."

[0170] Similarly, when a user uses the simulation function to practice for an interview, the server analyzes the user's responses to questions and provides specific advice in real time, such as, "Your voice is too quiet, so speak louder and with more confidence."

[0171] An example of a prompt is:

[0172] "Work has been hard lately and I'm tired."

[0173] "How can I speak more confidently when answering interview questions?"

[0174] Examples include:

[0175] In this way, the present invention provides a specific system for improving the mental health and work efficiency of employees in a distribution center.

[0176] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0177] Step 1:

[0178] The device (smartphone or robot) records the user's voice data. When the user speaks into the device, the device acquires and records this voice data. The input is the user's voice, and the output is voice data.

[0179] Step 2:

[0180] The device converts the recorded voice data into text data using Google's speech recognition API. Specifically, the device sends the voice data to the API and receives the result as text data. The input is voice data, and the output is the converted text data. It is then sent to the server.

[0181] Step 3:

[0182] The server performs sentiment analysis on the received text data. For this analysis, it uses the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library. The input is the text data, and the output is the result of sentiment analysis (e.g., "fatigue" or "stress").

[0183] Step 4:

[0184] The server generates appropriate feedback and advice based on the results of the sentiment analysis. This feedback is provided using a generative AI model. For example, specific advice such as "Take better care of yourself and take a short break when it comes to relaxation methods" is provided. The input is the result of the sentiment analysis, and the output is the feedback text.

[0185] Step 5:

[0186] The server sends the generated feedback to the terminal for providing to the user. The terminal provides the feedback to the user by displaying or reading it out loud. The input is the feedback sentence, and the output is what is presented to the user.

[0187] Step 6:

[0188] When a user wishes to simulate an everyday situation, the server selects a simulation scenario and sends it to the terminal. The user then performs the simulation, speaking and acting within it. The input is the simulation scenario, and the output is the user's response behavior data.

[0189] Step 7:

[0190] The server analyzes the user's simulated response data and provides feedback on how to improve performance. For example, it provides specific advice such as, "Your voice is too quiet. Speak louder and more confidently." The input is the simulated response data, and the output is feedback for improvement.

[0191] Step 8:

[0192] If necessary, the server connects users to empathy communities and matches them with users in similar situations, allowing users to exchange information and receive support from other users. The input is the user's emotional state and profile information, and the output is the matching results of empathy communities.

[0193] Step 9:

[0194] The server provides a means to connect the user to specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to an appropriate specialist so that they can receive support. The input is the emotion analysis results and the user's needs, and the output is connection information to specialists or medical institutions.

[0195] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0196] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[0197] Analyzing user sentiment and providing feedback

[0198] The system's operation begins with receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert this voice data into text data. The emotion engine then analyzes the text and voice data to identify the user's emotional state in detail. For example, if the user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server generates appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about ways to relax?"

[0199] Skill improvement through simulation functions

[0200] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" may be provided.

[0201] Building a community of empathy

[0202] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0203] Providing professional support

[0204] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[0205] Specific examples

[0206] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0207] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0208] The processing flow will be explained below.

[0209] Analyzing user sentiment and providing feedback

[0210] Step 1:

[0211] User voice input

[0212] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0213] Step 2:

[0214] Sending audio data

[0215] The device records the user's voice data and sends the data to the server.

[0216] Step 3:

[0217] Converting audio data to text

[0218] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[0219] Step 4:

[0220] Sentiment analysis of text data

[0221] The server uses an emotion engine to perform a detailed analysis of the user's emotional state from the text and voice data, identifying emotions such as "fatigue" and "stress."

[0222] Step 5:

[0223] Feedback Generation

[0224] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[0225] Step 6:

[0226] Providing Feedback

[0227] The server provides the generated feedback to the user through the terminal.

[0228] Skill improvement through simulation functions

[0229] Step 1:

[0230] Presenting a simulation scenario

[0231] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[0232] Step 2:

[0233] Collecting and sending simulated responses

[0234] The user responds verbally according to the scenario, and the device records the voice data and sends it to the server.

[0235] Step 3:

[0236] Response text transformation and parsing

[0237] The server converts the voice data of the scenario response into text data and analyzes it using the emotion engine. For example, a response such as "I work at X Corporation" is analyzed.

[0238] Step 4:

[0239] Generate and provide improvement feedback

[0240] The server generates feedback on how to improve the user's response and speaking style, and provides it to the user via the device. For example, the server may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[0241] Building a community of empathy

[0242] Step 1:

[0243] Check user profile and emotional state

[0244] The server collects and checks the user's profile information and sentiment analysis results.

[0245] Step 2:

[0246] Matching with sympathetic communities

[0247] The server searches for other users with the same situation or emotional state and matches them together, for example, matching users who are suffering from work stress.

[0248] Step 3:

[0249] Providing community features

[0250] The server provides matched users with chat rooms and video calls, where they can share their experiences and feelings.

[0251] Providing professional support

[0252] Step 1:

[0253] Connecting to experts based on sentiment analysis results

[0254] Based on the results of the emotion analysis, the server connects the user to the appropriate specialist or medical institution as needed.

[0255] Step 2:

[0256] Providing integrated support

[0257] The server provides a link to professional support (such as psychological counseling) according to the user's needs.

[0258] Specific examples

[0259] Step 1:

[0260] User A's voice input

[0261] User A says to the device, "Work has been tough lately and I'm tired."

[0262] Step 2:

[0263] Sending voice data and converting it to text

[0264] The terminal transmits the voice data to the server, which converts it into text data.

[0265] Step 3:

[0266] Sentiment Analysis and Feedback Generation

[0267] The server uses an emotion engine to analyze emotions such as "fatigue" and "stress" and provides feedback such as "Would you like to know more about ways to relax?"

[0268] Step 4:

[0269] Presenting a simulation scenario

[0270] When user A requests to practice for an interview, the device presents a simulation scenario that asks the user to "introduce themselves."

[0271] Step 5:

[0272] Collecting and analyzing responses and providing improvement feedback

[0273] User A responds, "I work for X Corporation," and the voice data is sent to the server for analysis. The server then generates and provides improvement feedback, such as, "Speak louder and more confidently."

[0274] Step 6:

[0275] Matching with sympathetic communities

[0276] The server matches User A with User B, who is also suffering from work-related stress, and provides a chat room.

[0277] Step 7:

[0278] Providing professional support

[0279] Based on the results of the emotion analysis, the server introduces user A to a psychological counselor.

[0280] In this way, the system starts with the user's voice data and concretely implements each step of analysis, feedback, simulation, community formation, and professional support.

[0281] Example 2

[0282] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] Conventional emotion analysis systems have limitations in their technology for analyzing emotions from user voice data, resulting in problems such as insufficient real-time emotion analysis or appropriate feedback. They also lack simulation functions to help users improve their everyday communication skills, or effective methods for matching users with other users in similar situations. Furthermore, they lack a means for smoothly connecting users with experts or medical institutions when needed. To solve these problems, a system is needed that can accurately analyze a user's emotional state in real time and provide appropriate feedback and advice based on the analysis results.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0285] In this invention, the server includes: means for receiving a user's voice data and converting it into text data; means for analyzing the user's emotions from the text data; means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions; means for allowing the user to simulate everyday situations; means for forming an empathy community with other users and matching users with similar situations; means for connecting to experts or medical institutions as needed; means for analyzing the user's emotional state in real time and dynamically generating feedback based on the analysis results; means for identifying a detailed emotional state using the user's speech data and providing specific advice based on the identified emotions; means for forming an empathy community that supports communication between users and promotes empathy; and means for simulating specific situations such as interviews and providing dynamic, real-time feedback on the user's responses. This enables the user's emotions to be analyzed with high accuracy and appropriate feedback to be provided in real time. Furthermore, it makes it easier for users to improve their everyday communication skills and share empathy with other users.

[0286] 1. "Voice Data" means data that is a digital recording of the voice spoken by a User.

[0287] 2. "Text data" means character string data converted from voice data using voice recognition technology.

[0288] 3. "Sentiment analysis" is the process of inferring and identifying a user's emotional state from input text or voice data.

[0289] 4. "Feedback" means advice or suggestions provided to the user based on the results of the analysis.

[0290] 5. "Simulation function" means a function that allows users to virtually experience specific scenes or situations and practice them.

[0291] 6. An "empathy community" is an online space where users in the same position or emotional state can share support and empathy with each other.

[0292] 7. "Matching" is the process of connecting users with other users who are in a similar state based on their emotional state and profile.

[0293] 8. "Means for connecting with specialists and medical institutions" means the means for referring and contacting users with appropriate specialists and medical institutions when necessary.

[0294] 9. "Real-time" refers to the ability to respond to user actions and inputs immediately, with immediate processing and results returned.

[0295] 10. "Means for generating dynamic feedback" refers to technology for generating appropriate feedback according to the user's emotional state and behavior.

[0296] 11. "Specific Advice" means detailed, actionable advice tailored to a user's specific situation and feelings.

[0297] 12. "Means to support communication" are technologies that promote and facilitate dialogue and interaction between users.

[0298] 13. "Simulating" situations such as "interviews" is a system feature that allows users to experience and practice hypothetical interview situations.

[0299] 14. "Means for providing dynamic, real-time feedback to users' responses" refers to technology that provides immediate feedback to users based on their responses and actions.

[0300] MODE FOR CARRYING OUT THE INVENTION

[0301] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[0302] Analyzing user sentiment and providing feedback

[0303] The system begins by receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data. An emotion engine (e.g., IBM Watson Tone Analyzer API) then analyzes the text and voice data to identify the user's emotional state in detail.

[0304] For example, if a user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server will generate appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about how to relax?"

[0305] Skill improvement through simulation functions

[0306] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server, where it is converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses.

[0307] For example, specific advice is provided such as, "Your voice is too quiet, so speak louder and with more confidence."

[0308] Building a community of empathy

[0309] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0310] Providing professional support

[0311] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[0312] Specific examples

[0313] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0314] Example of input prompt for generative AI model

[0315] Prompt: "Please explain sentiment analysis and feedback systems. Please provide examples."

[0316] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0317] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0318] Step 1:

[0319] User utterances and voice data collection

[0320] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0321] The device records the user's speech using a built-in microphone. The voice data (input) is temporarily stored in the device's memory in digital format (output).

[0322] Step 2:

[0323] Sending audio data

[0324] The device encrypts the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0325] The voice data (input) is encrypted and sent to the server (output).

[0326] Step 3:

[0327] Converting audio data to text

[0328] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data.

[0329] Audio data (input) is converted into text data (output).

[0330] Step 4:

[0331] Emotion analysis

[0332] The server uses an emotion engine (specifically, IBM Watson Tone Analyzer API) to perform emotion analysis of text data and voice data.

[0333] From the text data and voice data (input), the user's emotional state (e.g., fatigue, stress) is analyzed (output).

[0334] The server stores the analysis results in an internal database, including the user ID, analysis results, and timestamp.

[0335] Step 5:

[0336] Generate feedback

[0337] The server generates appropriate feedback and advice for the user based on the results of the emotion analysis.

[0338] Based on the analysis results (input), a specific feedback message (e.g., "Would you like to know more about how to relax?") is generated (output).

[0339] Step 6:

[0340] Providing Feedback

[0341] The server sends the generated feedback message to the terminal.

[0342] Feedback messages (input) are forwarded to the terminal (output).

[0343] Step 7:

[0344] View Feedback

[0345] The device displays the received feedback message to the user.

[0346] Feedback messages received on the device (input) are displayed on the screen as popups or notifications (output).

[0347] Step 8:

[0348] Presenting a simulation scenario

[0349] The server generates a simulation scenario such as an interview and transmits it to the terminal.

[0350] The simulation scenario (input) arrives at the terminal and is displayed to the user (output).

[0351] Step 9:

[0352] Collect user responses

[0353] When the user speaks to the simulation, the terminal records the voice data and transmits it back to the server.

[0354] The user's response voice data (input) is sent to the server and converted into text data (output).

[0355] Step 10:

[0356] Analyzing simulation responses and providing feedback

[0357] The server converts the voice data into text data and then analyzes it using an emotion engine.

[0358] It analyzes text data (input) and generates specific feedback (e.g., "You should speak more confidently") (output).

[0359] The server sends a feedback message to the terminal, which displays it to the user.

[0360] Step 11:

[0361] Building a community of empathy

[0362] The server searches for and matches users with other users in similar states based on their emotional state and profile information.

[0363] Matched user information (output) is generated from user information and sentiment analysis results (input).

[0364] Step 12:

[0365] Communication Support

[0366] The server provides an environment where matched users can share experiences through chat rooms and video calls.

[0367] The means of communication between users (links and connection information) (output) are displayed on the terminal.

[0368] Step 13:

[0369] Providing access to specialists and medical institutions

[0370] Based on the analysis results, the server provides a means to connect users to appropriate specialists or medical institutions as needed.

[0371] The analysis results (input) are used to provide contact information and appointment links for specialists and medical institutions (output).

[0372] In this way, this invention not only analyzes a user's emotions in real time using their voice data and provides feedback and advice based on the results, but also realizes a seamless system that includes simulation functions, an empathetic community, and even professional support.

[0373] This allows users to understand their emotional state in three dimensions, utilize feedback to improve their ability to respond to each situation, and access empathetic communities and professional support to solve problems and grow without feeling isolated.

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] While conventional emotion analysis systems can analyze users' emotions in real time, they lack mechanisms for providing appropriate relaxation content or expert support based on the results. As a result, they lack specific approaches to improving users' mental health. Furthermore, they lack the ability to form empathetic communities where users can support each other, and the ability to support users in improving their skills through simulation functions.

[0377] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts and support organizations as needed, and means for providing appropriate relaxation content based on the results of the user's emotion analysis. As a result, individually optimized relaxation content is provided based on the results of the user's emotion analysis, enabling users to support each other through the empathy community and receive support from experts.

[0378] "Voice data" refers to data that is a digital recording of what a user has said.

[0379] "Text data" refers to data obtained by converting voice data into character information.

[0380] "Sentiment analysis" is the process of analyzing a user's emotional state from their text and voice data.

[0381] "Feedback" refers to advice and information provided based on the results of user sentiment analysis.

[0382] The "simulation function" allows users to virtually recreate real-world situations for practice and experience.

[0383] An "empathy community" is an online community where users with similar positions and experiences can support each other.

[0384] "Matching" is the process of connecting users with the same situations or concerns within a community of empathy.

[0385] An "expert" is a professional with specific expertise, such as a psychological counselor or healthcare provider.

[0386] "Support organizations" are organizations that provide assistance, including medical institutions and specialized counseling centers.

[0387] "Relaxation content" refers to digital content such as music, meditation guides, and stories that are designed to reduce stress and promote relaxation for users.

[0388] This invention is a system that analyzes users' emotions in real time, provides personalized relaxation content based on the analysis results, creates a community of empathy, and connects them to experts and support organizations as needed.

[0389] Audio data collection and text conversion

[0390] The server receives the voice data spoken by the user into the device. The device records this voice data and sends it to the server. The server uses a speech recognition engine and the "speech_recognition" library to convert this voice data into text data. This library converts voice to text with high accuracy.

[0391] Emotion analysis

[0392] The server then analyzes the user's emotions using the converted text and audio data. This process uses a generative AI model with the "transformers" library to output an emotion label (e.g., "happiness," "sadness," "stress," etc.) and its score.

[0393] Providing personalized content

[0394] Based on the results of the emotion analysis, the server provides relaxation content appropriate for the user. For example, if the user is analyzed as feeling stressed, the server will provide content such as relaxation music or meditation guides. This selection is based on the emotion label and score.

[0395] Forming and matching communities of empathy

[0396] The server searches for and matches users with other users in similar situations based on their emotional state and profile information, creating a mutual support community that encourages users to share their experiences through chat rooms and video calls.

[0397] Providing professional support

[0398] If necessary, the server also provides a function to connect the user to experts or support organizations. For example, if the results of the emotion analysis indicate that psychological counseling is necessary, the user will be referred to an appropriate expert and receive appropriate support.

[0399] Specific examples

[0400] For example, if a user says, "I've been feeling a lot of stress at work lately," the device sends this voice data to the server. The server converts the voice data into text and then performs emotion analysis using the "transformers" library, calculating the emotion label "stress" and a high score. The server then provides the user with relaxation music and deep breathing guides, and attempts to match them with other users in similar situations. If necessary, it also provides guidance to professional counseling.

[0401] Prompt Sentence Examples

[0402] For example, the following prompts can be used to recommend relaxation music based on the user's emotional state:

[0403] User's emotional status: Stressed (score: 0.85)

[0404] Suitable content for users: Deep breathing guides, Stress relief stories

[0405] In this way, we have created a system that provides relaxation content that is individually optimized based on the user's emotional state, enables users to support each other through an empathetic community, and enables them to receive support from experts.

[0406] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0407] Step 1: Collecting audio data

[0408] The user speaks emotional content into their device, and this voice data is recorded through the device's microphone. The input is the user's speech, and the output is digital voice data.

[0409] Step 2: Sending audio data to the server

[0410] The device sends the recorded voice data to the server. The input is the digital voice data, and the output is the voice data transferred to the server.

[0411] Step 3: Convert audio data to text

[0412] The server converts the received voice data into text data using the "speech_recognition" library. The input is the voice data sent to the server, and the output is text data. This conversion process is performed using voice recognition technology.

[0413] Step 4: Sentiment Analysis

[0414] The server uses generative AI models from the "transformers" library to analyze user emotions from text and audio data. The input is text data, and the output is emotion labels and their scores. In this process, natural language processing algorithms identify emotional states.

[0415] Step 5: Generate feedback

[0416] The server selects relaxation content appropriate for the user based on the emotion analysis results. For example, if an emotion label indicating stress is detected, content such as relaxation music or meditation guides will be selected. The input is the emotion label and its score, and the output is a list of appropriate relaxation content.

[0417] Step 6: Providing content

[0418] The server provides selected relaxation content to users. The input is a list of relaxation content, and the output is links and data of the content that users can access.

[0419] Step 7: Matching with empathetic communities

[0420] The server matches users with other users based on their emotional state and profile information. The input is the user's emotional state and profile information, and the output is the matching results. This process searches for users in similar situations and forms a community of empathy.

[0421] Step 8: Providing professional support

[0422] If necessary, the server will refer the user to a specialist or support organization based on the results of the emotion analysis. The input is the emotion analysis result and the user's health status information, and the output is information on the referral to the appropriate specialist or support organization. This process provides a path for the user to smoothly receive support.

[0423] Through these steps, a system is realized that provides individually optimized relaxation content based on the results of user emotion analysis, enables users to support each other through empathetic communities, and allows them to receive support from experts.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0425] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0427] [Second embodiment]

[0428] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0429] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0430] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0432] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0434] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0435] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0436] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0439] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0440] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0441] Analyzing user sentiment and providing feedback

[0442] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert the voice data into text data. The server then analyzes the text data and applies a natural language processing algorithm to identify the user's emotional state. For example, if the user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice.

[0443] Improving skills through simulation of everyday situations

[0444] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0445] Building a community of empathy

[0446] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0447] Providing professional support

[0448] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0449] Specific examples

[0450] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0451] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0452] The processing flow will be explained below.

[0453] Step 1:

[0454] User voice input

[0455] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0456] Step 2:

[0457] Sending audio data

[0458] The device records the user's voice data and sends the data to the server.

[0459] Step 3:

[0460] Converting audio data to text

[0461] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[0462] Step 4:

[0463] Sentiment analysis of text data

[0464] The server uses natural language processing algorithms to analyze the user's emotional state from the text data, identifying emotions such as "fatigue" or "stress."

[0465] Step 5:

[0466] Feedback Generation

[0467] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[0468] Step 6:

[0469] Providing Feedback

[0470] The server provides the generated feedback to the user through the terminal.

[0471] Step 7:

[0472] Presenting a simulation scenario

[0473] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[0474] Step 8:

[0475] Collecting and sending simulated responses

[0476] The user responds according to the simulation, and the device records the user's voice response and sends it to the server.

[0477] Step 9:

[0478] Response text transformation and parsing

[0479] The server converts the simulation voice data into text data and analyzes the text data. For example, a response such as "I work for X Corporation" is analyzed.

[0480] Step 10:

[0481] Generate and provide improvement feedback

[0482] The server generates feedback on how to improve responses and speaking style, and provides it to the user via the device. For example, it may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[0483] Step 11:

[0484] Forming and matching communities of empathy

[0485] The server searches for and matches users with similar situations based on the user's emotional state and profile. For example, users who are suffering from work stress can be matched with each other.

[0486] Step 12:

[0487] Providing community features

[0488] The server provides a chat room or a place for video calls between successfully matched users, where users can share their experiences and feelings on the spot.

[0489] Step 13:

[0490] Collaboration with experts and medical institutions

[0491] The server provides a means to connect users to specialists or medical institutions based on their emotional state and concerns, as needed, for example by introducing them to a psychological counselor.

[0492] In this way, each step of the program concretely implements a series of steps, starting with user input and ending with analysis, feedback, simulation, community building, and professional support.

[0493] Example 1

[0494] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] Conventional emotion analysis systems lack the ability to process users' voice data in real time or provide feedback through simulation functions, making it difficult for users to immediately identify areas for improvement and receive feedback. Furthermore, they lacked the functionality to match users with similar situations and form communities, making it difficult to prevent users from becoming isolated. Furthermore, it was difficult to appropriately refer users to experts or medical institutions when necessary, making it difficult for users to receive specialized support.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0497] In this invention, the server includes means for recording and transmitting user voice data, means for converting the voice data into text data using a voice recognition engine, means for identifying the user's emotional state by applying a natural language processing algorithm, means for generating and providing feedback to the user based on the analysis results, and means for presenting a simulation scenario and collecting and analyzing the user's responses. This allows users to receive emotional analysis and feedback in real time, enabling them to improve their skills through simulations of everyday situations. In addition, by matching users in similar situations and forming empathetic communities, users can support each other and, by quickly connecting them to experts and medical institutions, make it easier for them to receive the support they need.

[0498] A "user" is an individual who uses the system to input voice and receive sentiment analysis and feedback.

[0499] A "terminal" is a device that allows a user to input voice, record the voice data, and transmit it to a server.

[0500] "Server" is a computer system that receives, analyzes, and provides feedback on voice data sent by users.

[0501] "Voice Data" means digital audio information uttered by a User and recorded by a Device.

[0502] "Text data" is character string information converted from voice data by a voice recognition engine.

[0503] A "voice recognition engine" is a software or hardware technology for converting voice data into text data.

[0504] A "natural language processing algorithm" is a computational method for analyzing text data and understanding its meaning and emotions.

[0505] "Sentiment analysis" is the process of identifying a user's emotional state from text data.

[0506] "Feedback" refers to advice and information generated based on the results of sentiment analysis.

[0507] A "simulation scenario" is a setting or situation in which users can virtually recreate everyday situations to practice and improve their skills.

[0508] An "empathy community" is a virtual group where users in the same position or situation interact and support each other.

[0509] "Matching" is the process of pairing users who share the same position or situation within a community of empathy.

[0510] An "expert" is a professional (e.g., a psychological counselor or medical institution staff) who provides the support that users need.

[0511] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0512] Analyzing user sentiment and providing feedback

[0513] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a voice recognition engine to convert this voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API. The server then analyzes this text data and applies a natural language processing algorithm to identify the user's emotional state. Specifically, it uses IBM Watson Natural Language Understanding. For example, if a user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice. For example, it may provide feedback such as, "Would you like to know more about how to relax?"

[0514] Improving skills through simulation of everyday situations

[0515] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0516] Building a community of empathy

[0517] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms or video calls. Specific software includes general-purpose software used for video calls and messaging applications used in chat rooms.

[0518] Providing professional support

[0519] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0520] Specific examples

[0521] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0522] Prompt Sentence Examples

[0523] "Your task will be to create a program that receives voice data from users, analyzes their emotions, and provides appropriate feedback. The speech recognition engine used will be the Google Cloud Speech-to-Text API, and the natural language processing engine will be IBM Watson Natural Language Understanding. Your task will also include the ability to present simulation scenarios, analyze responses, match users with empathetic communities, and provide professional support."

[0524] In this way, the system of the present invention can consistently provide users' sentiment analysis, simulation of everyday situations, building empathy communities, and professional support.

[0525] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0526] Step 1:

[0527] The user speaks into the device. For example, if the user says, "Work has been hard lately and I'm tired," the device's microphone captures this voice. This voice data is temporarily stored on the device as a digital file. The input is the user's voice, and the output is a digital audio file.

[0528] Step 2:

[0529] The device sends the recorded audio data to the server using an HTTP Post request. The input is a digital audio file, and the output is an HTTP request sent to the server.

[0530] Step 3:

[0531] The server passes the received voice data to the voice recognition engine. Specifically, it converts the voice data into text data using the Google Cloud Speech-to-Text API. At this stage, the server sends the voice data to the API and receives the response as text data. The input is voice data, and the output is text data.

[0532] Step 4:

[0533] The server passes the generated text data to a natural language processing algorithm. Specifically, it uses IBM Watson Natural Language Understanding to perform sentiment analysis on the text data. Here, the input is the text data, and the output is the analyzed emotional state (e.g., "fatigue" or "stress").

[0534] Step 5:

[0535] The server generates appropriate feedback based on the analysis results. For example, if the result of the emotion analysis is "fatigue," the server generates advice on relaxation methods and stress management. At this stage, the server creates text-based feedback based on the emotion analysis results. The input is the analyzed emotional state, and the output is the feedback text.

[0536] Step 6:

[0537] The server sends the generated feedback to the device. Specifically, it returns the feedback text to the device as an HTTP response. The device can then display the received feedback to the user or give it to them as audio. The input is the feedback text, and the output is the display or audio output for the user.

[0538] Step 7:

[0539] When a user uses the simulation function, the terminal presents a simulation scenario. For example, if the user selects an interview simulation, the terminal presents questions based on the scenario. The input is the user's simulation scenario selection, and the output is the presented questions.

[0540] Step 8:

[0541] When the user answers questions in the simulation, the response is recorded by the device and sent to the server. The server receives this voice data, converts it into text using a speech recognition engine, and analyzes it using a natural language processing algorithm. The input is the user's response, and the output is the analyzed response.

[0542] Step 9:

[0543] The server generates feedback based on the content of the user's simulated responses and sends it to the device, including specific advice such as "You should speak more confidently." The input is the parsed response content, and the output is the feedback text.

[0544] Step 10:

[0545] The server searches for and matches other users based on the user's emotional state and profile information. This allows for the formation of empathetic communities. For example, users who suffer from work-related stress can be matched. The input is the user's emotional state and profile information, and the output is the matching results.

[0546] Step 11:

[0547] If necessary, the server provides a means to connect the user to an appropriate specialist or medical institution. For example, if psychological counseling is determined to be necessary, the server will refer the user to a counselor. The input is the emotion analysis result and the user's condition, and the output is the connection information for the specialist or medical institution.

[0548] In this way, the inventive system can seamlessly provide users' sentiment analysis, feedback provision, simulation, empathetic community building and professional support.

[0549] (Application example 1)

[0550] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0551] Employees working at logistics centers are often exposed to mental stress and pressure in the course of their work, which has a negative impact on their work efficiency and health. Furthermore, due to a lack of communication between employees and limited opportunities to receive professional counseling and support, sufficient measures are not being taken to reduce stress and improve work efficiency. To solve these issues, a system is needed that can accurately analyze employees' emotional states and provide appropriate feedback and advice in real time.

[0552] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0553] In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts or medical institutions as needed, and means for supporting the user's mental health and improving work efficiency, thereby making it possible to maintain the mental health of employees working at logistics centers and improve work efficiency.

[0554] "User" refers to any individual or employee who uses the System.

[0555] "Voice data" means electronic data that is a recording of what a user says.

[0556] "Text data" refers to data obtained by converting voice data into character information.

[0557] "Sentiment analysis" is a technique that applies natural language processing algorithms to text data to identify a user's emotional state.

[0558] "Feedback" refers to advice or comments provided to users based on the results of sentiment analysis.

[0559] "Everyday scene simulation" is a function that allows users to simulate real-life situations.

[0560] An "empathy community" is a community where users in the same position or circumstances can support each other.

[0561] "Matching" refers to connecting appropriate users within a community of like-minded people.

[0562] "Means for connecting with experts and medical institutions" refers to means for providing the necessary professional support based on the results of the user's sentiment analysis.

[0563] "Mental health" refers to a user's state of psychological and emotional well-being.

[0564] "Work efficiency" refers to the degree to which logistics center employees perform their work effectively and efficiently.

[0565] This invention relates to a system for supporting the mental health of logistics center employees and improving their work efficiency. Specifically, the system receives and analyzes users' voice data and provides appropriate feedback and advice. It also provides a simulation function, builds a community of empathy among employees, and connects them to experts and medical institutions as needed.

[0566] Hardware and Software

[0567] The system uses hardware such as a smartphone, microphone, and robot, as well as Python, the SpeechRecognition library, the transformers library, and Google's speech recognition API.

[0568] Data processing and calculation

[0569] First, when a user speaks to a smartphone or robot, the voice data is recorded. For example, suppose the user says, "Work has been tough lately and I'm feeling tired." An application on the smartphone receives this voice data and converts it into text data via Google's speech recognition API. Next, this text data undergoes sentiment analysis using a natural language processing algorithm. Specifically, emotional states such as "fatigue" and "stress" are identified using the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library.

[0570] Based on the results of this sentiment analysis, the server then provides appropriate feedback and advice to the user, such as "Take better care of yourself and take a short break when it comes to relaxation methods."

[0571] Specific examples

[0572] User A says into his smartphone, "Work has been tough lately and I'm feeling tired." The smartphone records this voice data and converts it into text data using Google's speech recognition API. This text data is sent to a server, where sentiment analysis is performed using the BERT model in the Transformers library. As a result of the analysis, "fatigue" and "stress" are identified, and feedback advice is provided, such as "Regarding relaxation methods, you should take better care of yourself. I recommend taking a short break."

[0573] Similarly, when a user uses the simulation function to practice for an interview, the server analyzes the user's responses to questions and provides specific advice in real time, such as, "Your voice is too quiet, so speak louder and with more confidence."

[0574] An example of a prompt is:

[0575] "Work has been hard lately and I'm tired."

[0576] "How can I speak more confidently when answering interview questions?"

[0577] Examples include:

[0578] In this way, the present invention provides a specific system for improving the mental health and work efficiency of employees in a distribution center.

[0579] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0580] Step 1:

[0581] The device (smartphone or robot) records the user's voice data. When the user speaks into the device, the device acquires and records this voice data. The input is the user's voice, and the output is voice data.

[0582] Step 2:

[0583] The device converts the recorded voice data into text data using Google's speech recognition API. Specifically, the device sends the voice data to the API and receives the result as text data. The input is voice data, and the output is the converted text data. It is then sent to the server.

[0584] Step 3:

[0585] The server performs sentiment analysis on the received text data. For this analysis, it uses the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library. The input is the text data, and the output is the result of sentiment analysis (e.g., "fatigue" or "stress").

[0586] Step 4:

[0587] The server generates appropriate feedback and advice based on the results of the sentiment analysis. This feedback is provided using a generative AI model. For example, specific advice such as "Take better care of yourself and take a short break when it comes to relaxation methods" is provided. The input is the result of the sentiment analysis, and the output is the feedback text.

[0588] Step 5:

[0589] The server sends the generated feedback to the terminal for providing to the user. The terminal provides the feedback to the user by displaying or reading it out loud. The input is the feedback sentence, and the output is what is presented to the user.

[0590] Step 6:

[0591] When a user wishes to simulate an everyday situation, the server selects a simulation scenario and sends it to the terminal. The user then performs the simulation, speaking and acting within it. The input is the simulation scenario, and the output is the user's response behavior data.

[0592] Step 7:

[0593] The server analyzes the user's simulated response data and provides feedback on how to improve performance. For example, it provides specific advice such as, "Your voice is too quiet. Speak louder and more confidently." The input is the simulated response data, and the output is feedback for improvement.

[0594] Step 8:

[0595] If necessary, the server connects users to empathy communities and matches them with users in similar situations, allowing users to exchange information and receive support from other users. The input is the user's emotional state and profile information, and the output is the matching results of empathy communities.

[0596] Step 9:

[0597] The server provides a means to connect the user to specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to an appropriate specialist so that they can receive support. The input is the emotion analysis results and the user's needs, and the output is connection information to specialists or medical institutions.

[0598] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0599] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[0600] Analyzing user sentiment and providing feedback

[0601] The system's operation begins with receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert this voice data into text data. The emotion engine then analyzes the text and voice data to identify the user's emotional state in detail. For example, if the user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server generates appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about ways to relax?"

[0602] Skill improvement through simulation functions

[0603] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" may be provided.

[0604] Building a community of empathy

[0605] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0606] Providing professional support

[0607] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[0608] Specific examples

[0609] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0610] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0611] The processing flow will be explained below.

[0612] Analyzing user sentiment and providing feedback

[0613] Step 1:

[0614] User voice input

[0615] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0616] Step 2:

[0617] Sending audio data

[0618] The device records the user's voice data and sends the data to the server.

[0619] Step 3:

[0620] Converting audio data to text

[0621] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[0622] Step 4:

[0623] Sentiment analysis of text data

[0624] The server uses an emotion engine to perform a detailed analysis of the user's emotional state from the text and voice data, identifying emotions such as "fatigue" and "stress."

[0625] Step 5:

[0626] Feedback Generation

[0627] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[0628] Step 6:

[0629] Providing Feedback

[0630] The server provides the generated feedback to the user through the terminal.

[0631] Skill improvement through simulation functions

[0632] Step 1:

[0633] Presenting a simulation scenario

[0634] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[0635] Step 2:

[0636] Collecting and sending simulated responses

[0637] The user responds verbally according to the scenario, and the device records the voice data and sends it to the server.

[0638] Step 3:

[0639] Response text transformation and parsing

[0640] The server converts the voice data of the scenario response into text data and analyzes it using the emotion engine. For example, a response such as "I work at X Corporation" is analyzed.

[0641] Step 4:

[0642] Generate and provide improvement feedback

[0643] The server generates feedback on how to improve the user's response and speaking style, and provides it to the user via the device. For example, the server may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[0644] Building a community of empathy

[0645] Step 1:

[0646] Check user profile and emotional state

[0647] The server collects and checks the user's profile information and sentiment analysis results.

[0648] Step 2:

[0649] Matching with sympathetic communities

[0650] The server searches for other users with the same situation or emotional state and matches them together, for example, matching users who are suffering from work stress.

[0651] Step 3:

[0652] Providing community features

[0653] The server provides matched users with chat rooms and video calls, where they can share their experiences and feelings.

[0654] Providing professional support

[0655] Step 1:

[0656] Connecting to experts based on sentiment analysis results

[0657] Based on the results of the emotion analysis, the server connects the user to the appropriate specialist or medical institution as needed.

[0658] Step 2:

[0659] Providing integrated support

[0660] The server provides a link to professional support (such as psychological counseling) according to the user's needs.

[0661] Specific examples

[0662] Step 1:

[0663] User A's voice input

[0664] User A says to the device, "Work has been tough lately and I'm tired."

[0665] Step 2:

[0666] Sending voice data and converting it to text

[0667] The terminal transmits the voice data to the server, which converts it into text data.

[0668] Step 3:

[0669] Sentiment Analysis and Feedback Generation

[0670] The server uses an emotion engine to analyze emotions such as "fatigue" and "stress" and provides feedback such as "Would you like to know more about ways to relax?"

[0671] Step 4:

[0672] Presenting a simulation scenario

[0673] When user A requests to practice for an interview, the device presents a simulation scenario that asks the user to "introduce themselves."

[0674] Step 5:

[0675] Collecting and analyzing responses and providing improvement feedback

[0676] User A responds, "I work for X Corporation," and the voice data is sent to the server for analysis. The server then generates and provides improvement feedback, such as, "Speak louder and more confidently."

[0677] Step 6:

[0678] Matching with sympathetic communities

[0679] The server matches User A with User B, who is also suffering from work-related stress, and provides a chat room.

[0680] Step 7:

[0681] Providing professional support

[0682] Based on the results of the emotion analysis, the server introduces user A to a psychological counselor.

[0683] In this way, the system starts with the user's voice data and concretely implements each step of analysis, feedback, simulation, community formation, and professional support.

[0684] Example 2

[0685] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] Conventional emotion analysis systems have limitations in their technology for analyzing emotions from user voice data, resulting in problems such as insufficient real-time emotion analysis or appropriate feedback. They also lack simulation functions to help users improve their everyday communication skills, or effective methods for matching users with other users in similar situations. Furthermore, they lack a means for smoothly connecting users with experts or medical institutions when needed. To solve these problems, a system is needed that can accurately analyze a user's emotional state in real time and provide appropriate feedback and advice based on the analysis results.

[0687] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0688] In this invention, the server includes: means for receiving a user's voice data and converting it into text data; means for analyzing the user's emotions from the text data; means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions; means for allowing the user to simulate everyday situations; means for forming an empathy community with other users and matching users with similar situations; means for connecting to experts or medical institutions as needed; means for analyzing the user's emotional state in real time and dynamically generating feedback based on the analysis results; means for identifying a detailed emotional state using the user's speech data and providing specific advice based on the identified emotions; means for forming an empathy community that supports communication between users and promotes empathy; and means for simulating specific situations such as interviews and providing dynamic, real-time feedback on the user's responses. This enables the user's emotions to be analyzed with high accuracy and appropriate feedback to be provided in real time. Furthermore, it makes it easier for users to improve their everyday communication skills and share empathy with other users.

[0689] 1. "Voice Data" means data that is a digital recording of the voice spoken by a User.

[0690] 2. "Text data" means character string data converted from voice data using voice recognition technology.

[0691] 3. "Sentiment analysis" is the process of inferring and identifying a user's emotional state from input text or voice data.

[0692] 4. "Feedback" means advice or suggestions provided to the user based on the results of the analysis.

[0693] 5. "Simulation function" means a function that allows users to virtually experience specific scenes or situations and practice them.

[0694] 6. An "empathy community" is an online space where users in the same position or emotional state can share support and empathy with each other.

[0695] 7. "Matching" is the process of connecting users with other users who are in a similar state based on their emotional state and profile.

[0696] 8. "Means for connecting with specialists and medical institutions" means the means for referring and contacting users with appropriate specialists and medical institutions when necessary.

[0697] 9. "Real-time" refers to the ability to respond to user actions and inputs immediately, with immediate processing and results returned.

[0698] 10. "Means for generating dynamic feedback" refers to technology for generating appropriate feedback according to the user's emotional state and behavior.

[0699] 11. "Specific Advice" means detailed, actionable advice tailored to a user's specific situation and feelings.

[0700] 12. "Means to support communication" are technologies that promote and facilitate dialogue and interaction between users.

[0701] 13. "Simulating" situations such as "interviews" is a system feature that allows users to experience and practice hypothetical interview situations.

[0702] 14. "Means for providing dynamic, real-time feedback to users' responses" refers to technology that provides immediate feedback to users based on their responses and actions.

[0703] MODE FOR CARRYING OUT THE INVENTION

[0704] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[0705] Analyzing user sentiment and providing feedback

[0706] The system begins by receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data. An emotion engine (e.g., IBM Watson Tone Analyzer API) then analyzes the text and voice data to identify the user's emotional state in detail.

[0707] For example, if a user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server will generate appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about how to relax?"

[0708] Skill improvement through simulation functions

[0709] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server, where it is converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses.

[0710] For example, specific advice is provided such as, "Your voice is too quiet, so speak louder and with more confidence."

[0711] Building a community of empathy

[0712] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0713] Providing professional support

[0714] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[0715] Specific examples

[0716] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0717] Example of input prompt for generative AI model

[0718] Prompt: "Please explain sentiment analysis and feedback systems. Please provide examples."

[0719] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0720] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0721] Step 1:

[0722] User utterances and voice data collection

[0723] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0724] The device records the user's speech using a built-in microphone. The voice data (input) is temporarily stored in the device's memory in digital format (output).

[0725] Step 2:

[0726] Sending audio data

[0727] The device encrypts the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0728] The voice data (input) is encrypted and sent to the server (output).

[0729] Step 3:

[0730] Converting audio data to text

[0731] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data.

[0732] Audio data (input) is converted into text data (output).

[0733] Step 4:

[0734] Emotion analysis

[0735] The server uses an emotion engine (specifically, IBM Watson Tone Analyzer API) to perform emotion analysis of text data and voice data.

[0736] From the text data and voice data (input), the user's emotional state (e.g., fatigue, stress) is analyzed (output).

[0737] The server stores the analysis results in an internal database, including the user ID, analysis results, and timestamp.

[0738] Step 5:

[0739] Generate feedback

[0740] The server generates appropriate feedback and advice for the user based on the results of the emotion analysis.

[0741] Based on the analysis results (input), a specific feedback message (e.g., "Would you like to know more about how to relax?") is generated (output).

[0742] Step 6:

[0743] Providing Feedback

[0744] The server sends the generated feedback message to the terminal.

[0745] Feedback messages (input) are forwarded to the terminal (output).

[0746] Step 7:

[0747] View Feedback

[0748] The device displays the received feedback message to the user.

[0749] Feedback messages received on the device (input) are displayed on the screen as popups or notifications (output).

[0750] Step 8:

[0751] Presenting a simulation scenario

[0752] The server generates a simulation scenario such as an interview and transmits it to the terminal.

[0753] The simulation scenario (input) arrives at the terminal and is displayed to the user (output).

[0754] Step 9:

[0755] Collect user responses

[0756] When the user speaks to the simulation, the terminal records the voice data and transmits it back to the server.

[0757] The user's response voice data (input) is sent to the server and converted into text data (output).

[0758] Step 10:

[0759] Analyzing simulation responses and providing feedback

[0760] The server converts the voice data into text data and then analyzes it using an emotion engine.

[0761] It analyzes text data (input) and generates specific feedback (e.g., "You should speak more confidently") (output).

[0762] The server sends a feedback message to the terminal, which displays it to the user.

[0763] Step 11:

[0764] Building a community of empathy

[0765] The server searches for and matches users with other users in similar states based on their emotional state and profile information.

[0766] Matched user information (output) is generated from user information and sentiment analysis results (input).

[0767] Step 12:

[0768] Communication Support

[0769] The server provides an environment where matched users can share experiences through chat rooms and video calls.

[0770] The means of communication between users (links and connection information) (output) are displayed on the terminal.

[0771] Step 13:

[0772] Providing access to specialists and medical institutions

[0773] Based on the analysis results, the server provides a means to connect users to appropriate specialists or medical institutions as needed.

[0774] The analysis results (input) are used to provide contact information and appointment links for specialists and medical institutions (output).

[0775] In this way, this invention not only analyzes a user's emotions in real time using their voice data and provides feedback and advice based on the results, but also realizes a seamless system that includes simulation functions, an empathetic community, and even professional support.

[0776] This allows users to understand their emotional state in three dimensions, utilize feedback to improve their ability to respond to each situation, and access empathetic communities and professional support to solve problems and grow without feeling isolated.

[0777] (Application example 2)

[0778] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0779] While conventional emotion analysis systems can analyze users' emotions in real time, they lack mechanisms for providing appropriate relaxation content or expert support based on the results. As a result, they lack specific approaches to improving users' mental health. Furthermore, they lack the ability to form empathetic communities where users can support each other, and the ability to support users in improving their skills through simulation functions.

[0780] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts and support organizations as needed, and means for providing appropriate relaxation content based on the results of the user's emotion analysis. As a result, individually optimized relaxation content is provided based on the results of the user's emotion analysis, enabling users to support each other through the empathy community and receive support from experts.

[0781] "Voice data" refers to data that is a digital recording of what a user has said.

[0782] "Text data" refers to data obtained by converting voice data into character information.

[0783] "Sentiment analysis" is the process of analyzing a user's emotional state from their text and voice data.

[0784] "Feedback" refers to advice and information provided based on the results of user sentiment analysis.

[0785] The "simulation function" allows users to virtually recreate real-world situations for practice and experience.

[0786] An "empathy community" is an online community where users with similar positions and experiences can support each other.

[0787] "Matching" is the process of connecting users with the same situations or concerns within a community of empathy.

[0788] An "expert" is a professional with specific expertise, such as a psychological counselor or healthcare provider.

[0789] "Support organizations" are organizations that provide assistance, including medical institutions and specialized counseling centers.

[0790] "Relaxation content" refers to digital content such as music, meditation guides, and stories that are designed to reduce stress and promote relaxation for users.

[0791] This invention is a system that analyzes users' emotions in real time, provides personalized relaxation content based on the analysis results, creates a community of empathy, and connects them to experts and support organizations as needed.

[0792] Audio data collection and text conversion

[0793] The server receives the voice data spoken by the user into the device. The device records this voice data and sends it to the server. The server uses a speech recognition engine and the "speech_recognition" library to convert this voice data into text data. This library converts voice to text with high accuracy.

[0794] Emotion analysis

[0795] The server then analyzes the user's emotions using the converted text and audio data. This process uses a generative AI model with the "transformers" library to output an emotion label (e.g., "happiness," "sadness," "stress," etc.) and its score.

[0796] Providing personalized content

[0797] Based on the results of the emotion analysis, the server provides relaxation content appropriate for the user. For example, if the user is analyzed as feeling stressed, the server will provide content such as relaxation music or meditation guides. This selection is based on the emotion label and score.

[0798] Forming and matching communities of empathy

[0799] The server searches for and matches users with other users in similar situations based on their emotional state and profile information, creating a mutual support community that encourages users to share their experiences through chat rooms and video calls.

[0800] Providing professional support

[0801] If necessary, the server also provides a function to connect the user to experts or support organizations. For example, if the results of the emotion analysis indicate that psychological counseling is necessary, the user will be referred to an appropriate expert and receive appropriate support.

[0802] Specific examples

[0803] For example, if a user says, "I've been feeling a lot of stress at work lately," the device sends this voice data to the server. The server converts the voice data into text and then performs emotion analysis using the "transformers" library, calculating the emotion label "stress" and a high score. The server then provides the user with relaxation music and deep breathing guides, and attempts to match them with other users in similar situations. If necessary, it also provides guidance to professional counseling.

[0804] Prompt Sentence Examples

[0805] For example, the following prompts can be used to recommend relaxation music based on the user's emotional state:

[0806] User's emotional status: Stressed (score: 0.85)

[0807] Suitable content for users: Deep breathing guides, Stress relief stories

[0808] In this way, we have created a system that provides relaxation content that is individually optimized based on the user's emotional state, enables users to support each other through an empathetic community, and enables them to receive support from experts.

[0809] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0810] Step 1: Collecting audio data

[0811] The user speaks emotional content into their device, and this voice data is recorded through the device's microphone. The input is the user's speech, and the output is digital voice data.

[0812] Step 2: Sending audio data to the server

[0813] The device sends the recorded voice data to the server. The input is the digital voice data, and the output is the voice data transferred to the server.

[0814] Step 3: Convert audio data to text

[0815] The server converts the received voice data into text data using the "speech_recognition" library. The input is the voice data sent to the server, and the output is text data. This conversion process is performed using voice recognition technology.

[0816] Step 4: Sentiment Analysis

[0817] The server uses generative AI models from the "transformers" library to analyze user emotions from text and audio data. The input is text data, and the output is emotion labels and their scores. In this process, natural language processing algorithms identify emotional states.

[0818] Step 5: Generate feedback

[0819] The server selects relaxation content appropriate for the user based on the emotion analysis results. For example, if an emotion label indicating stress is detected, content such as relaxation music or meditation guides will be selected. The input is the emotion label and its score, and the output is a list of appropriate relaxation content.

[0820] Step 6: Providing content

[0821] The server provides selected relaxation content to users. The input is a list of relaxation content, and the output is links and data of the content that users can access.

[0822] Step 7: Matching with empathetic communities

[0823] The server matches users with other users based on their emotional state and profile information. The input is the user's emotional state and profile information, and the output is the matching results. This process searches for users in similar situations and forms a community of empathy.

[0824] Step 8: Providing professional support

[0825] If necessary, the server will refer the user to a specialist or support organization based on the results of the emotion analysis. The input is the emotion analysis result and the user's health status information, and the output is information on the referral to the appropriate specialist or support organization. This process provides a path for the user to smoothly receive support.

[0826] Through these steps, a system is realized that provides individually optimized relaxation content based on the results of user emotion analysis, enables users to support each other through empathetic communities, and allows them to receive support from experts.

[0827] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0828] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0829] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0830] [Third embodiment]

[0831] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0832] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0833] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0834] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0835] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0836] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0837] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0838] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0839] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0840] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0841] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0842] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0843] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0844] Analyzing user sentiment and providing feedback

[0845] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert the voice data into text data. The server then analyzes the text data and applies a natural language processing algorithm to identify the user's emotional state. For example, if the user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice.

[0846] Improving skills through simulation of everyday situations

[0847] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0848] Building a community of empathy

[0849] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[0850] Providing professional support

[0851] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0852] Specific examples

[0853] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0854] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[0855] The processing flow will be explained below.

[0856] Step 1:

[0857] User voice input

[0858] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[0859] Step 2:

[0860] Sending audio data

[0861] The device records the user's voice data and sends the data to the server.

[0862] Step 3:

[0863] Converting audio data to text

[0864] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[0865] Step 4:

[0866] Sentiment analysis of text data

[0867] The server uses natural language processing algorithms to analyze the user's emotional state from the text data, identifying emotions such as "fatigue" or "stress."

[0868] Step 5:

[0869] Feedback Generation

[0870] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[0871] Step 6:

[0872] Providing Feedback

[0873] The server provides the generated feedback to the user through the terminal.

[0874] Step 7:

[0875] Presenting a simulation scenario

[0876] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[0877] Step 8:

[0878] Collecting and sending simulated responses

[0879] The user responds according to the simulation, and the device records the user's voice response and sends it to the server.

[0880] Step 9:

[0881] Response text transformation and parsing

[0882] The server converts the simulation voice data into text data and analyzes the text data. For example, a response such as "I work for X Corporation" is analyzed.

[0883] Step 10:

[0884] Generate and provide improvement feedback

[0885] The server generates feedback on how to improve responses and speaking style, and provides it to the user via the device. For example, it may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[0886] Step 11:

[0887] Forming and matching communities of empathy

[0888] The server searches for and matches users with similar situations based on the user's emotional state and profile. For example, users who are suffering from work stress can be matched with each other.

[0889] Step 12:

[0890] Providing community features

[0891] The server provides a chat room or a place for video calls between successfully matched users, where users can share their experiences and feelings on the spot.

[0892] Step 13:

[0893] Collaboration with experts and medical institutions

[0894] The server provides a means to connect users to specialists or medical institutions based on their emotional state and concerns, as needed, for example by introducing them to a psychological counselor.

[0895] In this way, each step of the program concretely implements a series of steps, starting with user input and ending with analysis, feedback, simulation, community building, and professional support.

[0896] Example 1

[0897] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0898] Conventional emotion analysis systems lack the ability to process users' voice data in real time or provide feedback through simulation functions, making it difficult for users to immediately identify areas for improvement and receive feedback. Furthermore, they lacked the functionality to match users with similar situations and form communities, making it difficult to prevent users from becoming isolated. Furthermore, it was difficult to appropriately refer users to experts or medical institutions when necessary, making it difficult for users to receive specialized support.

[0899] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0900] In this invention, the server includes means for recording and transmitting user voice data, means for converting the voice data into text data using a voice recognition engine, means for identifying the user's emotional state by applying a natural language processing algorithm, means for generating and providing feedback to the user based on the analysis results, and means for presenting a simulation scenario and collecting and analyzing the user's responses. This allows users to receive emotional analysis and feedback in real time, enabling them to improve their skills through simulations of everyday situations. In addition, by matching users in similar situations and forming empathetic communities, users can support each other and, by quickly connecting them to experts and medical institutions, make it easier for them to receive the support they need.

[0901] A "user" is an individual who uses the system to input voice and receive sentiment analysis and feedback.

[0902] A "terminal" is a device that allows a user to input voice, record the voice data, and transmit it to a server.

[0903] "Server" is a computer system that receives, analyzes, and provides feedback on voice data sent by users.

[0904] "Voice Data" means digital audio information uttered by a User and recorded by a Device.

[0905] "Text data" is character string information converted from voice data by a voice recognition engine.

[0906] A "voice recognition engine" is a software or hardware technology for converting voice data into text data.

[0907] A "natural language processing algorithm" is a computational method for analyzing text data and understanding its meaning and emotions.

[0908] "Sentiment analysis" is the process of identifying a user's emotional state from text data.

[0909] "Feedback" refers to advice and information generated based on the results of sentiment analysis.

[0910] A "simulation scenario" is a setting or situation in which users can virtually recreate everyday situations to practice and improve their skills.

[0911] An "empathy community" is a virtual group where users in the same position or situation interact and support each other.

[0912] "Matching" is the process of pairing users who share the same position or situation within a community of empathy.

[0913] An "expert" is a professional (e.g., a psychological counselor or medical institution staff) who provides the support that users need.

[0914] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[0915] Analyzing user sentiment and providing feedback

[0916] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a voice recognition engine to convert this voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API. The server then analyzes this text data and applies a natural language processing algorithm to identify the user's emotional state. Specifically, it uses IBM Watson Natural Language Understanding. For example, if a user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice. For example, it may provide feedback such as, "Would you like to know more about how to relax?"

[0917] Improving skills through simulation of everyday situations

[0918] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[0919] Building a community of empathy

[0920] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms or video calls. Specific software includes general-purpose software used for video calls and messaging applications used in chat rooms.

[0921] Providing professional support

[0922] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[0923] Specific examples

[0924] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[0925] Prompt Sentence Examples

[0926] "Your task will be to create a program that receives voice data from users, analyzes their emotions, and provides appropriate feedback. The speech recognition engine used will be the Google Cloud Speech-to-Text API, and the natural language processing engine will be IBM Watson Natural Language Understanding. Your task will also include the ability to present simulation scenarios, analyze responses, match users with empathetic communities, and provide professional support."

[0927] In this way, the system of the present invention can consistently provide users' sentiment analysis, simulation of everyday situations, building empathy communities, and professional support.

[0928] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0929] Step 1:

[0930] The user speaks into the device. For example, if the user says, "Work has been hard lately and I'm tired," the device's microphone captures this voice. This voice data is temporarily stored on the device as a digital file. The input is the user's voice, and the output is a digital audio file.

[0931] Step 2:

[0932] The device sends the recorded audio data to the server using an HTTP Post request. The input is a digital audio file, and the output is an HTTP request sent to the server.

[0933] Step 3:

[0934] The server passes the received voice data to the voice recognition engine. Specifically, it converts the voice data into text data using the Google Cloud Speech-to-Text API. At this stage, the server sends the voice data to the API and receives the response as text data. The input is voice data, and the output is text data.

[0935] Step 4:

[0936] The server passes the generated text data to a natural language processing algorithm. Specifically, it uses IBM Watson Natural Language Understanding to perform sentiment analysis on the text data. Here, the input is the text data, and the output is the analyzed emotional state (e.g., "fatigue" or "stress").

[0937] Step 5:

[0938] The server generates appropriate feedback based on the analysis results. For example, if the result of the emotion analysis is "fatigue," the server generates advice on relaxation methods and stress management. At this stage, the server creates text-based feedback based on the emotion analysis results. The input is the analyzed emotional state, and the output is the feedback text.

[0939] Step 6:

[0940] The server sends the generated feedback to the device. Specifically, it returns the feedback text to the device as an HTTP response. The device can then display the received feedback to the user or give it to them as audio. The input is the feedback text, and the output is the display or audio output for the user.

[0941] Step 7:

[0942] When a user uses the simulation function, the terminal presents a simulation scenario. For example, if the user selects an interview simulation, the terminal presents questions based on the scenario. The input is the user's simulation scenario selection, and the output is the presented questions.

[0943] Step 8:

[0944] When the user answers questions in the simulation, the response is recorded by the device and sent to the server. The server receives this voice data, converts it into text using a speech recognition engine, and analyzes it using a natural language processing algorithm. The input is the user's response, and the output is the analyzed response.

[0945] Step 9:

[0946] The server generates feedback based on the content of the user's simulated responses and sends it to the device, including specific advice such as "You should speak more confidently." The input is the parsed response content, and the output is the feedback text.

[0947] Step 10:

[0948] The server searches for and matches other users based on the user's emotional state and profile information. This allows for the formation of empathetic communities. For example, users who suffer from work-related stress can be matched. The input is the user's emotional state and profile information, and the output is the matching results.

[0949] Step 11:

[0950] If necessary, the server provides a means to connect the user to an appropriate specialist or medical institution. For example, if psychological counseling is determined to be necessary, the server will refer the user to a counselor. The input is the emotion analysis result and the user's condition, and the output is the connection information for the specialist or medical institution.

[0951] In this way, the inventive system can seamlessly provide users' sentiment analysis, feedback provision, simulation, empathetic community building and professional support.

[0952] (Application example 1)

[0953] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0954] Employees working at logistics centers are often exposed to mental stress and pressure in the course of their work, which has a negative impact on their work efficiency and health. Furthermore, due to a lack of communication between employees and limited opportunities to receive professional counseling and support, sufficient measures are not being taken to reduce stress and improve work efficiency. To solve these issues, a system is needed that can accurately analyze employees' emotional states and provide appropriate feedback and advice in real time.

[0955] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0956] In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts or medical institutions as needed, and means for supporting the user's mental health and improving work efficiency, thereby making it possible to maintain the mental health of employees working at logistics centers and improve work efficiency.

[0957] "User" refers to any individual or employee who uses the System.

[0958] "Voice data" means electronic data that is a recording of what a user says.

[0959] "Text data" refers to data obtained by converting voice data into character information.

[0960] "Sentiment analysis" is a technique that applies natural language processing algorithms to text data to identify a user's emotional state.

[0961] "Feedback" refers to advice or comments provided to users based on the results of sentiment analysis.

[0962] "Everyday scene simulation" is a function that allows users to simulate real-life situations.

[0963] An "empathy community" is a community where users in the same position or circumstances can support each other.

[0964] "Matching" refers to connecting appropriate users within a community of like-minded people.

[0965] "Means for connecting with experts and medical institutions" refers to means for providing the necessary professional support based on the results of the user's sentiment analysis.

[0966] "Mental health" refers to a user's state of psychological and emotional well-being.

[0967] "Work efficiency" refers to the degree to which logistics center employees perform their work effectively and efficiently.

[0968] This invention relates to a system for supporting the mental health of logistics center employees and improving their work efficiency. Specifically, the system receives and analyzes users' voice data and provides appropriate feedback and advice. It also provides a simulation function, builds a community of empathy among employees, and connects them to experts and medical institutions as needed.

[0969] Hardware and Software

[0970] The system uses hardware such as a smartphone, microphone, and robot, as well as Python, the SpeechRecognition library, the transformers library, and Google's speech recognition API.

[0971] Data processing and calculation

[0972] First, when a user speaks to a smartphone or robot, the voice data is recorded. For example, suppose the user says, "Work has been tough lately and I'm feeling tired." An application on the smartphone receives this voice data and converts it into text data via Google's speech recognition API. Next, this text data undergoes sentiment analysis using a natural language processing algorithm. Specifically, emotional states such as "fatigue" and "stress" are identified using the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library.

[0973] Based on the results of this sentiment analysis, the server then provides appropriate feedback and advice to the user, such as "Take better care of yourself and take a short break when it comes to relaxation methods."

[0974] Specific examples

[0975] User A says into his smartphone, "Work has been tough lately and I'm feeling tired." The smartphone records this voice data and converts it into text data using Google's speech recognition API. This text data is sent to a server, where sentiment analysis is performed using the BERT model in the Transformers library. As a result of the analysis, "fatigue" and "stress" are identified, and feedback advice is provided, such as "Regarding relaxation methods, you should take better care of yourself. I recommend taking a short break."

[0976] Similarly, when a user uses the simulation function to practice for an interview, the server analyzes the user's responses to questions and provides specific advice in real time, such as, "Your voice is too quiet, so speak louder and with more confidence."

[0977] An example of a prompt is:

[0978] "Work has been hard lately and I'm tired."

[0979] "How can I speak more confidently when answering interview questions?"

[0980] Examples include:

[0981] In this way, the present invention provides a specific system for improving the mental health and work efficiency of employees in a distribution center.

[0982] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0983] Step 1:

[0984] The device (smartphone or robot) records the user's voice data. When the user speaks into the device, the device acquires and records this voice data. The input is the user's voice, and the output is voice data.

[0985] Step 2:

[0986] The device converts the recorded voice data into text data using Google's speech recognition API. Specifically, the device sends the voice data to the API and receives the result as text data. The input is voice data, and the output is the converted text data. It is then sent to the server.

[0987] Step 3:

[0988] The server performs sentiment analysis on the received text data. For this analysis, it uses the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library. The input is the text data, and the output is the result of sentiment analysis (e.g., "fatigue" or "stress").

[0989] Step 4:

[0990] The server generates appropriate feedback and advice based on the results of the sentiment analysis. This feedback is provided using a generative AI model. For example, specific advice such as "Take better care of yourself and take a short break when it comes to relaxation methods" is provided. The input is the result of the sentiment analysis, and the output is the feedback text.

[0991] Step 5:

[0992] The server sends the generated feedback to the terminal for providing to the user. The terminal provides the feedback to the user by displaying or reading it out loud. The input is the feedback sentence, and the output is what is presented to the user.

[0993] Step 6:

[0994] When a user wishes to simulate an everyday situation, the server selects a simulation scenario and sends it to the terminal. The user then performs the simulation, speaking and acting within it. The input is the simulation scenario, and the output is the user's response behavior data.

[0995] Step 7:

[0996] The server analyzes the user's simulated response data and provides feedback on how to improve performance. For example, it provides specific advice such as, "Your voice is too quiet. Speak louder and more confidently." The input is the simulated response data, and the output is feedback for improvement.

[0997] Step 8:

[0998] If necessary, the server connects users to empathy communities and matches them with users in similar situations, allowing users to exchange information and receive support from other users. The input is the user's emotional state and profile information, and the output is the matching results of empathy communities.

[0999] Step 9:

[1000] The server provides a means to connect the user to specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to an appropriate specialist so that they can receive support. The input is the emotion analysis results and the user's needs, and the output is connection information to specialists or medical institutions.

[1001] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1002] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[1003] Analyzing user sentiment and providing feedback

[1004] The system's operation begins with receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert this voice data into text data. The emotion engine then analyzes the text and voice data to identify the user's emotional state in detail. For example, if the user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server generates appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about ways to relax?"

[1005] Skill improvement through simulation functions

[1006] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" may be provided.

[1007] Building a community of empathy

[1008] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[1009] Providing professional support

[1010] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[1011] Specific examples

[1012] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1013] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[1014] The processing flow will be explained below.

[1015] Analyzing user sentiment and providing feedback

[1016] Step 1:

[1017] User voice input

[1018] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[1019] Step 2:

[1020] Sending audio data

[1021] The device records the user's voice data and sends the data to the server.

[1022] Step 3:

[1023] Converting audio data to text

[1024] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[1025] Step 4:

[1026] Sentiment analysis of text data

[1027] The server uses an emotion engine to perform a detailed analysis of the user's emotional state from the text and voice data, identifying emotions such as "fatigue" and "stress."

[1028] Step 5:

[1029] Feedback Generation

[1030] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[1031] Step 6:

[1032] Providing Feedback

[1033] The server provides the generated feedback to the user through the terminal.

[1034] Skill improvement through simulation functions

[1035] Step 1:

[1036] Presenting a simulation scenario

[1037] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[1038] Step 2:

[1039] Collecting and sending simulated responses

[1040] The user responds verbally according to the scenario, and the device records the voice data and sends it to the server.

[1041] Step 3:

[1042] Response text transformation and parsing

[1043] The server converts the voice data of the scenario response into text data and analyzes it using the emotion engine. For example, a response such as "I work at X Corporation" is analyzed.

[1044] Step 4:

[1045] Generate and provide improvement feedback

[1046] The server generates feedback on how to improve the user's response and speaking style, and provides it to the user via the device. For example, the server may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[1047] Building a community of empathy

[1048] Step 1:

[1049] Check user profile and emotional state

[1050] The server collects and checks the user's profile information and sentiment analysis results.

[1051] Step 2:

[1052] Matching with sympathetic communities

[1053] The server searches for other users with the same situation or emotional state and matches them together, for example, matching users who are suffering from work stress.

[1054] Step 3:

[1055] Providing community features

[1056] The server provides matched users with chat rooms and video calls, where they can share their experiences and feelings.

[1057] Providing professional support

[1058] Step 1:

[1059] Connecting to experts based on sentiment analysis results

[1060] Based on the results of the emotion analysis, the server connects the user to the appropriate specialist or medical institution as needed.

[1061] Step 2:

[1062] Providing integrated support

[1063] The server provides a link to professional support (such as psychological counseling) according to the user's needs.

[1064] Specific examples

[1065] Step 1:

[1066] User A's voice input

[1067] User A says to the device, "Work has been tough lately and I'm tired."

[1068] Step 2:

[1069] Sending voice data and converting it to text

[1070] The terminal transmits the voice data to the server, which converts it into text data.

[1071] Step 3:

[1072] Sentiment Analysis and Feedback Generation

[1073] The server uses an emotion engine to analyze emotions such as "fatigue" and "stress" and provides feedback such as "Would you like to know more about ways to relax?"

[1074] Step 4:

[1075] Presenting a simulation scenario

[1076] When user A requests to practice for an interview, the device presents a simulation scenario that asks the user to "introduce themselves."

[1077] Step 5:

[1078] Collecting and analyzing responses and providing improvement feedback

[1079] User A responds, "I work for X Corporation," and the voice data is sent to the server for analysis. The server then generates and provides improvement feedback, such as, "Speak louder and more confidently."

[1080] Step 6:

[1081] Matching with sympathetic communities

[1082] The server matches User A with User B, who is also suffering from work-related stress, and provides a chat room.

[1083] Step 7:

[1084] Providing professional support

[1085] Based on the results of the emotion analysis, the server introduces user A to a psychological counselor.

[1086] In this way, the system starts with the user's voice data and concretely implements each step of analysis, feedback, simulation, community formation, and professional support.

[1087] Example 2

[1088] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1089] Conventional emotion analysis systems have limitations in their technology for analyzing emotions from user voice data, resulting in problems such as insufficient real-time emotion analysis or appropriate feedback. They also lack simulation functions to help users improve their everyday communication skills, or effective methods for matching users with other users in similar situations. Furthermore, they lack a means for smoothly connecting users with experts or medical institutions when needed. To solve these problems, a system is needed that can accurately analyze a user's emotional state in real time and provide appropriate feedback and advice based on the analysis results.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1091] In this invention, the server includes: means for receiving a user's voice data and converting it into text data; means for analyzing the user's emotions from the text data; means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions; means for allowing the user to simulate everyday situations; means for forming an empathy community with other users and matching users with similar situations; means for connecting to experts or medical institutions as needed; means for analyzing the user's emotional state in real time and dynamically generating feedback based on the analysis results; means for identifying a detailed emotional state using the user's speech data and providing specific advice based on the identified emotions; means for forming an empathy community that supports communication between users and promotes empathy; and means for simulating specific situations such as interviews and providing dynamic, real-time feedback on the user's responses. This enables the user's emotions to be analyzed with high accuracy and appropriate feedback to be provided in real time. Furthermore, it makes it easier for users to improve their everyday communication skills and share empathy with other users.

[1092] 1. "Voice Data" means data that is a digital recording of the voice spoken by a User.

[1093] 2. "Text data" means character string data converted from voice data using voice recognition technology.

[1094] 3. "Sentiment analysis" is the process of inferring and identifying a user's emotional state from input text or voice data.

[1095] 4. "Feedback" means advice or suggestions provided to the user based on the results of the analysis.

[1096] 5. "Simulation function" means a function that allows users to virtually experience specific scenes or situations and practice them.

[1097] 6. An "empathy community" is an online space where users in the same position or emotional state can share support and empathy with each other.

[1098] 7. "Matching" is the process of connecting users with other users who are in a similar state based on their emotional state and profile.

[1099] 8. "Means for connecting with specialists and medical institutions" means the means for referring and contacting users with appropriate specialists and medical institutions when necessary.

[1100] 9. "Real-time" refers to the ability to respond to user actions and inputs immediately, with immediate processing and results returned.

[1101] 10. "Means for generating dynamic feedback" refers to technology for generating appropriate feedback according to the user's emotional state and behavior.

[1102] 11. "Specific Advice" means detailed, actionable advice tailored to a user's specific situation and feelings.

[1103] 12. "Means to support communication" are technologies that promote and facilitate dialogue and interaction between users.

[1104] 13. "Simulating" situations such as "interviews" is a system feature that allows users to experience and practice hypothetical interview situations.

[1105] 14. "Means for providing dynamic, real-time feedback to users' responses" refers to technology that provides immediate feedback to users based on their responses and actions.

[1106] MODE FOR CARRYING OUT THE INVENTION

[1107] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[1108] Analyzing user sentiment and providing feedback

[1109] The system begins by receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data. An emotion engine (e.g., IBM Watson Tone Analyzer API) then analyzes the text and voice data to identify the user's emotional state in detail.

[1110] For example, if a user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server will generate appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about how to relax?"

[1111] Skill improvement through simulation functions

[1112] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server, where it is converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses.

[1113] For example, specific advice is provided such as, "Your voice is too quiet, so speak louder and with more confidence."

[1114] Building a community of empathy

[1115] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[1116] Providing professional support

[1117] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[1118] Specific examples

[1119] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1120] Example of input prompt for generative AI model

[1121] Prompt: "Please explain sentiment analysis and feedback systems. Please provide examples."

[1122] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[1123] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1124] Step 1:

[1125] User utterances and voice data collection

[1126] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[1127] The device records the user's speech using a built-in microphone. The voice data (input) is temporarily stored in the device's memory in digital format (output).

[1128] Step 2:

[1129] Sending audio data

[1130] The device encrypts the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1131] The voice data (input) is encrypted and sent to the server (output).

[1132] Step 3:

[1133] Converting audio data to text

[1134] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data.

[1135] Audio data (input) is converted into text data (output).

[1136] Step 4:

[1137] Emotion analysis

[1138] The server uses an emotion engine (specifically, IBM Watson Tone Analyzer API) to perform emotion analysis of text data and voice data.

[1139] From the text data and voice data (input), the user's emotional state (e.g., fatigue, stress) is analyzed (output).

[1140] The server stores the analysis results in an internal database, including the user ID, analysis results, and timestamp.

[1141] Step 5:

[1142] Generate feedback

[1143] The server generates appropriate feedback and advice for the user based on the results of the emotion analysis.

[1144] Based on the analysis results (input), a specific feedback message (e.g., "Would you like to know more about how to relax?") is generated (output).

[1145] Step 6:

[1146] Providing Feedback

[1147] The server sends the generated feedback message to the terminal.

[1148] Feedback messages (input) are forwarded to the terminal (output).

[1149] Step 7:

[1150] View Feedback

[1151] The device displays the received feedback message to the user.

[1152] Feedback messages received on the device (input) are displayed on the screen as popups or notifications (output).

[1153] Step 8:

[1154] Presenting a simulation scenario

[1155] The server generates a simulation scenario such as an interview and transmits it to the terminal.

[1156] The simulation scenario (input) arrives at the terminal and is displayed to the user (output).

[1157] Step 9:

[1158] Collect user responses

[1159] When the user speaks to the simulation, the terminal records the voice data and transmits it back to the server.

[1160] The user's response voice data (input) is sent to the server and converted into text data (output).

[1161] Step 10:

[1162] Analyzing simulation responses and providing feedback

[1163] The server converts the voice data into text data and then analyzes it using an emotion engine.

[1164] It analyzes text data (input) and generates specific feedback (e.g., "You should speak more confidently") (output).

[1165] The server sends a feedback message to the terminal, which displays it to the user.

[1166] Step 11:

[1167] Building a community of empathy

[1168] The server searches for and matches users with other users in similar states based on their emotional state and profile information.

[1169] Matched user information (output) is generated from user information and sentiment analysis results (input).

[1170] Step 12:

[1171] Communication Support

[1172] The server provides an environment where matched users can share experiences through chat rooms and video calls.

[1173] The means of communication between users (links and connection information) (output) are displayed on the terminal.

[1174] Step 13:

[1175] Providing access to specialists and medical institutions

[1176] Based on the analysis results, the server provides a means to connect users to appropriate specialists or medical institutions as needed.

[1177] The analysis results (input) are used to provide contact information and appointment links for specialists and medical institutions (output).

[1178] In this way, this invention not only analyzes a user's emotions in real time using their voice data and provides feedback and advice based on the results, but also realizes a seamless system that includes simulation functions, an empathetic community, and even professional support.

[1179] This allows users to understand their emotional state in three dimensions, utilize feedback to improve their ability to respond to each situation, and access empathetic communities and professional support to solve problems and grow without feeling isolated.

[1180] (Application example 2)

[1181] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1182] While conventional emotion analysis systems can analyze users' emotions in real time, they lack mechanisms for providing appropriate relaxation content or expert support based on the results. As a result, they lack specific approaches to improving users' mental health. Furthermore, they lack the ability to form empathetic communities where users can support each other, and the ability to support users in improving their skills through simulation functions.

[1183] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts and support organizations as needed, and means for providing appropriate relaxation content based on the results of the user's emotion analysis. As a result, individually optimized relaxation content is provided based on the results of the user's emotion analysis, enabling users to support each other through the empathy community and receive support from experts.

[1184] "Voice data" refers to data that is a digital recording of what a user has said.

[1185] "Text data" refers to data obtained by converting voice data into character information.

[1186] "Sentiment analysis" is the process of analyzing a user's emotional state from their text and voice data.

[1187] "Feedback" refers to advice and information provided based on the results of user sentiment analysis.

[1188] The "simulation function" allows users to virtually recreate real-world situations for practice and experience.

[1189] An "empathy community" is an online community where users with similar positions and experiences can support each other.

[1190] "Matching" is the process of connecting users with the same situations or concerns within a community of empathy.

[1191] An "expert" is a professional with specific expertise, such as a psychological counselor or healthcare provider.

[1192] "Support organizations" are organizations that provide assistance, including medical institutions and specialized counseling centers.

[1193] "Relaxation content" refers to digital content such as music, meditation guides, and stories that are designed to reduce stress and promote relaxation for users.

[1194] This invention is a system that analyzes users' emotions in real time, provides personalized relaxation content based on the analysis results, creates a community of empathy, and connects them to experts and support organizations as needed.

[1195] Audio data collection and text conversion

[1196] The server receives the voice data spoken by the user into the device. The device records this voice data and sends it to the server. The server uses a speech recognition engine and the "speech_recognition" library to convert this voice data into text data. This library converts voice to text with high accuracy.

[1197] Emotion analysis

[1198] The server then analyzes the user's emotions using the converted text and audio data. This process uses a generative AI model with the "transformers" library to output an emotion label (e.g., "happiness," "sadness," "stress," etc.) and its score.

[1199] Providing personalized content

[1200] Based on the results of the emotion analysis, the server provides relaxation content appropriate for the user. For example, if the user is analyzed as feeling stressed, the server will provide content such as relaxation music or meditation guides. This selection is based on the emotion label and score.

[1201] Forming and matching communities of empathy

[1202] The server searches for and matches users with other users in similar situations based on their emotional state and profile information, creating a mutual support community that encourages users to share their experiences through chat rooms and video calls.

[1203] Providing professional support

[1204] If necessary, the server also provides a function to connect the user to experts or support organizations. For example, if the results of the emotion analysis indicate that psychological counseling is necessary, the user will be referred to an appropriate expert and receive appropriate support.

[1205] Specific examples

[1206] For example, if a user says, "I've been feeling a lot of stress at work lately," the device sends this voice data to the server. The server converts the voice data into text and then performs emotion analysis using the "transformers" library, calculating the emotion label "stress" and a high score. The server then provides the user with relaxation music and deep breathing guides, and attempts to match them with other users in similar situations. If necessary, it also provides guidance to professional counseling.

[1207] Prompt Sentence Examples

[1208] For example, the following prompts can be used to recommend relaxation music based on the user's emotional state:

[1209] User's emotional status: Stressed (score: 0.85)

[1210] Suitable content for users: Deep breathing guides, Stress relief stories

[1211] In this way, we have created a system that provides relaxation content that is individually optimized based on the user's emotional state, enables users to support each other through an empathetic community, and enables them to receive support from experts.

[1212] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1213] Step 1: Collecting audio data

[1214] The user speaks emotional content into their device, and this voice data is recorded through the device's microphone. The input is the user's speech, and the output is digital voice data.

[1215] Step 2: Sending audio data to the server

[1216] The device sends the recorded voice data to the server. The input is the digital voice data, and the output is the voice data transferred to the server.

[1217] Step 3: Convert audio data to text

[1218] The server converts the received voice data into text data using the "speech_recognition" library. The input is the voice data sent to the server, and the output is text data. This conversion process is performed using voice recognition technology.

[1219] Step 4: Sentiment Analysis

[1220] The server uses generative AI models from the "transformers" library to analyze user emotions from text and audio data. The input is text data, and the output is emotion labels and their scores. In this process, natural language processing algorithms identify emotional states.

[1221] Step 5: Generate feedback

[1222] The server selects relaxation content appropriate for the user based on the emotion analysis results. For example, if an emotion label indicating stress is detected, content such as relaxation music or meditation guides will be selected. The input is the emotion label and its score, and the output is a list of appropriate relaxation content.

[1223] Step 6: Providing content

[1224] The server provides selected relaxation content to users. The input is a list of relaxation content, and the output is links and data of the content that users can access.

[1225] Step 7: Matching with empathetic communities

[1226] The server matches users with other users based on their emotional state and profile information. The input is the user's emotional state and profile information, and the output is the matching results. This process searches for users in similar situations and forms a community of empathy.

[1227] Step 8: Providing professional support

[1228] If necessary, the server will refer the user to a specialist or support organization based on the results of the emotion analysis. The input is the emotion analysis result and the user's health status information, and the output is information on the referral to the appropriate specialist or support organization. This process provides a path for the user to smoothly receive support.

[1229] Through these steps, a system is realized that provides individually optimized relaxation content based on the results of user emotion analysis, enables users to support each other through empathetic communities, and allows them to receive support from experts.

[1230] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1231] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1232] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1233] [Fourth embodiment]

[1234] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1235] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1236] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1237] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1238] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1239] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1240] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1241] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1242] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1243] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1244] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1245] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1246] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1247] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[1248] Analyzing user sentiment and providing feedback

[1249] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert the voice data into text data. The server then analyzes the text data and applies a natural language processing algorithm to identify the user's emotional state. For example, if the user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice.

[1250] Improving skills through simulation of everyday situations

[1251] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[1252] Building a community of empathy

[1253] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[1254] Providing professional support

[1255] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[1256] Specific examples

[1257] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1258] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[1259] The processing flow will be explained below.

[1260] Step 1:

[1261] User voice input

[1262] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[1263] Step 2:

[1264] Sending audio data

[1265] The device records the user's voice data and sends the data to the server.

[1266] Step 3:

[1267] Converting audio data to text

[1268] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[1269] Step 4:

[1270] Sentiment analysis of text data

[1271] The server uses natural language processing algorithms to analyze the user's emotional state from the text data, identifying emotions such as "fatigue" or "stress."

[1272] Step 5:

[1273] Feedback Generation

[1274] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[1275] Step 6:

[1276] Providing Feedback

[1277] The server provides the generated feedback to the user through the terminal.

[1278] Step 7:

[1279] Presenting a simulation scenario

[1280] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[1281] Step 8:

[1282] Collecting and sending simulated responses

[1283] The user responds according to the simulation, and the device records the user's voice response and sends it to the server.

[1284] Step 9:

[1285] Response text transformation and parsing

[1286] The server converts the simulation voice data into text data and analyzes the text data. For example, a response such as "I work for X Corporation" is analyzed.

[1287] Step 10:

[1288] Generate and provide improvement feedback

[1289] The server generates feedback on how to improve responses and speaking style, and provides it to the user via the device. For example, it may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[1290] Step 11:

[1291] Forming and matching communities of empathy

[1292] The server searches for and matches users with similar situations based on the user's emotional state and profile. For example, users who are suffering from work stress can be matched with each other.

[1293] Step 12:

[1294] Providing community features

[1295] The server provides a chat room or a place for video calls between successfully matched users, where users can share their experiences and feelings on the spot.

[1296] Step 13:

[1297] Collaboration with experts and medical institutions

[1298] The server provides a means to connect users to specialists or medical institutions based on their emotional state and concerns, as needed, for example by introducing them to a psychological counselor.

[1299] In this way, each step of the program concretely implements a series of steps, starting with user input and ending with analysis, feedback, simulation, community building, and professional support.

[1300] Example 1

[1301] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1302] Conventional emotion analysis systems lack the ability to process users' voice data in real time or provide feedback through simulation functions, making it difficult for users to immediately identify areas for improvement and receive feedback. Furthermore, they lacked the functionality to match users with similar situations and form communities, making it difficult to prevent users from becoming isolated. Furthermore, it was difficult to appropriately refer users to experts or medical institutions when necessary, making it difficult for users to receive specialized support.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1304] In this invention, the server includes means for recording and transmitting user voice data, means for converting the voice data into text data using a voice recognition engine, means for identifying the user's emotional state by applying a natural language processing algorithm, means for generating and providing feedback to the user based on the analysis results, and means for presenting a simulation scenario and collecting and analyzing the user's responses. This allows users to receive emotional analysis and feedback in real time, enabling them to improve their skills through simulations of everyday situations. In addition, by matching users in similar situations and forming empathetic communities, users can support each other and, by quickly connecting them to experts and medical institutions, make it easier for them to receive the support they need.

[1305] A "user" is an individual who uses the system to input voice and receive sentiment analysis and feedback.

[1306] A "terminal" is a device that allows a user to input voice, record the voice data, and transmit it to a server.

[1307] "Server" is a computer system that receives, analyzes, and provides feedback on voice data sent by users.

[1308] "Voice Data" means digital audio information uttered by a User and recorded by a Device.

[1309] "Text data" is character string information converted from voice data by a voice recognition engine.

[1310] A "voice recognition engine" is a software or hardware technology for converting voice data into text data.

[1311] A "natural language processing algorithm" is a computational method for analyzing text data and understanding its meaning and emotions.

[1312] "Sentiment analysis" is the process of identifying a user's emotional state from text data.

[1313] "Feedback" refers to advice and information generated based on the results of sentiment analysis.

[1314] A "simulation scenario" is a setting or situation in which users can virtually recreate everyday situations to practice and improve their skills.

[1315] An "empathy community" is a virtual group where users in the same position or situation interact and support each other.

[1316] "Matching" is the process of pairing users who share the same position or situation within a community of empathy.

[1317] An "expert" is a professional (e.g., a psychological counselor or medical institution staff) who provides the support that users need.

[1318] This invention relates to a system that analyzes users' emotions, provides feedback, simulates their interactions, builds empathy communities, and provides professional support. This system, which is comprised of a server and a terminal as its core components, receives and analyzes users' voice data and provides appropriate feedback based on the results. It can also simulate everyday communication situations to help users improve their skills.

[1319] Analyzing user sentiment and providing feedback

[1320] The system first receives voice data from the user. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a voice recognition engine to convert this voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API. The server then analyzes this text data and applies a natural language processing algorithm to identify the user's emotional state. Specifically, it uses IBM Watson Natural Language Understanding. For example, if a user says, "Work has been tough and I'm tired lately," the server will derive the analysis results as "fatigue" or "stress." Based on the analyzed emotions, the server then provides the user with appropriate feedback and advice. For example, it may provide feedback such as, "Would you like to know more about how to relax?"

[1321] Improving skills through simulation of everyday situations

[1322] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data and provides feedback on the user's responses and speaking style. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" is provided.

[1323] Building a community of empathy

[1324] Based on a user's emotional state and profile information, the server searches for and matches other users in the same position or situation. This allows users to form empathetic communities where they can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms or video calls. Specific software includes general-purpose software used for video calls and messaging applications used in chat rooms.

[1325] Providing professional support

[1326] Based on the results of the user's emotion analysis, the server also provides a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will introduce the user to a counselor so that the user can receive appropriate support.

[1327] Specific examples

[1328] For example, if User A says, "Work has been tough and I'm feeling tired lately," the device sends this voice data to the server, which converts the voice into text data. A natural language processing algorithm then analyzes emotions such as "fatigue" and "stress," and provides feedback such as, "Would you like to know more about relaxation techniques?" Similarly, when User A simulates an interview, the server analyzes the responses and provides specific advice in real time, such as, "You should speak with more confidence." Furthermore, User A is matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1329] Prompt Sentence Examples

[1330] "Your task will be to create a program that receives voice data from users, analyzes their emotions, and provides appropriate feedback. The speech recognition engine used will be the Google Cloud Speech-to-Text API, and the natural language processing engine will be IBM Watson Natural Language Understanding. Your task will also include the ability to present simulation scenarios, analyze responses, match users with empathetic communities, and provide professional support."

[1331] In this way, the system of the present invention can consistently provide users' sentiment analysis, simulation of everyday situations, building empathy communities, and professional support.

[1332] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1333] Step 1:

[1334] The user speaks into the device. For example, if the user says, "Work has been hard lately and I'm tired," the device's microphone captures this voice. This voice data is temporarily stored on the device as a digital file. The input is the user's voice, and the output is a digital audio file.

[1335] Step 2:

[1336] The device sends the recorded audio data to the server using an HTTP Post request. The input is a digital audio file, and the output is an HTTP request sent to the server.

[1337] Step 3:

[1338] The server passes the received voice data to the voice recognition engine. Specifically, it converts the voice data into text data using the Google Cloud Speech-to-Text API. At this stage, the server sends the voice data to the API and receives the response as text data. The input is voice data, and the output is text data.

[1339] Step 4:

[1340] The server passes the generated text data to a natural language processing algorithm. Specifically, it uses IBM Watson Natural Language Understanding to perform sentiment analysis on the text data. Here, the input is the text data, and the output is the analyzed emotional state (e.g., "fatigue" or "stress").

[1341] Step 5:

[1342] The server generates appropriate feedback based on the analysis results. For example, if the result of the emotion analysis is "fatigue," the server generates advice on relaxation methods and stress management. At this stage, the server creates text-based feedback based on the emotion analysis results. The input is the analyzed emotional state, and the output is the feedback text.

[1343] Step 6:

[1344] The server sends the generated feedback to the device. Specifically, it returns the feedback text to the device as an HTTP response. The device can then display the received feedback to the user or give it to them as audio. The input is the feedback text, and the output is the display or audio output for the user.

[1345] Step 7:

[1346] When a user uses the simulation function, the terminal presents a simulation scenario. For example, if the user selects an interview simulation, the terminal presents questions based on the scenario. The input is the user's simulation scenario selection, and the output is the presented questions.

[1347] Step 8:

[1348] When the user answers questions in the simulation, the response is recorded by the device and sent to the server. The server receives this voice data, converts it into text using a speech recognition engine, and analyzes it using a natural language processing algorithm. The input is the user's response, and the output is the analyzed response.

[1349] Step 9:

[1350] The server generates feedback based on the content of the user's simulated responses and sends it to the device, including specific advice such as "You should speak more confidently." The input is the parsed response content, and the output is the feedback text.

[1351] Step 10:

[1352] The server searches for and matches other users based on the user's emotional state and profile information. This allows for the formation of empathetic communities. For example, users who suffer from work-related stress can be matched. The input is the user's emotional state and profile information, and the output is the matching results.

[1353] Step 11:

[1354] If necessary, the server provides a means to connect the user to an appropriate specialist or medical institution. For example, if psychological counseling is determined to be necessary, the server will refer the user to a counselor. The input is the emotion analysis result and the user's condition, and the output is the connection information for the specialist or medical institution.

[1355] In this way, the inventive system can seamlessly provide users' sentiment analysis, feedback provision, simulation, empathetic community building and professional support.

[1356] (Application example 1)

[1357] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] Employees working at logistics centers are often exposed to mental stress and pressure in the course of their work, which has a negative impact on their work efficiency and health. Furthermore, due to a lack of communication between employees and limited opportunities to receive professional counseling and support, sufficient measures are not being taken to reduce stress and improve work efficiency. To solve these issues, a system is needed that can accurately analyze employees' emotional states and provide appropriate feedback and advice in real time.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1360] In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts or medical institutions as needed, and means for supporting the user's mental health and improving work efficiency, thereby making it possible to maintain the mental health of employees working at logistics centers and improve work efficiency.

[1361] "User" refers to any individual or employee who uses the System.

[1362] "Voice data" means electronic data that is a recording of what a user says.

[1363] "Text data" refers to data obtained by converting voice data into character information.

[1364] "Sentiment analysis" is a technique that applies natural language processing algorithms to text data to identify a user's emotional state.

[1365] "Feedback" refers to advice or comments provided to users based on the results of sentiment analysis.

[1366] "Everyday scene simulation" is a function that allows users to simulate real-life situations.

[1367] An "empathy community" is a community where users in the same position or circumstances can support each other.

[1368] "Matching" refers to connecting appropriate users within a community of like-minded people.

[1369] "Means for connecting with experts and medical institutions" refers to means for providing the necessary professional support based on the results of the user's sentiment analysis.

[1370] "Mental health" refers to a user's state of psychological and emotional well-being.

[1371] "Work efficiency" refers to the degree to which logistics center employees perform their work effectively and efficiently.

[1372] This invention relates to a system for supporting the mental health of logistics center employees and improving their work efficiency. Specifically, the system receives and analyzes users' voice data and provides appropriate feedback and advice. It also provides a simulation function, builds a community of empathy among employees, and connects them to experts and medical institutions as needed.

[1373] Hardware and Software

[1374] The system uses hardware such as a smartphone, microphone, and robot, as well as Python, the SpeechRecognition library, the transformers library, and Google's speech recognition API.

[1375] Data processing and calculation

[1376] First, when a user speaks to a smartphone or robot, the voice data is recorded. For example, suppose the user says, "Work has been tough lately and I'm feeling tired." An application on the smartphone receives this voice data and converts it into text data via Google's speech recognition API. Next, this text data undergoes sentiment analysis using a natural language processing algorithm. Specifically, emotional states such as "fatigue" and "stress" are identified using the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library.

[1377] Based on the results of this sentiment analysis, the server then provides appropriate feedback and advice to the user, such as "Take better care of yourself and take a short break when it comes to relaxation methods."

[1378] Specific examples

[1379] User A says into his smartphone, "Work has been tough lately and I'm feeling tired." The smartphone records this voice data and converts it into text data using Google's speech recognition API. This text data is sent to a server, where sentiment analysis is performed using the BERT model in the Transformers library. As a result of the analysis, "fatigue" and "stress" are identified, and feedback advice is provided, such as "Regarding relaxation methods, you should take better care of yourself. I recommend taking a short break."

[1380] Similarly, when a user uses the simulation function to practice for an interview, the server analyzes the user's responses to questions and provides specific advice in real time, such as, "Your voice is too quiet, so speak louder and with more confidence."

[1381] An example of a prompt is:

[1382] "Work has been hard lately and I'm tired."

[1383] "How can I speak more confidently when answering interview questions?"

[1384] Examples include:

[1385] In this way, the present invention provides a specific system for improving the mental health and work efficiency of employees in a distribution center.

[1386] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1387] Step 1:

[1388] The device (smartphone or robot) records the user's voice data. When the user speaks into the device, the device acquires and records this voice data. The input is the user's voice, and the output is voice data.

[1389] Step 2:

[1390] The device converts the recorded voice data into text data using Google's speech recognition API. Specifically, the device sends the voice data to the API and receives the result as text data. The input is voice data, and the output is the converted text data. It is then sent to the server.

[1391] Step 3:

[1392] The server performs sentiment analysis on the received text data. For this analysis, it uses the BERT model (nlptown / bert-base-multilingual-uncased-sentiment) from the transformers library. The input is the text data, and the output is the result of sentiment analysis (e.g., "fatigue" or "stress").

[1393] Step 4:

[1394] The server generates appropriate feedback and advice based on the results of the sentiment analysis. This feedback is provided using a generative AI model. For example, specific advice such as "Take better care of yourself and take a short break when it comes to relaxation methods" is provided. The input is the result of the sentiment analysis, and the output is the feedback text.

[1395] Step 5:

[1396] The server sends the generated feedback to the terminal for providing to the user. The terminal provides the feedback to the user by displaying or reading it out loud. The input is the feedback sentence, and the output is what is presented to the user.

[1397] Step 6:

[1398] When a user wishes to simulate an everyday situation, the server selects a simulation scenario and sends it to the terminal. The user then performs the simulation, speaking and acting within it. The input is the simulation scenario, and the output is the user's response behavior data.

[1399] Step 7:

[1400] The server analyzes the user's simulated response data and provides feedback on how to improve performance. For example, it provides specific advice such as, "Your voice is too quiet. Speak louder and more confidently." The input is the simulated response data, and the output is feedback for improvement.

[1401] Step 8:

[1402] If necessary, the server connects users to empathy communities and matches them with users in similar situations, allowing users to exchange information and receive support from other users. The input is the user's emotional state and profile information, and the output is the matching results of empathy communities.

[1403] Step 9:

[1404] The server provides a means to connect the user to specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to an appropriate specialist so that they can receive support. The input is the emotion analysis results and the user's needs, and the output is connection information to specialists or medical institutions.

[1405] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1406] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[1407] Analyzing user sentiment and providing feedback

[1408] The system's operation begins with receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server then uses a speech recognition engine to convert this voice data into text data. The emotion engine then analyzes the text and voice data to identify the user's emotional state in detail. For example, if the user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server generates appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about ways to relax?"

[1409] Skill improvement through simulation functions

[1410] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server and converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses. For example, specific advice such as "Your voice is too quiet, so speak more loudly and with more confidence" may be provided.

[1411] Building a community of empathy

[1412] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[1413] Providing professional support

[1414] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[1415] Specific examples

[1416] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1417] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[1418] The processing flow will be explained below.

[1419] Analyzing user sentiment and providing feedback

[1420] Step 1:

[1421] User voice input

[1422] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[1423] Step 2:

[1424] Sending audio data

[1425] The device records the user's voice data and sends the data to the server.

[1426] Step 3:

[1427] Converting audio data to text

[1428] The server converts the received voice data into text data using a speech recognition engine. For example, the generated text might say, "Work has been tough and I'm tired lately."

[1429] Step 4:

[1430] Sentiment analysis of text data

[1431] The server uses an emotion engine to perform a detailed analysis of the user's emotional state from the text and voice data, identifying emotions such as "fatigue" and "stress."

[1432] Step 5:

[1433] Feedback Generation

[1434] The server generates appropriate feedback and advice based on the analyzed emotional state, for example, a suggestion such as "Would you like to know more about how to relax?"

[1435] Step 6:

[1436] Providing Feedback

[1437] The server provides the generated feedback to the user through the terminal.

[1438] Skill improvement through simulation functions

[1439] Step 1:

[1440] Presenting a simulation scenario

[1441] If the user wants to practice their communication skills, the device will present a simulation scenario, such as an interview simulation with the message "Please introduce yourself."

[1442] Step 2:

[1443] Collecting and sending simulated responses

[1444] The user responds verbally according to the scenario, and the device records the voice data and sends it to the server.

[1445] Step 3:

[1446] Response text transformation and parsing

[1447] The server converts the voice data of the scenario response into text data and analyzes it using the emotion engine. For example, a response such as "I work at X Corporation" is analyzed.

[1448] Step 4:

[1449] Generate and provide improvement feedback

[1450] The server generates feedback on how to improve the user's response and speaking style, and provides it to the user via the device. For example, the server may provide advice such as, "Your voice is too quiet, so speak louder and with more confidence."

[1451] Building a community of empathy

[1452] Step 1:

[1453] Check user profile and emotional state

[1454] The server collects and checks the user's profile information and sentiment analysis results.

[1455] Step 2:

[1456] Matching with sympathetic communities

[1457] The server searches for other users with the same situation or emotional state and matches them together, for example, matching users who are suffering from work stress.

[1458] Step 3:

[1459] Providing community features

[1460] The server provides matched users with chat rooms and video calls, where they can share their experiences and feelings.

[1461] Providing professional support

[1462] Step 1:

[1463] Connecting to experts based on sentiment analysis results

[1464] Based on the results of the emotion analysis, the server connects the user to the appropriate specialist or medical institution as needed.

[1465] Step 2:

[1466] Providing integrated support

[1467] The server provides a link to professional support (such as psychological counseling) according to the user's needs.

[1468] Specific examples

[1469] Step 1:

[1470] User A's voice input

[1471] User A says to the device, "Work has been tough lately and I'm tired."

[1472] Step 2:

[1473] Sending voice data and converting it to text

[1474] The terminal transmits the voice data to the server, which converts it into text data.

[1475] Step 3:

[1476] Sentiment Analysis and Feedback Generation

[1477] The server uses an emotion engine to analyze emotions such as "fatigue" and "stress" and provides feedback such as "Would you like to know more about ways to relax?"

[1478] Step 4:

[1479] Presenting a simulation scenario

[1480] When user A requests to practice for an interview, the device presents a simulation scenario that asks the user to "introduce themselves."

[1481] Step 5:

[1482] Collecting and analyzing responses and providing improvement feedback

[1483] User A responds, "I work for X Corporation," and the voice data is sent to the server for analysis. The server then generates and provides improvement feedback, such as, "Speak louder and more confidently."

[1484] Step 6:

[1485] Matching with sympathetic communities

[1486] The server matches User A with User B, who is also suffering from work-related stress, and provides a chat room.

[1487] Step 7:

[1488] Providing professional support

[1489] Based on the results of the emotion analysis, the server introduces user A to a psychological counselor.

[1490] In this way, the system starts with the user's voice data and concretely implements each step of analysis, feedback, simulation, community formation, and professional support.

[1491] Example 2

[1492] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1493] Conventional emotion analysis systems have limitations in their technology for analyzing emotions from user voice data, resulting in problems such as insufficient real-time emotion analysis or appropriate feedback. They also lack simulation functions to help users improve their everyday communication skills, or effective methods for matching users with other users in similar situations. Furthermore, they lack a means for smoothly connecting users with experts or medical institutions when needed. To solve these problems, a system is needed that can accurately analyze a user's emotional state in real time and provide appropriate feedback and advice based on the analysis results.

[1494] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1495] In this invention, the server includes: means for receiving a user's voice data and converting it into text data; means for analyzing the user's emotions from the text data; means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions; means for allowing the user to simulate everyday situations; means for forming an empathy community with other users and matching users with similar situations; means for connecting to experts or medical institutions as needed; means for analyzing the user's emotional state in real time and dynamically generating feedback based on the analysis results; means for identifying a detailed emotional state using the user's speech data and providing specific advice based on the identified emotions; means for forming an empathy community that supports communication between users and promotes empathy; and means for simulating specific situations such as interviews and providing dynamic, real-time feedback on the user's responses. This enables the user's emotions to be analyzed with high accuracy and appropriate feedback to be provided in real time. Furthermore, it makes it easier for users to improve their everyday communication skills and share empathy with other users.

[1496] 1. "Voice Data" means data that is a digital recording of the voice spoken by a User.

[1497] 2. "Text data" means character string data converted from voice data using voice recognition technology.

[1498] 3. "Sentiment analysis" is the process of inferring and identifying a user's emotional state from input text or voice data.

[1499] 4. "Feedback" means advice or suggestions provided to the user based on the results of the analysis.

[1500] 5. "Simulation function" means a function that allows users to virtually experience specific scenes or situations and practice them.

[1501] 6. An "empathy community" is an online space where users in the same position or emotional state can share support and empathy with each other.

[1502] 7. "Matching" is the process of connecting users with other users who are in a similar state based on their emotional state and profile.

[1503] 8. "Means for connecting with specialists and medical institutions" means the means for referring and contacting users with appropriate specialists and medical institutions when necessary.

[1504] 9. "Real-time" refers to the ability to respond to user actions and inputs immediately, with immediate processing and results returned.

[1505] 10. "Means for generating dynamic feedback" refers to technology for generating appropriate feedback according to the user's emotional state and behavior.

[1506] 11. "Specific Advice" means detailed, actionable advice tailored to a user's specific situation and feelings.

[1507] 12. "Means to support communication" are technologies that promote and facilitate dialogue and interaction between users.

[1508] 13. "Simulating" situations such as "interviews" is a system feature that allows users to experience and practice hypothetical interview situations.

[1509] 14. "Means for providing dynamic, real-time feedback to users' responses" refers to technology that provides immediate feedback to users based on their responses and actions.

[1510] MODE FOR CARRYING OUT THE INVENTION

[1511] This invention relates to a system that analyzes a user's emotions in real time and provides feedback and advice according to changes in emotions. This system also includes a means for users to simulate everyday situations, form empathy communities with other users, and collaborate with experts and medical institutions as needed.

[1512] Analyzing user sentiment and providing feedback

[1513] The system begins by receiving the user's voice data. When the user speaks into the device, the device records the voice data and sends it to the server. The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the voice data into text data. An emotion engine (e.g., IBM Watson Tone Analyzer API) then analyzes the text and voice data to identify the user's emotional state in detail.

[1514] For example, if a user says, "Work has been tough and I'm tired lately," the emotion engine will perform a detailed analysis of "fatigue" and "stress." Based on the results, the server will generate appropriate feedback and advice for the user, such as a suggestion like, "Would you like to know more about how to relax?"

[1515] Skill improvement through simulation functions

[1516] This system allows users to simulate real-life communication situations. For example, if a user wants to practice for an interview, the device will present a simulation scenario. After the user answers the questions, the voice data is sent back to the server, where it is converted into text data by a speech recognition engine. The server then analyzes this text data, and an emotion engine evaluates the emotional state of the voice and text, providing accurate feedback based on the user's responses.

[1517] For example, specific advice is provided such as, "Your voice is too quiet, so speak louder and with more confidence."

[1518] Building a community of empathy

[1519] Based on a user's emotional state and profile information, the server searches for and matches users in the same position or situation. This creates an empathetic community where users can support each other. For example, users who are suffering from work stress can be matched and share their experiences through chat rooms and video calls.

[1520] Providing professional support

[1521] Based on the results of the user's emotion analysis, the server will provide a means to connect the user to appropriate specialists or medical institutions as needed. For example, if psychological counseling is determined to be necessary, the server will refer the user to a specialist so that the user can receive appropriate support.

[1522] Specific examples

[1523] For example, if User A says, "Work has been tough and I'm tired lately," the device sends this voice data to the server, which converts it into text data. The emotion engine then performs a detailed analysis of emotions such as "fatigue" and "stress." The server then generates specific feedback, such as "Would you like to know more about relaxation techniques?" and provides it to User A. When User A simulates an interview, the server analyzes the responses and, based on the emotion engine's detailed evaluation, provides real-time suggestions for improvement, such as "You should speak with more confidence." User A is then matched with User B, who is also suffering from work stress, and the two can share their experiences in a community of empathy. At the same time, they are also provided with guidance to receive professional counseling, if necessary.

[1524] Example of input prompt for generative AI model

[1525] Prompt: "Please explain sentiment analysis and feedback systems. Please provide examples."

[1526] In this way, the system of the present invention seamlessly realizes user emotion analysis, simulation of everyday situations, building of empathetic communities, and provision of professional support.

[1527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1528] Step 1:

[1529] User utterances and voice data collection

[1530] The user speaks to the device, for example, saying, "Work has been tough lately and I'm tired."

[1531] The device records the user's speech using a built-in microphone. The voice data (input) is temporarily stored in the device's memory in digital format (output).

[1532] Step 2:

[1533] Sending audio data

[1534] The device encrypts the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1535] The voice data (input) is encrypted and sent to the server (output).

[1536] Step 3:

[1537] Converting audio data to text

[1538] The server uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data.

[1539] Audio data (input) is converted into text data (output).

[1540] Step 4:

[1541] Emotion analysis

[1542] The server uses an emotion engine (specifically, IBM Watson Tone Analyzer API) to perform emotion analysis of text data and voice data.

[1543] From the text data and voice data (input), the user's emotional state (e.g., fatigue, stress) is analyzed (output).

[1544] The server stores the analysis results in an internal database, including the user ID, analysis results, and timestamp.

[1545] Step 5:

[1546] Generate feedback

[1547] The server generates appropriate feedback and advice for the user based on the results of the emotion analysis.

[1548] Based on the analysis results (input), a specific feedback message (e.g., "Would you like to know more about how to relax?") is generated (output).

[1549] Step 6:

[1550] Providing Feedback

[1551] The server sends the generated feedback message to the terminal.

[1552] Feedback messages (input) are forwarded to the terminal (output).

[1553] Step 7:

[1554] View Feedback

[1555] The device displays the received feedback message to the user.

[1556] Feedback messages received on the device (input) are displayed on the screen as popups or notifications (output).

[1557] Step 8:

[1558] Presenting a simulation scenario

[1559] The server generates a simulation scenario such as an interview and transmits it to the terminal.

[1560] The simulation scenario (input) arrives at the terminal and is displayed to the user (output).

[1561] Step 9:

[1562] Collect user responses

[1563] When the user speaks to the simulation, the terminal records the voice data and transmits it back to the server.

[1564] The user's response voice data (input) is sent to the server and converted into text data (output).

[1565] Step 10:

[1566] Analyzing simulation responses and providing feedback

[1567] The server converts the voice data into text data and then analyzes it using an emotion engine.

[1568] It analyzes text data (input) and generates specific feedback (e.g., "You should speak more confidently") (output).

[1569] The server sends a feedback message to the terminal, which displays it to the user.

[1570] Step 11:

[1571] Building a community of empathy

[1572] The server searches for and matches users with other users in similar states based on their emotional state and profile information.

[1573] Matched user information (output) is generated from user information and sentiment analysis results (input).

[1574] Step 12:

[1575] Communication Support

[1576] The server provides an environment where matched users can share experiences through chat rooms and video calls.

[1577] The means of communication between users (links and connection information) (output) are displayed on the terminal.

[1578] Step 13:

[1579] Providing access to specialists and medical institutions

[1580] Based on the analysis results, the server provides a means to connect users to appropriate specialists or medical institutions as needed.

[1581] The analysis results (input) are used to provide contact information and appointment links for specialists and medical institutions (output).

[1582] In this way, this invention not only analyzes a user's emotions in real time using their voice data and provides feedback and advice based on the results, but also realizes a seamless system that includes simulation functions, an empathetic community, and even professional support.

[1583] This allows users to understand their emotional state in three dimensions, utilize feedback to improve their ability to respond to each situation, and access empathetic communities and professional support to solve problems and grow without feeling isolated.

[1584] (Application example 2)

[1585] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1586] While conventional emotion analysis systems can analyze users' emotions in real time, they lack mechanisms for providing appropriate relaxation content or expert support based on the results. As a result, they lack specific approaches to improving users' mental health. Furthermore, they lack the ability to form empathetic communities where users can support each other, and the ability to support users in improving their skills through simulation functions.

[1587] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving the user's voice data and converting it into text data, means for analyzing the user's emotions from the text data, means for providing appropriate feedback and advice when the user's emotions change based on the analyzed emotions, means for providing a function that allows the user to simulate everyday situations, means for forming an empathy community with other users and matching users in the same situation, means for connecting with experts and support organizations as needed, and means for providing appropriate relaxation content based on the results of the user's emotion analysis. As a result, individually optimized relaxation content is provided based on the results of the user's emotion analysis, enabling users to support each other through the empathy community and receive support from experts.

[1588] "Voice data" refers to data that is a digital recording of what a user has said.

[1589] "Text data" refers to data obtained by converting voice data into character information.

[1590] "Sentiment analysis" is the process of analyzing a user's emotional state from their text and voice data.

[1591] "Feedback" refers to advice and information provided based on the results of user sentiment analysis.

[1592] The "simulation function" allows users to virtually recreate real-world situations for practice and experience.

[1593] An "empathy community" is an online community where users with similar positions and experiences can support each other.

[1594] "Matching" is the process of connecting users with the same situations or concerns within a community of empathy.

[1595] An "expert" is a professional with specific expertise, such as a psychological counselor or healthcare provider.

[1596] "Support organizations" are organizations that provide assistance, including medical institutions and specialized counseling centers.

[1597] "Relaxation content" refers to digital content such as music, meditation guides, and stories that are designed to reduce stress and promote relaxation for users.

[1598] This invention is a system that analyzes users' emotions in real time, provides personalized relaxation content based on the analysis results, creates a community of empathy, and connects them to experts and support organizations as needed.

[1599] Audio data collection and text conversion

[1600] The server receives the voice data spoken by the user into the device. The device records this voice data and sends it to the server. The server uses a speech recognition engine and the "speech_recognition" library to convert this voice data into text data. This library converts voice to text with high accuracy.

[1601] Emotion analysis

[1602] The server then analyzes the user's emotions using the converted text and audio data. This process uses a generative AI model with the "transformers" library to output an emotion label (e.g., "happiness," "sadness," "stress," etc.) and its score.

[1603] Providing personalized content

[1604] Based on the results of the emotion analysis, the server provides relaxation content appropriate for the user. For example, if the user is analyzed as feeling stressed, the server will provide content such as relaxation music or meditation guides. This selection is based on the emotion label and score.

[1605] Forming and matching communities of empathy

[1606] The server searches for and matches users with other users in similar situations based on their emotional state and profile information, creating a mutual support community that encourages users to share their experiences through chat rooms and video calls.

[1607] Providing professional support

[1608] If necessary, the server also provides a function to connect the user to experts or support organizations. For example, if the results of the emotion analysis indicate that psychological counseling is necessary, the user will be referred to an appropriate expert and receive appropriate support.

[1609] Specific examples

[1610] For example, if a user says, "I've been feeling a lot of stress at work lately," the device sends this voice data to the server. The server converts the voice data into text and then performs emotion analysis using the "transformers" library, calculating the emotion label "stress" and a high score. The server then provides the user with relaxation music and deep breathing guides, and attempts to match them with other users in similar situations. If necessary, it also provides guidance to professional counseling.

[1611] Prompt Sentence Examples

[1612] For example, the following prompts can be used to recommend relaxation music based on the user's emotional state:

[1613] User's emotional status: Stressed (score: 0.85)

[1614] Suitable content for users: Deep breathing guides, Stress relief stories

[1615] In this way, we have created a system that provides relaxation content that is individually optimized based on the user's emotional state, enables users to support each other through an empathetic community, and enables them to receive support from experts.

[1616] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1617] Step 1: Collecting audio data

[1618] The user speaks emotional content into their device, and this voice data is recorded through the device's microphone. The input is the user's speech, and the output is digital voice data.

[1619] Step 2: Sending audio data to the server

[1620] The device sends the recorded voice data to the server. The input is the digital voice data, and the output is the voice data transferred to the server.

[1621] Step 3: Convert audio data to text

[1622] The server converts the received voice data into text data using the "speech_recognition" library. The input is the voice data sent to the server, and the output is text data. This conversion process is performed using voice recognition technology.

[1623] Step 4: Sentiment Analysis

[1624] The server uses generative AI models from the "transformers" library to analyze user emotions from text and audio data. The input is text data, and the output is emotion labels and their scores. In this process, natural language processing algorithms identify emotional states.

[1625] Step 5: Generate feedback

[1626] The server selects relaxation content appropriate for the user based on the emotion analysis results. For example, if an emotion label indicating stress is detected, content such as relaxation music or meditation guides will be selected. The input is the emotion label and its score, and the output is a list of appropriate relaxation content.

[1627] Step 6: Providing content

[1628] The server provides selected relaxation content to users. The input is a list of relaxation content, and the output is links and data of the content that users can access.

[1629] Step 7: Matching with empathetic communities

[1630] The server matches users with other users based on their emotional state and profile information. The input is the user's emotional state and profile information, and the output is the matching results. This process searches for users in similar situations and forms a community of empathy.

[1631] Step 8: Providing professional support

[1632] If necessary, the server will refer the user to a specialist or support organization based on the results of the emotion analysis. The input is the emotion analysis result and the user's health status information, and the output is information on the referral to the appropriate specialist or support organization. This process provides a path for the user to smoothly receive support.

[1633] Through these steps, a system is realized that provides individually optimized relaxation content based on the results of user emotion analysis, enables users to support each other through empathetic communities, and allows them to receive support from experts.

[1634] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1635] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1636] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1637] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1638] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1639] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1640] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1641] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1642] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1643] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1644] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1645] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1646] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1647] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1648] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1649] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1650] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1651] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1652] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1653] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1654] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1655] The following is further disclosed regarding the above embodiment.

[1656] (Claim 1)

[1657] means for receiving user voice data and converting it into text data;

[1658] A means of analyzing user emotions from text data,

[1659] A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and

[1660] A means to provide users with the ability to simulate everyday situations;

[1661] A way to form empathy communities with other users and match users in the same position,

[1662] and the means to connect with specialists and medical institutions as needed.

[1663] A system including:

[1664] (Claim 2)

[1665] 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using natural language processing algorithms.

[1666] (Claim 3)

[1667] 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on areas for improvement.

[1668] "Example 1"

[1669] (Claim 1)

[1670] means for receiving user voice data and converting it into text data;

[1671] A means of analyzing user emotions from text data,

[1672] A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and

[1673] A means to provide users with the ability to simulate everyday situations;

[1674] A way to form empathy communities with other users and match users in the same position,

[1675] and the means to connect with specialists and medical institutions as needed.

[1676] A means for recording and transmitting user voice data to a server;

[1677] means for converting voice data into text data using a voice recognition engine;

[1678] a means for applying natural language processing algorithms to identify the emotional state of a user;

[1679] a means for generating and providing feedback to the user based on the analysis results;

[1680] a means for presenting simulation scenarios and collecting and analyzing user responses;

[1681] A system including:

[1682] (Claim 2)

[1683] 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using natural language processing algorithms.

[1684] (Claim 3)

[1685] 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on areas for improvement.

[1686] "Application Example 1"

[1687] (Claim 1)

[1688] means for receiving user voice data and converting it into text data;

[1689] A means of analyzing user emotions from text data,

[1690] A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and

[1691] A means to provide users with the ability to simulate everyday situations;

[1692] A way to form empathy communities with other users and match users in the same position,

[1693] and the means to connect with specialists and medical institutions as needed.

[1694] Measures to support users' mental health and improve work efficiency,

[1695] A system including:

[1696] (Claim 2)

[1697] 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using natural language processing algorithms.

[1698] (Claim 3)

[1699] 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on areas for improvement.

[1700] "Example 2: Combining Emotion Engines"

[1701] (Claim 1)

[1702] means for receiving user voice data and converting it into text data;

[1703] A means of analyzing user emotions from text data,

[1704] A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and

[1705] A means to provide users with the ability to simulate everyday situations;

[1706] A way to form empathy communities with other users and match users in the same position,

[1707] and the means to connect with specialists and medical institutions as needed.

[1708] A means for analyzing the user's emotional state in real time and dynamically generating feedback based on the analysis results;

[1709] A means for identifying a detailed emotional state of a user using speech data and providing specific advice according to the identified emotional state;

[1710] A means to support communication between users and form a community of empathy that promotes empathy,

[1711] A means of simulating specific scenarios, such as interviews, and providing dynamic, real-time feedback to users' responses;

[1712] A system including:

[1713] (Claim 2)

[1714] 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using natural language processing algorithms.

[1715] (Claim 3)

[1716] 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on areas for improvement.

[1717] "Application example 2 when combining emotion engines"

[1718] (Claim 1)

[1719] means for receiving user voice data and converting it into text data;

[1720] A means of analyzing user emotions from text data,

[1721] A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and

[1722] A means to provide users with the ability to simulate everyday situations;

[1723] A way to form empathy communities with other users and match users in the same position,

[1724] and the means to connect with experts and support organizations as needed.

[1725] A means for providing appropriate relaxation content based on the result of the user's emotion analysis;

[1726] A system including:

[1727] (Claim 2)

[1728] 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using natural language processing algorithms.

[1729] (Claim 3)

[1730] 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on areas for improvement. [Explanation of symbols]

[1731] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving user voice data and converting it into text data; A means of analyzing user emotions from text data, A means for providing appropriate feedback and advice when there is a change in the user's emotions based on the analyzed emotions; and A means to provide users with the ability to simulate everyday situations; A way to form empathy communities with other users and match users in the same position, and the means to connect with specialists and medical institutions as needed. A system including:

2. 2. The system according to claim 1, further comprising means for converting user voice data into text in real time and performing sentiment analysis using a natural language processing algorithm.

3. 2. The system according to claim 1, further comprising means for analyzing a user's performance in the simulation function and providing feedback on improvements.

Images