System

Abstract

To provide a system.SOLUTION: A system includes: means for integrating sentences input into a communication tool with a generative AI; means for determining the feeling of a user who has created the sentences by the generative AI; means for analyzing the past communication data and the feeling of the user corresponding to the communication data; means for generating a feedback for the sentences on the basis of the determined feeling of the user and the analysis results of the analyzing means; and means for providing the user with the generated feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] When communicating with superiors or subordinates, it is important to understand not only the content of the communication but also the other person's emotional state and communicate at the appropriate timing. However, currently, it is difficult to accurately grasp the other person's emotional state and determine the optimal timing for communication. [Means for solving the problem]

[0005] This invention connects text entered into a communication tool to a generative AI, which then determines the emotion of the person who wrote the text.The results of this determination are then visualized in an app, enabling the determination of the optimal timing for communication.Furthermore, the generative AI determines the optimal timing for communication based on the emotion determination results and provides feedback to the user based on the emotion determination results. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention uses GOOGLE CHAT (registered trademark) as a communication tool. Text entered into Google (registered trademark) chat is linked to a generative AI, which determines human emotions from the text. The results are visualized in the app, and the user can use these results to select the timing of communication. Specifically, if the other person is showing positive emotions such as joy or relief, it is recommended that they communicate at that time.

[0029] "Example 2"

[0030] The generative AI also determines the optimal timing for communication based on the emotion assessment results. Specifically, the AI ​​learns past communication data and patterns of emotional fluctuations, and predicts the optimal timing for communication based on that. This prediction result is also presented to the user on the app, allowing the user to use it as a reference when selecting the timing for communication.

[0031] "Example 3"

[0032] Furthermore, the generative AI provides feedback to the user based on the emotion assessment results. Specifically, the AI ​​analyzes which emotions the user has displayed in the past that have resulted in the most effective communication, and provides feedback based on the results. This feedback allows the user to improve their own communication skills.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: User types a message into Google Chat.

[0036] Step 2: The input text is linked to the generative AI.

[0037] Step 3: The generative AI determines human emotions from the text.

[0038] Step 4: The results are visualized in the app.

[0039] Step 5: The user selects the timing of communication based on the visualized results.

[0040] "Example 2"

[0041] Step 1: The generative AI learns past communication data and patterns of emotional fluctuations.

[0042] Step 2: The AI ​​predicts the optimal timing for communication based on the learned data.

[0043] Step 3: The prediction results are presented to the user on the app.

[0044] Step 4: The user selects the timing of communication based on the presented prediction results.

[0045] "Example 3"

[0046] Step 1: Generative AI analyzes which emotions the user has displayed in the past that resulted in the most effective communication.

[0047] Step 2: The AI ​​provides feedback to the user based on the analysis results.

[0048] Step 3: Users use the provided feedback to improve their communication skills.

[0049] Example 1

[0050] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0051] With conventional communication tools, it was difficult to accurately grasp the emotions of the other person, making it difficult to communicate at the right time. In addition, there was a lack of means to provide users with the results of emotion assessment, and users were not given enough support to communicate effectively.

[0052] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0053] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for displaying the determination result on the user's device, and means for providing information for the user to select the timing of communication. This enables the user to accurately grasp the emotion of the other person and communicate effectively at the appropriate time.

[0054] A "communication tool" is software or an application that allows users to send and receive messages.

[0055] "Generative AI" is an artificial intelligence model that uses natural language processing technology to analyze input text and perform specific tasks (such as determining emotions).

[0056] "Emotion determination" is the process of identifying the emotions (e.g., joy, relief, anger, sadness, etc.) of the person who wrote the input text.

[0057] An "application" is a software program available to users that provides specific functions or services.

[0058] "User device" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0059] The "judgment result" is the emotion identification result obtained by the generative AI analyzing the input text.

[0060] "Visualization" is the process of visually displaying data or information, providing it in a way that is easy for users to understand.

[0061] "Information for selecting timing" refers to information such as emotion determination results that are provided to the user to determine the optimal timing for communicating with the other person.

[0062] This invention is a system that links text entered into a communication tool with generative AI, determines emotions, and provides the results to the user. Specifically, the server, terminal, and user elements work together.

[0063] The server receives sentences entered by the user into the communication tool. For example, if the user sends a message such as "I had so much fun today!", this message arrives at the server. The server then sends the received message to a generative AI model (e.g., OpenAI's GPT-4™). This generative AI model uses natural language processing technology to analyze the message and determine its sentiment.

[0064] The generative AI model determines the emotion from the message sent. For example, it determines the emotion "joy" from the message "I had so much fun today!" This process uses a machine learning algorithm. After the generative AI model determines the emotion, it sends the result back to the server. The server then sends this determination result to the user's device.

[0065] The user's device visualizes the emotion determination results received from the server. For example, the user's device might display "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[0066] As a concrete example, consider the case where a user sends a message to Google Chat saying, "I had so much fun today!" This message is sent to the server and passed to a generative AI model. The generative AI model determines the emotion "joy" from this message. The result of this determination is sent to the application, and the message "The other person is feeling happy" is displayed on the user's device. The user can use this information to decide when to communicate with the other person.

[0067] An example of a prompt might be:

[0068] Determine the emotion from the message, "Today was so much fun!"

[0069] By inputting this prompt into the generative AI model, the model determines the emotion "joy," thus providing support for users to communicate effectively.

[0070] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0071] Step 1:

[0072] The server receives the text entered by the user into the communication tool.

[0073] Specifically, when a user sends a message such as "Today was so much fun!", this message is received by the server.

[0074] Input: Text entered by the user into the communication tool

[0075] Output: The text received by the server

[0076] Step 2:

[0077] The server sends the received sentences to the generative AI model.

[0078] Specifically, the server sends the received message to a generative AI model (e.g., GPT-4), which then analyzes the message using natural language processing techniques.

[0079] Input: The text received by the server

[0080] Output: The sentence sent to the generative AI model

[0081] Step 3:

[0082] The generative AI model determines the sentiment from the text sent.

[0083] Specifically, the generative AI model analyzes the message "I had so much fun today!" and determines the emotion "joy." This process uses machine learning algorithms.

[0084] Input: The sentence sent to the generative AI model

[0085] Output: Determined emotion (e.g., "joy")

[0086] Step 4:

[0087] The generative AI model returns the emotion determination results to the server.

[0088] Specifically, the generative AI model determines the emotion of "joy" and then sends the result back to the server.

[0089] Input: Determined emotion

[0090] Output: Emotion judgment result sent back to the server

[0091] Step 5:

[0092] The server transmits the emotion determination result to the user's terminal.

[0093] Specifically, the server sends the emotion determination results received from the generative AI model to the user's device.

[0094] Input: Emotion judgment result returned to the server

[0095] Output: Emotion judgment result sent to the user's device

[0096] Step 6:

[0097] The user's device visualizes the emotion determination results.

[0098] Specifically, the user's device will display a message saying, "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[0099] Input: Emotion judgment result sent to the user's device

[0100] Output: Visualized emotion judgment result (e.g., "The other person is feeling happy")

[0101] In this way, the system provides support for users to communicate effectively.

[0102] (Application example 1)

[0103] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In physical stores, in order to communicate smoothly with customers, it is important to properly understand the customer's emotions and provide customer service at the optimal timing. However, with conventional methods, it is difficult for store staff to accurately read the customer's emotions, which can result in a decline in the quality of customer service. To solve this problem, a system is needed that analyzes customer emotions in real time and provides appropriate feedback.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0106] In this invention, the server includes a means for linking text entered into the communication tool to a generative AI, a means for using the generative AI to determine the emotions of the person who wrote the text, and a means for visualizing the determination results in an application. This makes it possible to input the content of conversations with customers in a physical store and analyze their emotions in real time. By including a means for recommending the timing of customer service based on the analysis results, store staff can properly understand customer emotions and provide customer service at the optimal time.

[0107] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[0108] "Generative AI" is an artificial intelligence technology that analyzes input data and generates appropriate output for a specific task.

[0109] "Means of collaboration" are methods and technologies for exchanging data between different systems and software, making it mutually usable.

[0110] A "means for determining" is a method or technology for analyzing input data and evaluating the results based on specific criteria.

[0111] "Visualization means" refers to methods and techniques for displaying analysis results and data in a way that is easy for users to understand.

[0112] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[0113] "Customers" are people who visit a physical store to purchase goods or services.

[0114] "Conversation content" refers to the words and sentences exchanged between the customer and the store clerk.

[0115] "Means for analyzing emotions in real time" refers to methods or technologies for instantly analyzing input conversation content and determining customer emotions.

[0116] A "means for recommending the timing of customer service" is a method or technology for instructing store staff to respond to customers at the optimal timing based on the analysis results.

[0117] In order to implement the present invention, the following system configuration and program are required.

[0118] System Configuration

[0119] 1. Hardware:

[0120] Smartphone: Used by store clerks to input conversations with customers and check the analysis results.

[0121] Server: Provides the computational resources to run generative AI models and perform sentiment analysis.

[0122] 2. Software:

[0123] Communication tool: An application that allows store clerks to enter conversations with customers.

[0124] Generative AI model: An artificial intelligence technology that analyzes input text and determines emotions.

[0125] Google Cloud Natural Language API: An API for analyzing the sentiment score of a sentence.

[0126] OpenAI API: An API for running generative AI models and determining emotions.

[0127] Program processing

[0128] The server processes the data in the following steps:

[0129] 1. Data Entry:

[0130] The store clerk enters the details of the conversation with the customer into the communication tool on their smartphone.

[0131] 2. Emotion analysis:

[0132] The server sends the input conversation content to the Google Cloud Natural Language API and obtains an emotion score.

[0133] In addition, it uses the OpenAI API to perform detailed sentiment determination of the input text.

[0134] 3. Visualizing the results:

[0135] The server sends the analysis results to a smartphone application and displays the visualized results to the store clerk.

[0136] Based on the analysis, store staff are encouraged to continue serving customers if they are displaying positive emotions.

[0137] Specific examples

[0138] For example, consider the case where a store clerk inputs the following conversation content:

[0139] Type: "It's a beautiful day today. Can I see your new items?"

[0140] The server sends this input to the Google Cloud Natural Language API to obtain an emotion score, then uses the OpenAI API to generate the following prompt sentence and perform emotion determination:

[0141] Example prompt sentence:

[0142] Determine the sentiment of the following sentences: The weather is very nice today. Can you show me your new products?

[0143] If the analysis results in a positive emotion, the smartphone application will display the message, "The customer is expressing a positive emotion. Please continue serving the customer."

[0144] In this way, store staff can properly understand the customer's emotions and provide service at the optimal time.

[0145] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0146] Step 1:

[0147] The user (store clerk) inputs the contents of the conversation with the customer into the communication tool on the smartphone. The inputted contents of the conversation are saved on the smartphone as text data.

[0148] Step 2:

[0149] The device (smartphone) sends the entered conversation content to the server. The data sent is the conversation content in text format.

[0150] Step 3:

[0151] The server sends the received conversation content to the Google Cloud Natural Language API and obtains the sentiment score. The input data is the conversation content in text format, and the output data is the sentiment score (positive, negative, neutral, etc.).

[0152] Step 4:

[0153] The server uses the OpenAI API to perform detailed emotion determination based on the emotion score. The server generates a prompt from the conversation content and sends it to the OpenAI API. The input data is the prompt, and the output data is the detailed emotion determination result.

[0154] Step 5:

[0155] The server integrates the emotion analysis results obtained from the Google Cloud Natural Language API and the OpenAI API to generate a final emotion judgment result. The input data is the emotion score and detailed emotion judgment result, and the output data is the integrated emotion judgment result.

[0156] Step 6:

[0157] The server then transmits the integrated emotion determination results to the smartphone. The transmitted data is a visualized version of the emotion determination results.

[0158] Step 7:

[0159] The device (smartphone) receives the emotion assessment results and displays them in the application. The displayed data is information about the customer's emotional state and the recommended timing for serving the customer.

[0160] Step 8:

[0161] The user (store clerk) refers to the displayed emotion determination result and adjusts the timing of serving the customer. Specifically, if the customer shows a positive emotion, the store clerk continues serving the customer, and if the customer shows a negative emotion, the store clerk reconsiders the timing of serving the customer.

[0162] Example 2

[0163] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] Conventional communication tools have difficulty accurately assessing users' emotions and providing optimal communication timing based on those emotions. Furthermore, there are also insufficient ways for users to utilize past communication data to obtain feedback for effective communication. This has resulted in users being unable to communicate effectively, leading to a decline in the quality of communication.

[0165] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for linking a sentence input to a communication tool to a generative AI, means for determining the emotion of the person who wrote the sentence using the generative AI, means for visualizing the determination result using an application, means for collecting past communication data, means for preprocessing the collected data, means for learning emotion fluctuation patterns, means for predicting the optimal communication timing, and means for presenting the prediction result to the user. This allows the user to know the optimal communication timing based on the emotion fluctuation pattern, enabling effective communication.

[0166] "Communication Tools" refers to software or hardware that allows users to exchange information with other users through messages, voice, video, etc.

[0167] "Generative AI" refers to artificial intelligence that uses natural language processing and machine learning techniques to generate and analyze data such as text and voice.

[0168] "Emotion determination" refers to the process of analyzing a user's emotions from text or voice data entered by the user and classifying them into specific emotional categories.

[0169] "Visualization" refers to displaying analysis results and data in a way that is easy for users to understand, such as in graphs, charts, or text format.

[0170] "Past communication data" refers to records of messages, voice and video calls, etc. that a user has previously made.

[0171] "Preprocessing" refers to a series of operations performed on collected data to transform it into an analyzable format, including denoising, tokenization, and normalization.

[0172] "Emotional patterns" refer to trends and patterns in data that show how users' emotions change over time and across situations.

[0173] "Optimal communication timing" refers to the time period or timing when a user is predicted to be able to communicate most effectively with other users.

[0174] "Prediction results" refer to the conclusions and estimates that generative AI reaches through analysis and learning.

[0175] MODE FOR CARRYING OUT THE INVENTION

[0176] This invention provides a system for enabling users to communicate effectively with other users. The system uses a generative AI model to determine the user's emotions and predict the optimal timing for communication. A specific embodiment of this system is described below.

[0177] 1. Program Generation

[0178] The server uses a generative AI model to generate a program that determines emotions and optimizes communication timing. This program is built using machine learning libraries such as Python (registered trademark) and TENSORFLOW (registered trademark).

[0179] 2. Program Processing

[0180] The server first collects the user's past communication data, including text messages, voice data, and facial expression data. For example, it collects the user's message history with friends and facial expression data during video calls.

[0181] The server then preprocesses the collected data. For text data, unnecessary characters and symbols are removed and the data is tokenized. For audio data, noise is removed and the audio is converted into text. For facial expression data, facial recognition technology is used to extract facial expression features.

[0182] The preprocessed data is then input into an emotion detection model, which uses generative AI models such as BERT and GPT-3 (registered trademark) to analyze user emotions. For example, emotions such as "joy," "sadness," and "anger" can be detected from text messages.

[0183] Once emotion determination is complete, the server performs time series analysis to learn the emotional fluctuation patterns. This is done using recurrent neural networks (RNNs) such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit). After learning is complete, the server predicts the optimal timing for communication.

[0184] The prediction results are presented to the user through the application. The user can use this information to communicate at the optimal time. For example, the app may notify the user, "Now is the best time to message a friend."

[0185] 3. Examples of concrete examples and prompts

[0186] For example, when a user is messaging a friend, the server analyzes past message data to learn the user's emotional fluctuation patterns.The server then predicts when the friend is most likely to respond and notifies the user of that timing.

[0187] Specific prompt examples:

[0188] "Analyze message exchange data with friends and predict the best timing for communication."

[0189] In this way, users can communicate more effectively with the help of generative AI.

[0190] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0191] Step 1: Data collection

[0192] The server collects the user's past communication data. Specifically, it acquires message history, voice data, and facial expression data during video calls from the user's smartphone or PC. The input is data from the user's device, and the output is the collected raw data.

[0193] Step 2: Data Preprocessing

[0194] The server preprocesses the collected data. In the case of text data, unnecessary characters and symbols are removed and the data is tokenized. In the case of audio data, noise is removed and the audio is converted into text. In the case of facial expression data, facial expression features are extracted using facial recognition technology. The input is the collected raw data, and the output is the preprocessed data.

[0195] Step 3: Emotion determination

[0196] The server inputs the preprocessed data into an emotion detection model. The emotion detection model uses generative AI models such as BERT and GPT-3 to analyze the user's emotions. For example, it detects emotions such as "joy," "sadness," and "anger" from a text message. The input is the preprocessed data, and the output is the emotion detection result.

[0197] Step 4: Learn emotional patterns

[0198] The server learns patterns of emotional fluctuations based on the results of emotion determination. For this, it uses recurrent neural networks (RNNs) such as LSTM and GRU. For example, it learns whether a user tends to show the emotion of "happiness" at a certain time of day. The input is the result of emotion determination, and the output is the learned pattern of emotional fluctuations.

[0199] Step 5: Anticipate communication timing

[0200] The server predicts the optimal timing for communication based on the learned emotional fluctuation patterns. For example, it predicts the time of day when a user can communicate most effectively with friends. The input is the learned emotional fluctuation patterns, and the output is the predicted optimal timing for communication.

[0201] Step 6: Presenting the results

[0202] The server presents the prediction results to the user through the application. The user can use this information to communicate at the optimal time. For example, the application may notify the user, "Now is the best time to send a message to a friend." The input is the predicted optimal communication timing, and the output is the notification to the user.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] With conventional communication tools, it was difficult to accurately assess users' emotions and support communication at the optimal timing. In addition, in physical stores, it was difficult for store staff to grasp customers' emotions in real time and determine the optimal timing to interact with them. This resulted in inconsistent customer service quality, making it difficult to improve customer satisfaction.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning past communication data and patterns of emotional fluctuations, means for predicting the optimal timing for communication, means for presenting the prediction results to a user, means for the user to select the timing of communication based on the presented prediction results, and means for supporting customer service in a physical store. This enables the user to select the optimal timing for communication based on emotions, and enables the emotions of customers in a physical store to be grasped in real time and the optimal timing for customer service to be determined.

[0208] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[0209] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[0210] "Emotion determination" is the process of analyzing a user's emotional state from input text or data and identifying a specific emotion.

[0211] "Visualization" refers to displaying data and information visually and providing it in a form that is easy for users to understand.

[0212] "Past communication data" refers to records or histories of previous communications, including message content and timing.

[0213] "Emotional fluctuation patterns" refer to emotional tendencies or patterns that change over time or in response to specific situations.

[0214] "Optimal communication timing" refers to the timing for the most effective communication based on the user's emotional state and past data.

[0215] "Predicted results" refers to future results and trends derived by generative AI through analysis and learning.

[0216] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[0217] "Customer service" refers to the service and response provided by store staff to customers.

[0218] As an embodiment of the present invention, a system for supporting customer service in a brick-and-mortar store will be described as an example.

[0219] First, the server provides a means for linking text entered into the communication tool to the generative AI. Specifically, text entered by the user using a device such as a smartphone or tablet is sent to the server via the internet. The server then passes this text to the generative AI, which determines its emotion.

[0220] Next, generative AI analyzes the input text and determines the user's emotions. Natural language processing techniques and machine learning algorithms are used to determine emotions. For example, a sentiment analysis model can be built using the Python libraries scikit-learn and TensorFlow.

[0221] The server visualizes the emotion results in an application. The application displays the emotion results on the screen of a smartphone or tablet, allowing the user to understand them intuitively. For example, the emotion score could be displayed as a graph or icon.

[0222] Furthermore, the server provides a means to learn past communication data and emotional fluctuation patterns, using past communication history stored in a database. Using this data, the server trains a model for the generative AI to predict optimal communication timing.

[0223] To predict the optimal timing for communication, the generative AI receives the current time and emotion score as input and generates a prediction result. This prediction result is sent to the application via the server and presented to the user. The user can refer to this prediction result to communicate at the optimal time.

[0224] In physical stores, store clerks use their smartphones to capture customers' facial expressions. The application calculates an emotion score in real time and suggests the optimal timing for customer interaction. This allows store clerks to understand the emotional state of customers and provide assistance at the optimal time.

[0225] As a concrete example, the following prompt might be input to a generative AI:

[0226] "It is currently 10 o'clock. The customer's sentiment score is 0.7. Please predict the optimal time to serve them."

[0227] In this way, users can choose the optimal timing for communication based on their emotions, and even in physical stores, they can grasp the emotions of customers in real time and determine the optimal timing for customer service.

[0228] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0229] Step 1:

[0230] A user inputs text into a communication tool using a device such as a smartphone or tablet.

[0231] Input: Text entered by the user

[0232] Output: The input text is sent from the terminal to the server.

[0233] Specific actions: The user opens an application on the device, enters text in the text box, and presses the send button.

[0234] Step 2:

[0235] The server receives the text sent from the device and connects it to the generative AI.

[0236] Input: Text sent from the terminal

[0237] Output: Text passed to the generative AI

[0238] Specific operation: The server receives an HTTP request, analyzes the text data, and passes it to the generative AI API.

[0239] Step 3:

[0240] The generative AI analyzes the received text and determines the user's emotions.

[0241] Input: Text given to the generative AI

[0242] Output: Sentiment score (e.g. 0.7)

[0243] How it works: Generative AI uses natural language processing techniques to analyze text and calculates a sentiment score using a sentiment analysis model (e.g., scikit-learn or TensorFlow).

[0244] Step 4:

[0245] The server visualizes the emotion scores received from the generative AI in the application.

[0246] Input: Sentiment score

[0247] Output: Visualized sentiment scores (e.g., graphs and icons)

[0248] Specific operation: The server receives the emotion score, generates data to reflect in the application UI, and sends it to the device.

[0249] Step 5:

[0250] The server learns past communication data and patterns of emotional fluctuations.

[0251] Input: Historical communication data

[0252] Output: The trained model

[0253] Specific operation: The server retrieves past communication history from the database and passes it to the generative AI to train the model.

[0254] Step 6:

[0255] The generative AI takes the current time and emotion score as input and predicts the optimal timing for communication.

[0256] Input: Current time, Sentiment score

[0257] Output: Prediction of optimal communication timing

[0258] How it works: The generative AI uses a trained model to predict the optimal timing using the current time and emotion score as input.

[0259] Step 7:

[0260] The server presents the prediction results to the user.

[0261] Input: Prediction results of optimal communication timing

[0262] Output: Predictions presented to the user

[0263] Specific operation: The server receives the prediction results, generates data to be reflected in the application UI, and sends it to the device.

[0264] Step 8:

[0265] The user refers to the presented prediction results and communicates at the optimal time.

[0266] Input: Predicted results

[0267] Output: User-selected timing of communication

[0268] Specific behavior: The user checks the results presented by the application and sends a message at the optimal time.

[0269] Step 9:

[0270] When providing customer service support in a physical store, store staff carry smartphones and use the camera to capture the facial expressions of customers.

[0271] Input: Customer facial expression data

[0272] Output: Real-time emotion score

[0273] How it works: A store associate uses a smartphone camera to capture a customer's facial expression, and the application determines an emotion score in real time.

[0274] Step 10:

[0275] The application suggests the best time to serve customers.

[0276] Input: Real-time emotion score

[0277] Output: Recommendation of optimal timing for customer service

[0278] Specific operation: The application calculates the optimal timing to serve the customer based on the emotion score and notifies the store clerk.

[0279] Example 3

[0280] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Conventional communication tools have difficulty accurately assessing users' emotions and providing feedback based on them. Furthermore, there have been no systems that analyze past emotional data and the effectiveness of communication to provide effective advice to users. As a result, users have been unable to obtain specific guidance on how to improve their communication skills.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0283] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an application, means for analyzing past emotion data and the effectiveness of communication, and means for providing feedback to the user based on the analysis results. This allows the user to receive effective feedback based on their own emotions and obtain specific guidelines for improving their communication skills.

[0284] "Communication tools" is a general term for software or hardware that allows users to exchange information with others through text, voice, etc.

[0285] "Generative AI" is a general term for artificial intelligence technology that generates new information based on input data.

[0286] "Means for determining emotions" is a general term for algorithms and technologies that analyze a user's emotions from input text or data and identify specific emotions.

[0287] "Application" is a general term for a software program designed to provide a specific function or service.

[0288] "Visualization means" is a general term for techniques and methods for visually displaying data and information so that users can easily understand it.

[0289] "Past emotion data" is a general term for records and data relating to emotions expressed by a user in the past.

[0290] "Communication effectiveness" is a general term for the impact or results that a particular communication method has on users.

[0291] "Analytical methods" is a general term for techniques and methods used to analyze data and find specific patterns and trends.

[0292] "Means for providing feedback" is a general term for techniques and methods for providing advice and information to users based on analysis results.

[0293] This invention is a system that links text entered by a user through a communication tool to a generative AI, determines emotions, and provides feedback based on the results. A specific embodiment of this system is described below.

[0294] Hardware and software used

[0295] server

[0296] The server provides the computational resources to run the generative AI model. Specifically, it uses the following software and hardware:

[0297] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[0298] Database management system: Relational database such as MySQL (registered trademark)

[0299] Machine learning algorithms: Machine learning libraries such as scikit-learn

[0300] Terminal

[0301] A terminal is a device for users to provide input data and receive feedback. Specifically, it uses the following software and hardware:

[0302] Browser: A web browser such as GOOGLE CHROME (registered trademark)

[0303] Communication protocol: HTTP / HTTPS

[0304] User

[0305] A user uses a communication tool to input text data and receive feedback.

[0306] Program processing

[0307] server

[0308] The server receives text data sent from the device and uses the generative AI model to determine the emotion. For example, if a user enters "Recently, communication at work has been going poorly," the server analyzes the text data and determines the emotion as "anxiety."

[0309] The server then uses a database management system to analyze the user's past emotional data and the effectiveness of communication. For example, it might determine that presenting specific solutions to problems was effective when the user expressed anxiety.

[0310] The server then uses the generative AI model to generate feedback for the user, such as "By providing specific solutions to the problem, you may be able to improve communication in the workplace."

[0311] Terminal

[0312] The device displays the feedback received from the server to the user, who can then check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[0313] Examples of specific examples and prompts

[0314] As a concrete example, consider the case where a user inputs, "Recently, communication at work has not been going well." The server receives this input and uses a generative AI model to determine the user's emotion as "anxiety." It then analyzes past data to identify the most effective communication method when the user expressed "anxiety." For example, suppose "presenting specific solutions to the problem" was effective. Based on this information, the server provides the user with feedback such as, "Providing specific solutions to the problem may improve communication at work."

[0315] Example prompt sentence:

[0316] User: Communication at work has been difficult lately.

[0317] Server: Determines user sentiment and analyzes the most effective communication method based on past data.

[0318] Server: When users expressed anxiety, the most effective thing we did was to offer concrete solutions to their problems.

[0319] Server: Providing concrete solutions to problems can improve communication in the workplace.

[0320] In this way, a system is realized in which the server, the terminal, and the user work together to provide effective feedback to the user. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0321] Step 1:

[0322] The user provides the input data.

[0323] A user inputs text data through the interface of a communication tool. For example, the user inputs, "Recently, communication at work has not been going well." The input data is in text format and expresses the user's feelings and situation.

[0324] Step 2:

[0325] The terminal sends the input data to the server.

[0326] The terminal converts the text data entered by the user into JSON format and sends it to the server as an HTTP POST request. The input data is a text document, and the output is the JSON format data sent to the server.

[0327] Step 3:

[0328] The server uses a generative AI model to determine emotions.

[0329] The server inputs the received text data into a generative AI model (e.g., GPT-4) to determine the user's emotion. The input data is text data in JSON format, and the output is the emotion determination result (e.g., "anxiety"). The server invokes the generative AI model, analyzes the text data, and identifies the emotion.

[0330] Step 4:

[0331] The server analyzes past data and identifies effective communication methods.

[0332] The server uses a database management system (e.g., MySQL) to analyze the user's past emotional data and the effectiveness of communication. The input data is the emotional judgment result and past emotional data, and the output is an effective communication method (e.g., "present a specific solution to the problem"). The server analyzes the data using a machine learning algorithm (e.g., scikit-learn) and identifies the optimal communication method.

[0333] Step 5:

[0334] The server generates the feedback.

[0335] The server generates feedback using a generative AI model based on the identified effective communication methods. The input data are effective communication methods, and the output is feedback to the user (e.g., "Providing specific solutions to problems may improve communication in the workplace"). The server then invokes the generative AI model again to generate feedback in natural language.

[0336] Step 6:

[0337] The device displays the feedback to the user.

[0338] The device displays the feedback received from the server to the user. The input data is the feedback text, and the output is the feedback displayed on the user's screen. The user can check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[0339] (Application example 3)

[0340] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] Conventional communication tools have difficulty accurately determining a user's emotions and providing appropriate feedback based on them. Security services are required to determine a user's emotions in real time and provide appropriate responses, but there has been a lack of effective means to achieve this. Furthermore, there has been no system that utilizes past emotion data to suggest the most effective communication method.

[0342] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence using the generative AI, a means for visualizing the determination result in an app, a means for analyzing past emotion data and proposing the most effective communication method, and a means for determining emotion in real time and providing feedback according to the security situation. This makes it possible to accurately determine a user's emotion and provide appropriate feedback. Furthermore, in security services, emotions can be determined in real time and appropriate responses can be taken. Furthermore, the most effective communication method can be proposed by utilizing past emotion data.

[0343] A "communication tool" is software or hardware that allows a user to exchange information with others through text or voice.

[0344] "Generative AI" is an artificial intelligence technology that generates new data based on input data, and is particularly used in natural language processing and image generation.

[0345] "Emotion determination" is the process of identifying a user's emotional state from the text or voice input they provide.

[0346] "Visualization" is a method of visually displaying data and information to make it easier for users to understand.

[0347] "Past emotion data" is a record of emotions a user has expressed in the past, and analyzing this data can reveal patterns and trends.

[0348] The "most effective communication method" is the method that provides the best response and feedback to the user's emotional state.

[0349] "Real-time" refers to data processing and feedback occurring immediately, without delay.

[0350] "Security situation" refers to the safety-related state and risks faced by users and systems.

[0351] "Feedback" is a response or advice provided to a user's actions or status.

[0352] The system for implementing this invention mainly consists of three elements: a server, a terminal, and a user. The function of each element will be explained below.

[0353] server

[0354] The server has a means of linking text entered into the communication tool to the generative AI. Specifically, it receives text data entered by the user into the device and sends it to the generative AI. The generative AI analyzes the entered text data and determines the user's emotions. The results of this emotion determination are saved by the server and used for later analysis.

[0355] Furthermore, the server has a means of analyzing past emotional data and proposing the most effective communication method, which allows it to learn which emotions the user expressed in the past resulted in the most effective feedback, and use this knowledge for future communications.

[0356] Terminal

[0357] The device is responsible for sending the text data entered by the user to the server. It also has a means of visualizing the emotion assessment results and feedback received from the server to the user, allowing the user to check their own emotional state in real time.

[0358] The device also has a means of detecting emotions in real time and providing feedback according to the security situation. For example, if a user inputs "I'm very anxious right now," the device will provide advice such as "Take a deep breath and calm down" based on the feedback received from the server.

[0359] User

[0360] Users input text data through their devices and receive feedback from the server. This allows users to understand their own emotional state and take appropriate action. Furthermore, by receiving suggestions on the most effective communication methods based on past emotional data, users can improve their communication skills.

[0361] Hardware and software used

[0362] Hardware: Smartphone (iOS or ANDROID (registered trademark))

[0363] Software: Python, scikit-learn (machine learning library), joblib (saving and loading models)

[0364] Specific examples

[0365] For example, if a user types "I'm very anxious right now," the server sends this text data to the generative AI, which determines the emotion as "anxiety." The server then analyzes past emotion data and suggests the most effective feedback: "Take a deep breath and calm down." The device displays this feedback to the user, who can then act accordingly.

[0366] Prompt Sentence Examples

[0367] "Write a Python program that determines the emotion of text entered by a user and provides the most effective feedback. Use a pre-trained model to determine the emotion, and a model trained on past data to provide the feedback."

[0368] In this way, the present invention accurately determines the user's emotions and provides appropriate feedback, enabling the user to improve their communication skills and respond appropriately to security situations.

[0369] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0370] Step 1:

[0371] The user inputs text data into the terminal. The input text data is used as initial data for determining the user's emotion. Input: Text data input by the user. Output: Text data sent to the terminal.

[0372] Step 2:

[0373] The device sends the input text data to the server. The server receives this data and connects it to the generative AI. Input: Text data sent from the device. Output: Text data sent to the server.

[0374] Step 3:

[0375] The server sends the received text data to the generative AI, which determines the emotion. The generative AI analyzes the text data and determines the user's emotion. Input: Text data sent to the server. Output: Emotion data determined by the generative AI.

[0376] Step 4:

[0377] The server stores the emotional data received from the generative AI and analyzes it together with past emotional data. This generates data to suggest the most effective communication methods. Input: Emotional data received from the generative AI. Output: Analyzed communication method data.

[0378] Step 5:

[0379] The server generates the most effective feedback based on the analysis results and sends it to the device. Input: Analyzed communication method data. Output: Feedback data sent to the device.

[0380] Step 6:

[0381] The device visualizes the feedback data received from the server to the user, allowing the user to check their own emotional state and the appropriate feedback in real time. Input: Feedback data received from the server. Output: Feedback information displayed to the user.

[0382] Step 7:

[0383] The user adjusts their behavior based on the feedback displayed on the device. This allows the user to take the optimal action based on their emotional state. Input: Feedback information displayed on the device. Output: User behavior adjustment.

[0384] In this way, a system is realized that can accurately determine a user's emotions and provide appropriate feedback, enabling them to improve their communication skills and respond appropriately to security situations.

[0385] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0386] "Example 1"

[0387] In one embodiment of the present invention, when a user inputs a sentence into a communication tool, the sentence is linked to a generative AI. The generative AI uses an emotion engine to determine the user's emotion from the sentence. The emotion engine infers the emotion from, for example, words, phrases, emojis, etc. in the sentence. The determined emotion is visualized in the app, and the user can use the results as a reference when selecting the timing of communication.

[0388] "Example 2"

[0389] In another embodiment, the generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns patterns of emotional fluctuation from past communication data. It then predicts the optimal timing for communication based on the learned data. The predicted timing is presented to the user on the app, allowing the user to select the timing of communication based on this information.

[0390] "Example 3"

[0391] In yet another embodiment, the generative AI uses an emotion engine to analyze what emotions the user displayed in the past to achieve the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotion assessment results at that time. It then provides feedback to the user based on these results. The user can use this feedback to improve their own communication skills.

[0392] The processing flow of each embodiment will be described below.

[0393] "Example 1"

[0394] Step 1: The user enters text into the communication tool.

[0395] Step 2: The input text is linked to the generative AI.

[0396] Step 3: The generative AI uses an emotion engine to determine the user's emotion from the text. The emotion engine infers emotion from words, phrases, emojis, etc. in the text.

[0397] Step 4: The determined emotions are visualized in the app, and the user can use this result to choose the timing of communication.

[0398] "Example 2"

[0399] Step 1: The generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns emotional fluctuation patterns from past communication data.

[0400] Step 2: The generative AI predicts the optimal timing for communication based on the learned data.

[0401] Step 3: The predicted timing is presented to the user on the app, and the user uses this information to choose the timing of communication.

[0402] "Example 3"

[0403] Step 1: The generative AI uses an emotion engine to analyze what emotions the user displayed in the past that resulted in the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotional assessment results at that time.

[0404] Step 2: The generative AI provides feedback to the user based on the results. Step 3: The user uses this feedback to improve their communication skills.

[0405] Example 1

[0406] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] Conventional communication tools have had the problem that it is difficult for users to accurately grasp the emotions of others, making it difficult to communicate at the right time. In addition, there are insufficient means for providing users with the emotion assessment results, and it is unclear how users should use the results.

[0408] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence by the generative AI, a means for visualizing the determination result in an app, and a means for the user to select the timing of communication based on the emotion determination result. This allows the user to accurately understand the other person's emotion and communicate at an appropriate time.

[0409] A "communication tool" is software or a platform that allows users to exchange messages with other users.

[0410] "Generative AI" is an artificial intelligence technology that analyzes input data and generates specific results.

[0411] The "means for determining emotion" is an algorithm or system for analyzing emotion from input text and identifying that emotion.

[0412] "Visualization means" refers to an interface or function that displays the analysis results in a form that is easy for the user to understand.

[0413] The "means for the user to select the timing of communication based on the emotion determination result" is a guideline or function that allows the user to determine the optimal timing of communication based on the displayed emotion determination result.

[0414] The "means for providing feedback" is a function for providing advice and information to the user based on the emotion determination result.

[0415] MODE FOR CARRYING OUT THE INVENTION

[0416] This invention is a system that links sentences entered by users into a communication tool with generative AI, determines emotions from the sentences, and visualizes the determination results in an app. A specific embodiment of this system is described below.

[0417] A user types a sentence into Google Chat

[0418] A user uses a communication tool such as Google Chat to type a message, for example, "Today was a great day!" This message is then sent over the Internet to a server.

[0419] The server connects the text to a generative AI

[0420] The server receives messages sent by users using the Google Chat API. The received messages are sent to the generative AI, which has an emotion engine and prepares the text for analysis.

[0421] Generative AI analyzes text and determines emotions

[0422] Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in input text. For example, it can determine positive emotions from the word "fun." This analysis uses natural language processing technology and machine learning algorithms.

[0423] The determined emotion is returned to the server

[0424] The generative AI returns the analysis results to the server, which then receives the results and proceeds to the next step.

[0425] The server sends the emotion determination result to the app.

[0426] The server sends the emotion determination results to the user's device, which receives the results and prepares them for display in the app.

[0427] The device displays the emotion determination result in the app.

[0428] The device displays the received emotion determination result in the app. The user can check the emotion determination result through the app. For example, the app may display "positive emotion."

[0429] Users choose when to communicate

[0430] The user can choose the timing of communication based on the emotion assessment results displayed on the app. For example, if the other person is showing positive emotions, they can choose to reply at that time.

[0431] Specific examples

[0432] A user types "Today was such a fun day!" into Google Chat and presses the send button. This message is sent to the server, which receives the text via the Google Chat API. The server connects the received text to a generative AI, which analyzes the word "fun" and determines the positive emotion. The generative AI returns the positive emotion determination result to the server, which then sends the emotion determination result to the user's device. The emotion determination result received by the device is displayed in the app, and is displayed as "positive emotion." The user checks the "positive emotion" displayed in the app and chooses to reply at that time.

[0433] Prompt Sentence Examples

[0434] "Create a generative AI program that analyzes text entered in Google Chat and determines the emotion. Include a function that uses an emotion engine to infer emotion from words, phrases, emojis, etc. in the text, and visualize the results in the app."

[0435] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0436] Step 1: User types a sentence into Google Chat

[0437] What happens: A user opens the Google Chat app and types a message, for example, "Today was such a fun day!"

[0438] Input: Text entered by the user

[0439] Output: Message sent to Google Chat

[0440] Step 2: The input text is sent to the server

[0441] How it works: When a user sends a message, the text is sent over the Internet to a server, which then receives the message using the Google Chat API.

[0442] Input: Message sent from Google Chat

[0443] Output: Message received by server

[0444] Step 3: The server connects the text to the generative AI

[0445] Specific operation: The server sends the received text to the generative AI, which has an emotion engine and prepares to analyze the text.

[0446] Input: Message received by the server

[0447] Output: Message sent to the generative AI

[0448] Step 4: Generative AI analyzes the text and determines the sentiment

[0449] How it works: Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in the input text. For example, it determines positive emotions from the word "fun."

[0450] Input: Message sent to the generative AI

[0451] Output: Emotion determination result (e.g., positive emotion)

[0452] Step 5: The determined emotion is returned to the server

[0453] Specific operation: The generative AI returns the analysis results to the server, which receives the results and proceeds to the next step.

[0454] Input: Emotion determination result

[0455] Output: Emotion judgment result returned to the server

[0456] Step 6: The server sends the emotion result to the app

[0457] Specific operation: The server sends the emotion determination result to the user's device. The device receives the result and prepares to display it in the app.

[0458] Input: Emotion judgment result returned to the server

[0459] Output: Emotion judgment result sent to the user's device

[0460] Step 7: The device displays the emotion detection result in the app.

[0461] Specific operation: The device displays the received emotion judgment result in the app. The user can check the emotion judgment result through the app. For example, the app will display "positive emotion."

[0462] Input: Emotion judgment result sent to the user's device

[0463] Output: Emotion judgment results displayed on the app

[0464] Step 8: Users choose when to communicate

[0465] Specific operation: The user chooses the timing of communication based on the emotion judgment results displayed in the app. For example, if the other person is showing positive emotions, they choose to reply at that time.

[0466] Input: Emotion judgment result displayed on the app

[0467] Output: User-selected timing of communication

[0468] (Application example 1)

[0469] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0470] With conventional communication tools, it was difficult for users to accurately grasp the emotions of the other party, making it difficult to communicate at the right time. Also, in physical stores, store staff are required to understand customers' emotions in real time and respond optimally, but there was a lack of effective means to do so. This led to the issue of not being able to improve customer satisfaction or provide efficient customer service.

[0471] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0472] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an app, means for converting voice data into text, means for displaying the emotion determination result on a display device, and means for providing a response guide according to the emotion. This allows users to understand the other person's emotion in real time and communicate and respond to customers at the optimal time.

[0473] A "communication tool" is software or hardware that allows users to exchange information with each other through text or voice.

[0474] "Generative AI" is an artificial intelligence system that generates new information based on input data.

[0475] "Emotion determination" is the process of identifying and classifying the emotions contained in the content of input text or audio data.

[0476] "Visualization" is a method of visually displaying data or information so that users can easily understand it.

[0477] "Audio data" refers to data in which audio is recorded in digital format.

[0478] "Text conversion" is the process of converting audio data into text data.

[0479] A "display device" is a device for visually displaying information, and includes displays and screens.

[0480] A "response guide" is a set of instructions or advice that suggests appropriate actions or words to use in a given situation or emotion.

[0481] As an embodiment of the present invention, a system for supporting customer service in brick-and-mortar stores will be described as an example. This system is configured using hardware and software such as smart glasses, a microphone, a generative AI model, an emotion engine, and a display device.

[0482] First, a microphone built into the smart glasses captures what the customer says. The captured voice data is converted into text using the Google Cloud Speech-to-Text API. The converted text is sent to a generative AI model (e.g., OpenAI GPT-4) and emotion is determined using an emotion engine (e.g., IBM Watson® Tone Analyzer).

[0483] The detected emotion is displayed on the smart glasses' display. For example, positive emotions (such as joy and relief) are displayed in green, while negative emotions (such as anger and anxiety) are displayed in red. A guide to how to respond based on the emotion is also displayed. For example, if a customer is feeling anxious, phrases or actions to reassure them are suggested.

[0484] This system allows store staff to understand customer emotions in real time and respond optimally, thereby improving customer satisfaction and enabling more efficient customer service.

[0485] For example, the following prompt is input to a generative AI model:

[0486] Customer says: "I'd like to learn more about this product, but I'm a little unsure."

[0487] Prompt: "Determine the customer's sentiment from this statement."

[0488] Based on this prompt, the generative AI model determines the customer's emotions and displays the results on the smart glasses' display, allowing the store clerk to respond optimally based on the customer's emotions.

[0489] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0490] Step 1:

[0491] A microphone in the smart glasses captures what the customer says.

[0492] Input: Customer voice

[0493] Output: Audio data

[0494] Specific operation: The microphone built into the smart glasses records the customer's speech in real time and saves it as audio data.

[0495] Step 2:

[0496] The server converts the audio data into text.

[0497] Input: Audio data

[0498] Output: Text data

[0499] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[0500] Step 3:

[0501] The server sends the text data to a generative AI model to determine the emotion.

[0502] Input: Text data

[0503] Output: Emotion determination result

[0504] Specific operation: The server sends text data to a generative AI model (e.g., OpenAI GPT-4), which then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to determine the emotion.

[0505] Step 4:

[0506] The server displays the emotion determination result on the display of the smart glasses.

[0507] Input: Emotion determination result

[0508] Output: Display

[0509] Specific operation: The server sends the emotion judgment result to the smart glasses display, and positive emotions are displayed in green and negative emotions in red.

[0510] Step 5:

[0511] The server provides a response guide based on the emotion.

[0512] Input: Emotion determination result

[0513] Output: Corresponding guide

[0514] Specific operation: Based on the emotion determination result, the server generates a response guide and displays it on the smart glasses display. For example, if the customer is feeling anxious, it will suggest phrases or actions to reassure them.

[0515] Example 2

[0516] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0517] With conventional communication tools, it was difficult to accurately assess a user's emotions and provide optimal communication timing based on the results. In addition, there was a lack of means for users to use past communication data to understand patterns of emotional fluctuations and communicate effectively. This meant that users were unable to communicate at the appropriate time, resulting in a decline in the quality of communication.

[0518] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0519] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for collecting past communication data, means for extracting patterns of emotional fluctuation from the collected data, means for predicting the optimal communication timing based on the extracted patterns, and means for presenting the prediction result in the application. This enables the user to understand patterns of emotional fluctuation and communicate at the optimal timing.

[0520] "Communication Tools" means software or applications that allow users to exchange messages or information with other users.

[0521] "Generative AI" is a system that uses artificial intelligence technology to generate new information and results from input data.

[0522] "Emotion determination" is the process of analyzing a user's emotions from input text and data and identifying the type and intensity of those emotions.

[0523] An "application" is a software program that a user uses to perform a specific task.

[0524] "Data collection" is the process of collecting and storing a user's past communication data.

[0525] "Emotional fluctuation patterns" are trends or patterns in data that show how a user's emotions change over time and in different situations.

[0526] "Communication timing" refers to the optimal time and situation for a user to communicate with other users.

[0527] "Forecasting" is the process of predicting future events or outcomes based on past data and patterns.

[0528] "Visualization" is the process of visually displaying data and information and providing it in a form that is easy for users to understand.

[0529] "Feedback" refers to the evaluation and advice the system provides to the user.

[0530] The present invention is a system for determining a user's emotions and providing optimal communication timing. Specific embodiments of this system will be described below.

[0531] Hardware and software used

[0532] Hardware: Servers (with high-performance data processing capabilities), user devices (smartphones, tablets, PCs, etc.)

[0533] Software: Emotion engine, generative AI model, application (user interface)

[0534] Program processing

[0535] Server Processing

[0536] The server collects users' past communication data, including message content, sending time, receiving time, reply time, etc. The server stores this data in a database for later processing.

[0537] Next, the server uses the collected communication data to determine emotions in the emotion engine. The emotion engine uses natural language processing (NLP) technology to extract emotions from the content of the message. Emotions include joy, sadness, anger, surprise, etc.

[0538] The server then uses the emotion determination results to train a generative AI model. It learns past patterns of emotion fluctuations and predicts the optimal timing for communication. This prediction is made using machine learning algorithms (e.g., LSTM, Random Forest).

[0539] Finally, the server sends the prediction results to the application, which then suggests optimal communication timings to the user via notifications or dashboard displays.

[0540] Terminal handling

[0541] The device receives the emotion determination results and the prediction results for the optimal communication timing sent from the server. This information is displayed to the user on the application. The user can refer to this information to select the best timing for communication.

[0542] User Action

[0543] The user can check the emotion assessment results and the optimal communication timing predictions provided by the server through the application, which allows the user to communicate at the optimal timing.

[0544] Specific examples

[0545] The server collects communication data from user A over the past month. The server uses an emotion engine to extract patterns of emotional fluctuations from this data and trains it using a generative AI model. Based on the learning results, the server predicts the optimal timing for communication for user A and presents the results to user A via the application. User A uses this information to decide when to send a message to friend B.

[0546] Prompt Sentence Examples

[0547] "Based on communication data from the past month, please learn User A's emotional fluctuation patterns and predict the optimal timing for communication."

[0548] In this way, by clarifying the roles of the server, terminal, and user, and explaining how to use specific hardware and software, it becomes easier to understand the overall picture of the system.

[0549] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0550] Step 1: Data collection

[0551] Subject: Server

[0552] The server collects users' past communication data, including message content, sending time, receiving time, reply time, etc. The server stores this data in a database.

[0553] Input: User's message data

[0554] Output: Communication data stored in a database

[0555] What it does: The server retrieves data from the user's messaging application through an API, such as the content and timestamps of messages the user has sent and received in the past month.

[0556] Step 2: Emotion determination

[0557] Subject: Server

[0558] The server uses the collected communication data to determine emotions using an emotion engine. The emotion engine uses natural language processing (NLP) technology to extract emotions from the content of messages. Emotions include joy, sadness, anger, and surprise.

[0559] Input: Communication data stored in a database

[0560] Output: Emotion determination result

[0561] What it does: The server uses an NLP library (e.g., spaCy or NLTK) to analyze the text of the message and determine its sentiment. For example, it extracts the sentiment "joy" from the message "I'm so happy today!"

[0562] Step 3: Learning and Prediction

[0563] Subject: Server

[0564] The server uses the emotion determination results to train a generative AI model. It learns past patterns of emotion fluctuations and predicts the optimal timing for communication. This prediction is made using machine learning algorithms (e.g., LSTM, Random Forest).

[0565] Input: Emotion judgment result

[0566] Output: Prediction of optimal communication timing

[0567] How it works: The server inputs the emotion determination results as features into a generative AI model and trains the model. For example, it learns a pattern from past data that "users' emotions are stable on Monday afternoons."

[0568] Step 4: Presenting the results

[0569] Subject: Server

[0570] The server sends the prediction results to the application, which then suggests optimal communication timings to the user. This can be done via notifications or dashboard displays.

[0571] Input: Prediction results of optimal communication timing

[0572] Output: The prediction results displayed in the application

[0573] What it does: The server sends the prediction results in JSON format to the application. The application receives this data and displays it in a user interface, for example, by notifying the user that "next Monday at 3:00 PM is the best time."

[0574] Step 5: User Actions

[0575] Subject: User

[0576] The user checks the optimal communication timings suggested by the application and uses this information to choose when to send a message.

[0577] Input: The prediction result displayed in the application

[0578] Output: User's communication actions

[0579] Specific behavior: The user checks the application's notification and sends a message to a friend. For example, if the application notifies the user that "next Monday at 3:00 PM is the best time," the user sends a message to a friend at that time.

[0580] (Application example 2)

[0581] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0582] Conventional communication tools have had the problem of being unable to accurately assess a user's emotions and communicate at the optimal timing. In particular, when serving customers in physical stores, it is necessary to analyze customer emotions and behavioral patterns in real time and communicate at the optimal timing, but there has been no effective means to achieve this.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0584] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning patterns of emotional fluctuations using past communication data, means for predicting the optimal timing for communication based on new data, and means for notifying the user of the predicted timing. This makes it possible to analyze customer emotions and behavioral patterns in real time when serving customers in physical stores, and to serve customers at the optimal timing.

[0585] "Communication Tools" means software or hardware that a User uses to exchange information with other Users.

[0586] "Generative AI" refers to artificial intelligence that generates new data based on input data.

[0587] "Emotion determination" refers to the process of identifying a user's emotional state from their input data.

[0588] "Visualization" is the process of displaying data or information visually to make it easier for users to understand.

[0589] "Past communication data" refers to recorded data relating to previous communications.

[0590] "Emotional fluctuation patterns" refer to the tendency or characteristics of emotions that change over time.

[0591] The "optimal communication timing" is the time when users are predicted to be able to communicate most effectively.

[0592] A "notification" is a message or alert that informs the user of specific information.

[0593] A system for carrying out this invention is configured as follows: The server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning patterns of emotional fluctuations using past communication data, means for predicting the optimal timing for communication based on new data, and means for notifying the user of the predicted timing.

[0594] Specifically, the server uses Python and the scikit-learn library to analyze past communication data and learn patterns of emotional fluctuations. This allows it to predict the optimal timing for communication based on new data entered by the user. The server then notifies the smart glasses of the predicted timing, helping store clerks to communicate with customers at the optimal time.

[0595] For example, in a brick-and-mortar store, if a salesperson is wearing smart glasses, they can analyze the customer's behavioral patterns and facial expressions in real time when they pick up a product. The server compares this with past data to predict when the customer will be most likely to accept service and notify the salesperson. This can improve customer satisfaction.

[0596] An example of a prompt sentence is, "Please predict the best time to speak to the customer after they have picked up the product." By inputting this prompt sentence into the generative AI model, it is possible to predict the best timing for communication and notify the store clerk.

[0597] This system will make it possible to improve the quality of customer service in physical stores and increase customer satisfaction.

[0598] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0599] Step 1:

[0600] The server receives text entered into the communication tool. The input data is a text message entered by the user. This data is converted into an appropriate format for linking with the generative AI. Specifically, the text data is tokenized and converted into a format that can be processed by the generative AI.

[0601] Step 2:

[0602] The server uses generative AI to determine the user's emotion from the input text. The input data is tokenized text, and the output data is emotion labels (e.g., joy, sadness, anger, etc.). The generative AI uses a pre-trained model to extract emotions from the text data.

[0603] Step 3:

[0604] The server visualizes the determined emotions in an application. The input data is emotion labels, and the output data is a graphical representation of the emotion displayed on the user interface. Specifically, the emotional state is visually shown to the user using icons and colors corresponding to the emotion labels.

[0605] Step 4:

[0606] The server uses past communication data to learn patterns of emotional fluctuations. The input data are past text messages and their corresponding emotion labels, and the output data is a pattern model of emotional fluctuations. Specifically, a machine learning algorithm is used to analyze trends in emotions that change over time and build the model.

[0607] Step 5:

[0608] The server predicts the optimal timing for communication based on new data. The input data are text messages and emotion labels captured in real time, and the output data is the optimal timing prediction. Specifically, the server uses a trained emotion fluctuation pattern model to calculate the optimal timing based on the current emotional state.

[0609] Step 6:

[0610] The server notifies the user of the predicted optimal communication timing. The input data is the prediction result, and the output data is a notification message to be displayed on the user's smart glasses or smartphone. Specifically, the server generates the notification message and sends it to the user's device.

[0611] Step 7:

[0612] The user receives the notification and communicates with the customer at the optimal time. The input data is the notification message from the server, and the output data is the user's actions (e.g., when to speak to the customer). Specifically, the user serves the customer at the appropriate time according to the notification message.

[0613] Example 3

[0614] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0615] With conventional communication tools, it was difficult to accurately assess a user's emotions and provide effective feedback based on those results. Furthermore, there was a lack of concrete methods for utilizing past communication data to improve users' communication skills. This made it difficult for users to understand the relationship between their own emotions and communication and improve their skills.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0617] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an application, means for analyzing past communication data and the determination results of emotions at that time, means for generating feedback for the user based on the analysis results, and means for providing the generated feedback to the user. This enables users to understand the relationship between their own emotions and communication and improve their skills.

[0618] "Communication Tools" means software or applications that allow users to exchange messages or information with other users.

[0619] "Generative AI" is an artificial intelligence technology that generates new information and results based on input data.

[0620] "Emotion determination" is the process of analyzing a user's emotions from input text or data and assigning a specific emotion label.

[0621] An "application" is a software program designed to provide a particular function or service.

[0622] A "database" is a system for efficiently storing, managing, and retrieving data.

[0623] "Feedback" refers to evaluations and advice provided to users.

[0624] "Analysis" is the process of examining data in detail to find meaning and patterns.

[0625] "User" means any individual or entity that uses the system or application.

[0626] The present invention is a system for determining a user's emotion and providing feedback based on the result. A specific embodiment of this system will be described below.

[0627] Hardware and software used

[0628] Hardware: Servers, devices (PCs, smartphones, etc.)

[0629] Software: Emotion engine, generative AI model, database management system (DBMS)

[0630] System Overview

[0631] This system collects communication data when a user communicates using a device and uses an emotion engine to determine the user's emotions. The results are stored in a database, and a generative AI model is used to analyze past communication data and the emotion determination results. The system extracts the situations in which communication was most effective and generates and provides feedback to the user based on the results.

[0632] Specific examples

[0633] If the emotion engine determines that User A felt "joy" based on past communication data, then that communication was the most effective. The generative AI model analyzes this information and provides feedback to User A, such as, "The communication you received when you felt joy was the most effective. Try to keep this emotion in mind next time."

[0634] Prompt Sentence Examples

[0635] "Analyze the emotions the user has expressed in the past and the effectiveness of communication at that time to extract the most effective situations. Then, generate feedback for the user based on the results."

[0636] This system enables users to understand the relationship between their own emotions and communication and improve their skills. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0637] Step 1:

[0638] Users communicate using terminals.

[0639] Typed: Messages typed by users into communication tools such as chat apps or emails.

[0640] Specific action: For example, a user sends a message saying, "I'm so happy today!"

[0641] Output: Message data sent from the device to the server.

[0642] Step 2:

[0643] The server inputs the received message data into an emotion engine to determine the user's emotion.

[0644] Input: Message data sent from the terminal.

[0645] Specific behavior: The server inputs the message "I'm so happy today!" into the emotion engine.

[0646] Data processing: The emotion engine uses natural language processing technology to analyze the sentiment of the message.

[0647] Output: An emotion label, such as "joy."

[0648] Step 3:

[0649] The server stores the emotion determination results in a database.

[0650] Input: The emotion labels output by the emotion engine and the original message data.

[0651] Specific behavior: The server stores the message "I'm so happy today!" and the emotion label "joy" in a database.

[0652] Output: Message data and emotion labels stored in a database.

[0653] Step 4:

[0654] The server inputs past communication data and the emotional assessment results at that time into the generative AI model for analysis.

[0655] Input: Historical communication data and emotion labels obtained from the database.

[0656] Specific operation: The server inputs past messages labeled with the emotion "joy" into the generative AI model.

[0657] Data computation: Generative AI models analyze this data and extract the most effective communication situations.

[0658] Output: Patterns of effective communication.

[0659] Step 5:

[0660] The server generates feedback for the user based on the analysis results.

[0661] Input: Effective communication patterns output from a generative AI model.

[0662] Specific action: The server generates a feedback message saying, "The communication was most effective when you felt joy. Please try to keep this emotion in mind next time."

[0663] Output: The feedback message.

[0664] Step 6:

[0665] The server generates feedback and sends it to the terminal for display to the user.

[0666] Input: The server-generated feedback message.

[0667] Specific operation: The server sends a feedback message to the user's device.

[0668] Output: Feedback messages displayed on the terminal.

[0669] (Application example 3)

[0670] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0671] Conventional communication tools have difficulty accurately assessing users' emotions and providing optimal feedback based on those emotions. Furthermore, because they cannot assess emotions and provide feedback in real time, they are unable to contribute to improving users' communication skills. Furthermore, there is a lack of a way to provide effective feedback using past communication data.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0673] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for analyzing past communication data and the emotion determination results, means for providing feedback to the user based on the analysis results, and means for determining emotions in real time and displaying optimal feedback. This makes it possible to determine the user's emotions in real time and provide optimal feedback by utilizing past data.

[0674] "Communication Tools" means software or hardware that a User uses to exchange information with others.

[0675] "Generative AI" refers to artificial intelligence that generates new data based on input data.

[0676] "Emotion determination" refers to the process of identifying the user's current emotion from their facial expressions, writing, etc.

[0677] "Visualization" is the process of displaying data or information visually to make it easier for users to understand.

[0678] "Past Communication Data" means records of communications previously conducted by a User.

[0679] "Analysis" is the process of examining data in detail to find meaning and patterns.

[0680] "Feedback" refers to providing reactions and advice to the user's actions or state.

[0681] "Real-time" refers to data processing and information delivery occurring immediately, without delay.

[0682] A "smart device" is an electronic device that has internet connectivity and advanced computing capabilities.

[0683] The following system configuration will be described as an embodiment of the present invention.

[0684] System Configuration

[0685] The system includes a communication tool, generative AI, an emotion determination engine, a data analysis module, a feedback generation module, and a smart device (smart glasses).

[0686] Program processing

[0687] The server connects the text entered into the communication tool to the generative AI. The generative AI analyzes the entered text and determines the emotion of the person who wrote it. The emotion determination engine analyzes past communication data and the emotion determination results based on the determined emotion. The data analysis module extracts the situations in which the most effective communication took place and sends the results to the feedback generation module. The feedback generation module provides feedback to the user based on the analysis results.

[0688] The smart device (smart glasses) detects emotions in real time and displays appropriate feedback, allowing users to learn optimal responses in real time and improve their communication skills.

[0689] Hardware and software used

[0690] Hardware: Smart glasses (with camera and display)

[0691] Software: OpenCV (image processing), emotion detection engine, feedback generation module

[0692] Specific examples

[0693] For example, if a customer looks dissatisfied while a waiter is wearing smart glasses, the emotion recognition model will determine that emotion. Based on past data, feedback will be generated that "when a customer looks dissatisfied, a quick response and apology is effective," and the smart glasses will display the message "Please respond quickly and apologize."

[0694] Prompt Sentence Examples

[0695] Examples of prompts to input to a generative AI model include:

[0696] If a customer looks unhappy, learn from past data how to best respond and provide real-time feedback.

[0697] In this way, customer service staff can learn optimal responses in real time, improving customer satisfaction.

[0698] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0699] Step 1:

[0700] The server receives the text entered into the communication tool and connects to the generative AI. The input is the text entered by the user into the communication tool, and the output is the text to be analyzed that is passed to the generative AI. Specifically, the server sends the text data to the generative AI via an API.

[0701] Step 2:

[0702] The generative AI analyzes the received text and determines the emotion of the person who wrote it. The input is the text received in step 1, and the output is the emotion determination result. Specifically, the generative AI uses natural language processing technology to analyze the emotion of the text and assigns an emotion label such as positive, negative, or neutral.

[0703] Step 3:

[0704] The server analyzes past communication data and emotion determination results based on the emotions determined using the emotion determination engine. The input is the emotion determination results and past communication data, and the output is the analysis results. Specifically, the server retrieves past communication data from the database, compares it with the emotion labels obtained by the emotion determination engine, and extracts the most effective communication pattern.

[0705] Step 4:

[0706] The data analysis module extracts the situations in which the most effective communication took place and sends the results to the feedback generation module. The input is the analysis results obtained in step 3, and the output is the feedback data sent to the feedback generation module. Specifically, the data analysis module generates feedback that is optimal for the current situation based on past success stories.

[0707] Step 5:

[0708] The feedback generation module provides feedback to the user based on the analysis results. The input is the feedback data obtained in step 4, and the output is a feedback message to be provided to the user. Specifically, the feedback generation module generates a text message or a voice message and presents it to the user.

[0709] Step 6:

[0710] The smart device (smart glasses) determines emotions in real time and displays appropriate feedback. The input is a feedback message from the feedback generation module, and the output is feedback information displayed on the smart glasses display. Specifically, a message such as "Please respond promptly and apologize" is displayed on the smart glasses display.

[0711] In this way, users can learn optimal responses in real time and improve their communication skills.

[0712] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0713] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0714] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0715] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0716] [Second embodiment]

[0717] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0718] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0719] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0720] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0721] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0722] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0723] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0724] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0725] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0726] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0727] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0728] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0729] "Example 1"

[0730] One embodiment of the present invention is to use Google Chat as a communication tool. Text entered into Google Chat is linked to a generative AI, which determines human emotions from the text. The results are visualized in the app, and the user can use these results to select the timing of communication. Specifically, if the other person is showing positive emotions such as joy or relief, it is recommended that they communicate at that time.

[0731] "Example 2"

[0732] The generative AI also determines the optimal timing for communication based on the emotion assessment results. Specifically, the AI ​​learns past communication data and patterns of emotional fluctuations, and predicts the optimal timing for communication based on that. This prediction result is also presented to the user on the app, allowing the user to use it as a reference when selecting the timing for communication.

[0733] "Example 3"

[0734] Furthermore, the generative AI provides feedback to the user based on the emotion assessment results. Specifically, the AI ​​analyzes which emotions the user has displayed in the past that have resulted in the most effective communication, and provides feedback based on the results. This feedback allows the user to improve their own communication skills.

[0735] The processing flow of each embodiment will be described below.

[0736] "Example 1"

[0737] Step 1: User types a message into Google Chat.

[0738] Step 2: The input text is linked to the generative AI.

[0739] Step 3: The generative AI determines human emotions from the text.

[0740] Step 4: The results are visualized in the app.

[0741] Step 5: The user chooses the timing of communication based on the visualized results.

[0742] Select.

[0743] "Example 2"

[0744] Step 1: The generative AI learns past communication data and patterns of emotional fluctuations.

[0745] Step 2: The AI ​​predicts the optimal timing for communication based on the learned data.

[0746] Step 3: The prediction results are presented to the user on the app.

[0747] Step 4: The user selects the timing of communication based on the presented prediction results.

[0748] "Example 3"

[0749] Step 1: Generative AI analyzes which emotions the user has displayed in the past that resulted in the most effective communication.

[0750] Step 2: The AI ​​provides feedback to the user based on the analysis results.

[0751] Step 3: Users use the provided feedback to improve their communication skills.

[0752] Example 1

[0753] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] With conventional communication tools, it was difficult to accurately grasp the emotions of the other person, making it difficult to communicate at the right time. In addition, there was a lack of means to provide users with the results of emotion assessment, and users were not given enough support to communicate effectively.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0756] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for displaying the determination result on the user's device, and means for providing information for the user to select the timing of communication. This enables the user to accurately grasp the emotion of the other person and communicate effectively at the appropriate time.

[0757] A "communication tool" is software or an application that allows users to send and receive messages.

[0758] "Generative AI" is an artificial intelligence model that uses natural language processing technology to analyze input text and perform specific tasks (such as determining emotions).

[0759] "Emotion determination" is the process of identifying the emotions (e.g., joy, relief, anger, sadness, etc.) of the person who wrote the input text.

[0760] An "application" is a software program available to users that provides specific functions or services.

[0761] "User device" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0762] The "judgment result" is the emotion identification result obtained by the generative AI analyzing the input text.

[0763] "Visualization" is the process of visually displaying data or information, providing it in a way that is easy for users to understand.

[0764] "Information for selecting timing" refers to information such as emotion determination results that are provided to the user to determine the optimal timing for communicating with the other person.

[0765] This invention is a system that links text entered into a communication tool with generative AI, determines emotions, and provides the results to the user. Specifically, the server, terminal, and user elements work together.

[0766] The server receives the text entered by the user into the communication tool. For example, if the user sends the message "I had so much fun today!", this message arrives at the server. The server then sends the received message to a generative AI model (e.g., OpenAI's GPT-4). This generative AI model uses natural language processing technology to analyze the message and determine its sentiment.

[0767] The generative AI model determines the emotion from the message sent. For example, it determines the emotion "joy" from the message "I had so much fun today!" This process uses a machine learning algorithm. After the generative AI model determines the emotion, it sends the result back to the server. The server then sends this determination result to the user's device.

[0768] The user's device visualizes the emotion determination results received from the server. For example, the user's device might display "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[0769] As a concrete example, consider the case where a user sends a message to Google Chat saying, "I had so much fun today!" This message is sent to the server and passed to a generative AI model. The generative AI model determines the emotion "joy" from this message. The result of this determination is sent to the application, and the message "The other person is feeling happy" is displayed on the user's device. The user can use this information to decide when to communicate with the other person.

[0770] An example of a prompt might be:

[0771] Determine the emotion from the message, "Today was so much fun!"

[0772] By inputting this prompt into the generative AI model, the model determines the emotion "joy," thus providing support for users to communicate effectively.

[0773] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0774] Step 1:

[0775] The server receives the text entered by the user into the communication tool.

[0776] Specifically, when a user sends a message such as "Today was so much fun!", this message is received by the server.

[0777] Input: Text entered by the user into the communication tool

[0778] Output: The text received by the server

[0779] Step 2:

[0780] The server sends the received sentences to the generative AI model.

[0781] Specifically, the server sends the received message to a generative AI model (e.g., GPT-4), which then analyzes the message using natural language processing techniques.

[0782] Input: The text received by the server

[0783] Output: The sentence sent to the generative AI model

[0784] Step 3:

[0785] The generative AI model determines the sentiment from the text sent.

[0786] Specifically, the generative AI model analyzes the message "I had so much fun today!" and determines the emotion "joy." This process uses machine learning algorithms.

[0787] Input: The sentence sent to the generative AI model

[0788] Output: Determined emotion (e.g., "joy")

[0789] Step 4:

[0790] The generative AI model returns the emotion determination results to the server.

[0791] Specifically, the generative AI model determines the emotion of "joy" and then sends the result back to the server.

[0792] Input: Determined emotion

[0793] Output: Emotion judgment result sent back to the server

[0794] Step 5:

[0795] The server transmits the emotion determination result to the user's terminal.

[0796] Specifically, the server sends the emotion determination results received from the generative AI model to the user's device.

[0797] Input: Emotion judgment result returned to the server

[0798] Output: Emotion judgment result sent to the user's device

[0799] Step 6:

[0800] The user's device visualizes the emotion determination results.

[0801] Specifically, the user's device will display a message saying, "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[0802] Input: Emotion judgment result sent to the user's device

[0803] Output: Visualized emotion judgment result (e.g., "The other person is feeling happy")

[0804] In this way, the system provides support for users to communicate effectively.

[0805] (Application example 1)

[0806] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0807] In physical stores, in order to communicate smoothly with customers, it is important to properly understand the customer's emotions and provide customer service at the optimal timing. However, with conventional methods, it is difficult for store staff to accurately read the customer's emotions, which can result in a decline in the quality of customer service. To solve this problem, a system is needed that analyzes customer emotions in real time and provides appropriate feedback.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0809] In this invention, the server includes a means for linking text entered into the communication tool to a generative AI, a means for using the generative AI to determine the emotions of the person who wrote the text, and a means for visualizing the determination results in an application. This makes it possible to input the content of conversations with customers in a physical store and analyze their emotions in real time. By including a means for recommending the timing of customer service based on the analysis results, store staff can properly understand customer emotions and provide customer service at the optimal time.

[0810] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[0811] "Generative AI" is an artificial intelligence technology that analyzes input data and generates appropriate output for a specific task.

[0812] "Means of collaboration" are methods and technologies for exchanging data between different systems and software, making it mutually usable.

[0813] A "means for determining" is a method or technology for analyzing input data and evaluating the results based on specific criteria.

[0814] "Visualization means" refers to methods and techniques for displaying analysis results and data in a way that is easy for users to understand.

[0815] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[0816] "Customers" are people who visit a physical store to purchase goods or services.

[0817] "Conversation content" refers to the words and sentences exchanged between the customer and the store clerk.

[0818] "Means for analyzing emotions in real time" refers to methods or technologies for instantly analyzing input conversation content and determining customer emotions.

[0819] A "means for recommending the timing of customer service" is a method or technology for instructing store staff to respond to customers at the optimal timing based on the analysis results.

[0820] In order to implement the present invention, the following system configuration and program are required.

[0821] System Configuration

[0822] 1. Hardware:

[0823] Smartphone: Used by store clerks to input conversations with customers and check the analysis results.

[0824] Server: Provides the computational resources to run generative AI models and perform sentiment analysis.

[0825] 2. Software:

[0826] Communication tool: An application that allows store clerks to enter conversations with customers.

[0827] Generative AI model: An artificial intelligence technology that analyzes input text and determines emotions.

[0828] Google Cloud Natural Language API: An API for analyzing the sentiment score of a sentence.

[0829] OpenAI API: An API for running generative AI models and determining emotions.

[0830] Program processing

[0831] The server processes the data in the following steps:

[0832] 1. Data Entry:

[0833] The store clerk enters the details of the conversation with the customer into the communication tool on their smartphone.

[0834] 2. Emotion analysis:

[0835] The server sends the input conversation content to the Google Cloud Natural Language API and obtains an emotion score.

[0836] In addition, it uses the OpenAI API to perform detailed sentiment determination of the input text.

[0837] 3. Visualizing the results:

[0838] The server sends the analysis results to a smartphone application and displays the visualized results to the store clerk.

[0839] Based on the analysis, store staff are encouraged to continue serving customers if they are displaying positive emotions.

[0840] Specific examples

[0841] For example, consider the case where a store clerk inputs the following conversation content:

[0842] Type: "It's a beautiful day today. Can I see your new items?"

[0843] The server sends this input to the Google Cloud Natural Language API to obtain an emotion score, then uses the OpenAI API to generate the following prompt sentence and perform emotion determination:

[0844] Example prompt sentence:

[0845] Determine the sentiment of the following sentences: The weather is very nice today. Can you show me your new products?

[0846] If the analysis results in a positive emotion, the smartphone application will display the message, "The customer is expressing a positive emotion. Please continue serving the customer."

[0847] In this way, store staff can properly understand the customer's emotions and provide service at the optimal time.

[0848] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0849] Step 1:

[0850] The user (store clerk) inputs the contents of the conversation with the customer into the communication tool on the smartphone. The inputted contents of the conversation are saved on the smartphone as text data.

[0851] Step 2:

[0852] The device (smartphone) sends the entered conversation content to the server. The data sent is the conversation content in text format.

[0853] Step 3:

[0854] The server sends the received conversation content to the Google Cloud Natural Language API and obtains the sentiment score. The input data is the conversation content in text format, and the output data is the sentiment score (positive, negative, neutral, etc.).

[0855] Step 4:

[0856] The server uses the OpenAI API to perform detailed emotion determination based on the emotion score. The server generates a prompt from the conversation content and sends it to the OpenAI API. The input data is the prompt, and the output data is the detailed emotion determination result.

[0857] Step 5:

[0858] The server integrates the emotion analysis results obtained from the Google Cloud Natural Language API and the OpenAI API to generate a final emotion judgment result. The input data is the emotion score and detailed emotion judgment result, and the output data is the integrated emotion judgment result.

[0859] Step 6:

[0860] The server then transmits the integrated emotion determination results to the smartphone. The transmitted data is a visualized version of the emotion determination results.

[0861] Step 7:

[0862] The device (smartphone) receives the emotion assessment results and displays them in the application. The displayed data is information about the customer's emotional state and the recommended timing for serving the customer.

[0863] Step 8:

[0864] The user (store clerk) refers to the displayed emotion determination result and adjusts the timing of serving the customer. Specifically, if the customer shows a positive emotion, the store clerk continues serving the customer, and if the customer shows a negative emotion, the store clerk reconsiders the timing of serving the customer.

[0865] Example 2

[0866] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0867] Conventional communication tools have difficulty accurately assessing users' emotions and providing optimal communication timing based on those emotions. Furthermore, there are also insufficient ways for users to utilize past communication data to obtain feedback for effective communication. This has resulted in users being unable to communicate effectively, leading to a decline in the quality of communication.

[0868] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for linking a sentence input to a communication tool to a generative AI, means for determining the emotion of the person who wrote the sentence using the generative AI, means for visualizing the determination result using an application, means for collecting past communication data, means for preprocessing the collected data, means for learning emotion fluctuation patterns, means for predicting the optimal communication timing, and means for presenting the prediction result to the user. This allows the user to know the optimal communication timing based on the emotion fluctuation pattern, enabling effective communication.

[0869] "Communication Tools" refers to software or hardware that allows users to exchange information with other users through messages, voice, video, etc.

[0870] "Generative AI" refers to artificial intelligence that uses natural language processing and machine learning techniques to generate and analyze data such as text and voice.

[0871] "Emotion determination" refers to the process of analyzing a user's emotions from text or voice data entered by the user and classifying them into specific emotional categories.

[0872] "Visualization" refers to displaying analysis results and data in a way that is easy for users to understand, such as in graphs, charts, or text format.

[0873] "Past communication data" refers to records of messages, voice and video calls, etc. that a user has previously made.

[0874] "Preprocessing" refers to a series of operations performed on collected data to transform it into an analyzable format, including denoising, tokenization, and normalization.

[0875] "Emotional patterns" refer to trends and patterns in data that show how users' emotions change over time and across situations.

[0876] "Optimal communication timing" refers to the time period or timing when a user is predicted to be able to communicate most effectively with other users.

[0877] "Prediction results" refer to the conclusions and estimates that generative AI reaches through analysis and learning.

[0878] MODE FOR CARRYING OUT THE INVENTION

[0879] This invention provides a system for enabling users to communicate effectively with other users. The system uses a generative AI model to determine the user's emotions and predict the optimal timing for communication. A specific embodiment of this system is described below.

[0880] 1. Program Generation

[0881] The server uses a generative AI model to generate a program that determines emotions and optimizes communication timing. This program is built using machine learning libraries such as Python and TensorFlow.

[0882] 2. Program Processing

[0883] The server first collects the user's past communication data, including text messages, voice data, and facial expression data. For example, it collects the user's message history with friends and facial expression data during video calls.

[0884] The server then preprocesses the collected data. For text data, unnecessary characters and symbols are removed and the data is tokenized. For audio data, noise is removed and the audio is converted into text. For facial expression data, facial recognition technology is used to extract facial expression features.

[0885] The preprocessed data is then input into an emotion detection model, which uses generative AI models such as BERT and GPT-3 to analyze user emotions. For example, it can detect emotions such as "joy," "sadness," and "anger" from text messages.

[0886] Once emotion determination is complete, the server performs time series analysis to learn the emotional fluctuation patterns. This is done using recurrent neural networks (RNNs) such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit). After learning is complete, the server predicts the optimal timing for communication.

[0887] The prediction results are presented to the user through the application. The user can use this information to communicate at the optimal time. For example, the app may notify the user, "Now is the best time to message a friend."

[0888] 3. Examples of concrete examples and prompts

[0889] For example, when a user is messaging a friend, the server analyzes past message data to learn the user's emotional fluctuation patterns.The server then predicts when the friend is most likely to respond and notifies the user of that timing.

[0890] Specific prompt examples:

[0891] "Analyze message exchange data with friends and predict the best timing for communication."

[0892] In this way, users can communicate more effectively with the help of generative AI.

[0893] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0894] Step 1: Data collection

[0895] The server collects the user's past communication data. Specifically, it acquires message history, voice data, and facial expression data during video calls from the user's smartphone or PC. The input is data from the user's device, and the output is the collected raw data.

[0896] Step 2: Data Preprocessing

[0897] The server preprocesses the collected data. In the case of text data, unnecessary characters and symbols are removed and the data is tokenized. In the case of audio data, noise is removed and the audio is converted into text. In the case of facial expression data, facial expression features are extracted using facial recognition technology. The input is the collected raw data, and the output is the preprocessed data.

[0898] Step 3: Emotion determination

[0899] The server inputs the preprocessed data into an emotion detection model. The emotion detection model uses generative AI models such as BERT and GPT-3 to analyze the user's emotions. For example, it detects emotions such as "joy," "sadness," and "anger" from a text message. The input is the preprocessed data, and the output is the emotion detection result.

[0900] Step 4: Learn emotional patterns

[0901] The server learns patterns of emotional fluctuations based on the results of emotion determination. For this, it uses recurrent neural networks (RNNs) such as LSTM and GRU. For example, it learns whether a user tends to show the emotion of "happiness" at a certain time of day. The input is the result of emotion determination, and the output is the learned pattern of emotional fluctuations.

[0902] Step 5: Anticipate communication timing

[0903] The server predicts the optimal timing for communication based on the learned emotional fluctuation patterns. For example, it predicts the time of day when a user can communicate most effectively with friends. The input is the learned emotional fluctuation patterns, and the output is the predicted optimal timing for communication.

[0904] Step 6: Presenting the results

[0905] The server presents the prediction results to the user through the application. The user can use this information to communicate at the optimal time. For example, the application may notify the user, "Now is the best time to send a message to a friend." The input is the predicted optimal communication timing, and the output is the notification to the user.

[0906] (Application example 2)

[0907] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0908] With conventional communication tools, it was difficult to accurately assess users' emotions and support communication at the optimal timing. In addition, in physical stores, it was difficult for store staff to grasp customers' emotions in real time and determine the optimal timing to interact with them. This resulted in inconsistent customer service quality, making it difficult to improve customer satisfaction.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0910] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning past communication data and patterns of emotional fluctuations, means for predicting the optimal timing for communication, means for presenting the prediction results to a user, means for the user to select the timing of communication based on the presented prediction results, and means for supporting customer service in a physical store. This enables the user to select the optimal timing for communication based on emotions, and enables the emotions of customers in a physical store to be grasped in real time and the optimal timing for customer service to be determined.

[0911] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[0912] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[0913] "Emotion determination" is the process of analyzing a user's emotional state from input text or data and identifying a specific emotion.

[0914] "Visualization" refers to displaying data and information visually and providing it in a form that is easy for users to understand.

[0915] "Past communication data" refers to records or histories of previous communications, including message content and timing.

[0916] "Emotional fluctuation patterns" refer to emotional tendencies or patterns that change over time or in response to specific situations.

[0917] "Optimal communication timing" refers to the timing for the most effective communication based on the user's emotional state and past data.

[0918] "Predicted results" refers to future results and trends derived by generative AI through analysis and learning.

[0919] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[0920] "Customer service" refers to the service and response provided by store staff to customers.

[0921] As an embodiment of the present invention, a system for supporting customer service in a brick-and-mortar store will be described as an example.

[0922] First, the server provides a means for linking text entered into the communication tool to the generative AI. Specifically, text entered by the user using a device such as a smartphone or tablet is sent to the server via the internet. The server then passes this text to the generative AI, which determines its emotion.

[0923] Next, generative AI analyzes the input text and determines the user's emotions. Natural language processing techniques and machine learning algorithms are used to determine emotions. For example, a sentiment analysis model can be built using the Python libraries scikit-learn and TensorFlow.

[0924] The server visualizes the emotion results in an application. The application displays the emotion results on the screen of a smartphone or tablet, allowing the user to understand them intuitively. For example, the emotion score could be displayed as a graph or icon.

[0925] Furthermore, the server provides a means to learn past communication data and emotional fluctuation patterns, using past communication history stored in a database. Using this data, the server trains a model for the generative AI to predict optimal communication timing.

[0926] To predict the optimal timing for communication, the generative AI receives the current time and emotion score as input and generates a prediction result. This prediction result is sent to the application via the server and presented to the user. The user can refer to this prediction result to communicate at the optimal time.

[0927] In physical stores, store clerks use their smartphones to capture customers' facial expressions. The application calculates an emotion score in real time and suggests the optimal timing for customer interaction. This allows store clerks to understand the emotional state of customers and provide assistance at the optimal time.

[0928] As a concrete example, the following prompt might be input to a generative AI:

[0929] "It is currently 10 o'clock. The customer's sentiment score is 0.7. Please predict the optimal time to serve them."

[0930] In this way, users can choose the optimal timing for communication based on their emotions, and even in physical stores, they can grasp the emotions of customers in real time and determine the optimal timing for customer service.

[0931] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0932] Step 1:

[0933] A user inputs text into a communication tool using a device such as a smartphone or tablet.

[0934] Input: Text entered by the user

[0935] Output: The input text is sent from the terminal to the server.

[0936] Specific actions: The user opens an application on the device, enters text in the text box, and presses the send button.

[0937] Step 2:

[0938] The server receives the text sent from the device and connects it to the generative AI.

[0939] Input: Text sent from the terminal

[0940] Output: Text passed to the generative AI

[0941] Specific operation: The server receives an HTTP request, analyzes the text data, and passes it to the generative AI API.

[0942] Step 3:

[0943] The generative AI analyzes the received text and determines the user's emotions.

[0944] Input: Text given to the generative AI

[0945] Output: Sentiment score (e.g. 0.7)

[0946] How it works: Generative AI uses natural language processing techniques to analyze text and calculates a sentiment score using a sentiment analysis model (e.g., scikit-learn or TensorFlow).

[0947] Step 4:

[0948] The server visualizes the emotion scores received from the generative AI in the application.

[0949] Input: Sentiment score

[0950] Output: Visualized sentiment scores (e.g., graphs and icons)

[0951] Specific operation: The server receives the emotion score, generates data to reflect in the application UI, and sends it to the device.

[0952] Step 5:

[0953] The server learns past communication data and patterns of emotional fluctuations.

[0954] Input: Historical communication data

[0955] Output: The trained model

[0956] Specific operation: The server retrieves past communication history from the database and passes it to the generative AI to train the model.

[0957] Step 6:

[0958] The generative AI takes the current time and emotion score as input and predicts the optimal timing for communication.

[0959] Input: Current time, Sentiment score

[0960] Output: Prediction of optimal communication timing

[0961] How it works: The generative AI uses a trained model to predict the optimal timing using the current time and emotion score as input.

[0962] Step 7:

[0963] The server presents the prediction results to the user.

[0964] Input: Prediction results of optimal communication timing

[0965] Output: Predictions presented to the user

[0966] Specific operation: The server receives the prediction results, generates data to be reflected in the application UI, and sends it to the device.

[0967] Step 8:

[0968] The user refers to the presented prediction results and communicates at the optimal time.

[0969] Input: Predicted results

[0970] Output: User-selected timing of communication

[0971] Specific behavior: The user checks the results presented by the application and sends a message at the optimal time.

[0972] Step 9:

[0973] When providing customer service support in a physical store, store staff carry smartphones and use the camera to capture the facial expressions of customers.

[0974] Input: Customer facial expression data

[0975] Output: Real-time emotion score

[0976] How it works: A store associate uses a smartphone camera to capture a customer's facial expression, and the application determines an emotion score in real time.

[0977] Step 10:

[0978] The application suggests the best time to serve customers.

[0979] Input: Real-time emotion score

[0980] Output: Recommendation of optimal timing for customer service

[0981] Specific operation: The application calculates the optimal timing to serve the customer based on the emotion score and notifies the store clerk.

[0982] Example 3

[0983] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] Conventional communication tools have difficulty accurately assessing users' emotions and providing feedback based on them. Furthermore, there have been no systems that analyze past emotional data and the effectiveness of communication to provide effective advice to users. As a result, users have been unable to obtain specific guidance on how to improve their communication skills.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0986] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an application, means for analyzing past emotion data and the effectiveness of communication, and means for providing feedback to the user based on the analysis results. This allows the user to receive effective feedback based on their own emotions and obtain specific guidelines for improving their communication skills.

[0987] "Communication tools" is a general term for software or hardware that allows users to exchange information with others through text, voice, etc.

[0988] "Generative AI" is a general term for artificial intelligence technology that generates new information based on input data.

[0989] "Means for determining emotions" is a general term for algorithms and technologies that analyze a user's emotions from input text or data and identify specific emotions.

[0990] "Application" is a general term for a software program designed to provide a specific function or service.

[0991] "Visualization means" is a general term for techniques and methods for visually displaying data and information so that users can easily understand it.

[0992] "Past emotion data" is a general term for records and data relating to emotions expressed by a user in the past.

[0993] "Communication effectiveness" is a general term for the impact or results that a particular communication method has on users.

[0994] "Analytical methods" is a general term for techniques and methods used to analyze data and find specific patterns and trends.

[0995] "Means for providing feedback" is a general term for techniques and methods for providing advice and information to users based on analysis results.

[0996] This invention is a system that links text entered by a user through a communication tool to a generative AI, determines emotions, and provides feedback based on the results. A specific embodiment of this system is described below.

[0997] Hardware and software used

[0998] server

[0999] The server provides the computational resources to run the generative AI model. Specifically, it uses the following software and hardware:

[1000] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[1001] Database Management System: Relational database such as MySQL

[1002] Machine learning algorithms: Machine learning libraries such as scikit-learn

[1003] Terminal

[1004] A terminal is a device for users to provide input data and receive feedback. Specifically, it uses the following software and hardware:

[1005] Browser: A web browser such as Google Chrome

[1006] Communication protocol: HTTP / HTTPS

[1007] User

[1008] A user uses a communication tool to input text data and receive feedback.

[1009] Program processing

[1010] server

[1011] The server receives text data sent from the device and uses the generative AI model to determine the emotion. For example, if a user enters "Recently, communication at work has been going poorly," the server analyzes the text data and determines the emotion as "anxiety."

[1012] The server then uses a database management system to analyze the user's past emotional data and the effectiveness of communication. For example, it might determine that presenting specific solutions to problems was effective when the user expressed anxiety.

[1013] The server then uses the generative AI model to generate feedback for the user, such as "By providing specific solutions to the problem, you may be able to improve communication in the workplace."

[1014] Terminal

[1015] The device displays the feedback received from the server to the user, who can then check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[1016] Examples of specific examples and prompts

[1017] As a concrete example, consider the case where a user inputs, "Recently, communication at work has not been going well." The server receives this input and uses a generative AI model to determine the user's emotion as "anxiety." It then analyzes past data to identify the most effective communication method when the user expressed "anxiety." For example, suppose "presenting specific solutions to the problem" was effective. Based on this information, the server provides the user with feedback such as, "Providing specific solutions to the problem may improve communication at work."

[1018] Example prompt sentence:

[1019] User: Communication at work has been difficult lately.

[1020] Server: Determines user sentiment and analyzes the most effective communication method based on past data.

[1021] Server: When users expressed anxiety, the most effective thing we did was to offer concrete solutions to their problems.

[1022] Server: Providing concrete solutions to problems can improve communication in the workplace.

[1023] In this way, a system is realized in which the server, the terminal, and the user work together to provide effective feedback to the user. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1024] Step 1:

[1025] The user provides the input data.

[1026] A user inputs text data through the interface of a communication tool. For example, the user inputs, "Recently, communication at work has not been going well." The input data is in text format and expresses the user's feelings and situation.

[1027] Step 2:

[1028] The terminal sends the input data to the server.

[1029] The terminal converts the text data entered by the user into JSON format and sends it to the server as an HTTP POST request. The input data is a text document, and the output is the JSON format data sent to the server.

[1030] Step 3:

[1031] The server uses a generative AI model to determine emotions.

[1032] The server inputs the received text data into a generative AI model (e.g., GPT-4) to determine the user's emotion. The input data is text data in JSON format, and the output is the emotion determination result (e.g., "anxiety"). The server invokes the generative AI model, analyzes the text data, and identifies the emotion.

[1033] Step 4:

[1034] The server analyzes past data and identifies effective communication methods.

[1035] The server uses a database management system (e.g., MySQL) to analyze the user's past emotional data and the effectiveness of communication. The input data is the emotional judgment result and past emotional data, and the output is an effective communication method (e.g., "present a specific solution to the problem"). The server analyzes the data using a machine learning algorithm (e.g., scikit-learn) and identifies the optimal communication method.

[1036] Step 5:

[1037] The server generates the feedback.

[1038] The server generates feedback using a generative AI model based on the identified effective communication methods. The input data are effective communication methods, and the output is feedback to the user (e.g., "Providing specific solutions to problems may improve communication in the workplace"). The server then invokes the generative AI model again to generate feedback in natural language.

[1039] Step 6:

[1040] The device displays the feedback to the user.

[1041] The device displays the feedback received from the server to the user. The input data is the feedback text, and the output is the feedback displayed on the user's screen. The user can check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[1042] (Application example 3)

[1043] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1044] Conventional communication tools have difficulty accurately determining a user's emotions and providing appropriate feedback based on them. Security services are required to determine a user's emotions in real time and provide appropriate responses, but there has been a lack of effective means to achieve this. Furthermore, there has been no system that utilizes past emotion data to suggest the most effective communication method.

[1045] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence using the generative AI, a means for visualizing the determination result in an app, a means for analyzing past emotion data and proposing the most effective communication method, and a means for determining emotion in real time and providing feedback according to the security situation. This makes it possible to accurately determine a user's emotion and provide appropriate feedback. Furthermore, in security services, emotions can be determined in real time and appropriate responses can be taken. Furthermore, the most effective communication method can be proposed by utilizing past emotion data.

[1046] A "communication tool" is software or hardware that allows a user to exchange information with others through text or voice.

[1047] "Generative AI" is an artificial intelligence technology that generates new data based on input data, and is particularly used in natural language processing and image generation.

[1048] "Emotion determination" is the process of identifying a user's emotional state from the text or voice input they provide.

[1049] "Visualization" is a method of visually displaying data and information to make it easier for users to understand.

[1050] "Past emotion data" is a record of emotions a user has expressed in the past, and analyzing this data can reveal patterns and trends.

[1051] The "most effective communication method" is the method that provides the best response and feedback to the user's emotional state.

[1052] "Real-time" refers to data processing and feedback occurring immediately, without delay.

[1053] "Security situation" refers to the safety-related state and risks faced by users and systems.

[1054] "Feedback" is a response or advice provided to a user's actions or status.

[1055] The system for implementing this invention mainly consists of three elements: a server, a terminal, and a user. The function of each element will be explained below.

[1056] server

[1057] The server has a means of linking text entered into the communication tool to the generative AI. Specifically, it receives text data entered by the user into the device and sends it to the generative AI. The generative AI analyzes the entered text data and determines the user's emotions. The results of this emotion determination are saved by the server and used for later analysis.

[1058] Furthermore, the server has a means of analyzing past emotional data and proposing the most effective communication method, which allows it to learn which emotions the user expressed in the past resulted in the most effective feedback, and use this knowledge for future communications.

[1059] Terminal

[1060] The device is responsible for sending the text data entered by the user to the server. It also has a means of visualizing the emotion assessment results and feedback received from the server to the user, allowing the user to check their own emotional state in real time.

[1061] The device also has a means of detecting emotions in real time and providing feedback according to the security situation. For example, if a user inputs "I'm very anxious right now," the device will provide advice such as "Take a deep breath and calm down" based on the feedback received from the server.

[1062] User

[1063] Users input text data through their devices and receive feedback from the server. This allows users to understand their own emotional state and take appropriate action. Furthermore, by receiving suggestions on the most effective communication methods based on past emotional data, users can improve their communication skills.

[1064] Hardware and software used

[1065] Hardware: Smartphone (iOS or Android)

[1066] Software: Python, scikit-learn (machine learning library), joblib (saving and loading models)

[1067] Specific examples

[1068] For example, if a user types "I'm very anxious right now," the server sends this text data to the generative AI, which determines the emotion as "anxiety." The server then analyzes past emotion data and suggests the most effective feedback: "Take a deep breath and calm down." The device displays this feedback to the user, who can then act accordingly.

[1069] Prompt Sentence Examples

[1070] "Write a Python program that determines the emotion of text entered by a user and provides the most effective feedback. Use a pre-trained model to determine the emotion, and a model trained on past data to provide the feedback."

[1071] In this way, the present invention accurately determines the user's emotions and provides appropriate feedback, enabling the user to improve their communication skills and respond appropriately to security situations.

[1072] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1073] Step 1:

[1074] The user inputs text data into the terminal. The input text data is used as initial data for determining the user's emotion. Input: Text data input by the user. Output: Text data sent to the terminal.

[1075] Step 2:

[1076] The device sends the input text data to the server. The server receives this data and connects it to the generative AI. Input: Text data sent from the device. Output: Text data sent to the server.

[1077] Step 3:

[1078] The server sends the received text data to the generative AI, which determines the emotion. The generative AI analyzes the text data and determines the user's emotion. Input: Text data sent to the server. Output: Emotion data determined by the generative AI.

[1079] Step 4:

[1080] The server stores the emotional data received from the generative AI and analyzes it together with past emotional data. This generates data to suggest the most effective communication methods. Input: Emotional data received from the generative AI. Output: Analyzed communication method data.

[1081] Step 5:

[1082] The server generates the most effective feedback based on the analysis results and sends it to the device. Input: Analyzed communication method data. Output: Feedback data sent to the device.

[1083] Step 6:

[1084] The device visualizes the feedback data received from the server to the user, allowing the user to check their own emotional state and the appropriate feedback in real time. Input: Feedback data received from the server. Output: Feedback information displayed to the user.

[1085] Step 7:

[1086] The user adjusts their behavior based on the feedback displayed on the device. This allows the user to take the optimal action based on their emotional state. Input: Feedback information displayed on the device. Output: User behavior adjustment.

[1087] In this way, a system is realized that can accurately determine a user's emotions and provide appropriate feedback, enabling them to improve their communication skills and respond appropriately to security situations.

[1088] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1089] "Example 1"

[1090] In one embodiment of the present invention, when a user inputs a sentence into a communication tool, the sentence is linked to a generative AI. The generative AI uses an emotion engine to determine the user's emotion from the sentence. The emotion engine infers the emotion from, for example, words, phrases, emojis, etc. in the sentence. The determined emotion is visualized in the app, and the user can use the results as a reference when selecting the timing of communication.

[1091] "Example 2"

[1092] In another embodiment, the generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns patterns of emotional fluctuation from past communication data. It then predicts the optimal timing for communication based on the learned data. The predicted timing is presented to the user on the app, allowing the user to select the timing of communication based on this information.

[1093] "Example 3"

[1094] In yet another embodiment, the generative AI uses an emotion engine to analyze what emotions the user displayed in the past to achieve the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotion assessment results at that time. It then provides feedback to the user based on these results. The user can use this feedback to improve their own communication skills.

[1095] The processing flow of each embodiment will be described below.

[1096] "Example 1"

[1097] Step 1: The user enters text into the communication tool.

[1098] Step 2: The input text is linked to the generative AI.

[1099] Step 3: The generative AI uses an emotion engine to determine the user's emotion from the text. The emotion engine infers emotion from words, phrases, emojis, etc. in the text.

[1100] Step 4: The determined emotions are visualized in the app, and the user can use this result to choose the timing of communication.

[1101] "Example 2"

[1102] Step 1: The generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns emotional fluctuation patterns from past communication data.

[1103] Step 2: The generative AI predicts the optimal timing for communication based on the learned data.

[1104] Step 3: The predicted timing is presented to the user on the app, and the user uses this information to choose the timing of communication.

[1105] "Example 3"

[1106] Step 1: The generative AI uses an emotion engine to analyze what emotions the user displayed in the past that resulted in the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotional assessment results at that time.

[1107] Step 2: The generative AI provides feedback to the user based on the results. Step 3: The user uses this feedback to improve their communication skills.

[1108] Example 1

[1109] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1110] Conventional communication tools have had the problem that it is difficult for users to accurately grasp the emotions of others, making it difficult to communicate at the right time. In addition, there are insufficient means for providing users with the emotion assessment results, and it is unclear how users should use the results.

[1111] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence by the generative AI, a means for visualizing the determination result in an app, and a means for the user to select the timing of communication based on the emotion determination result. This allows the user to accurately understand the other person's emotion and communicate at an appropriate time.

[1112] A "communication tool" is software or a platform that allows users to exchange messages with other users.

[1113] "Generative AI" is an artificial intelligence technology that analyzes input data and generates specific results.

[1114] The "means for determining emotion" is an algorithm or system for analyzing emotion from input text and identifying that emotion.

[1115] "Visualization means" refers to an interface or function that displays the analysis results in a form that is easy for the user to understand.

[1116] The "means for the user to select the timing of communication based on the emotion determination result" is a guideline or function that allows the user to determine the optimal timing of communication based on the displayed emotion determination result.

[1117] The "means for providing feedback" is a function for providing advice and information to the user based on the emotion determination result.

[1118] MODE FOR CARRYING OUT THE INVENTION

[1119] This invention is a system that links sentences entered by users into a communication tool with generative AI, determines emotions from the sentences, and visualizes the determination results in an app. A specific embodiment of this system is described below.

[1120] A user types a sentence into Google Chat

[1121] A user uses a communication tool such as Google Chat to type a message, for example, "Today was a great day!" This message is then sent over the Internet to a server.

[1122] The server connects the text to a generative AI

[1123] The server receives messages sent by users using the Google Chat API. The received messages are sent to the generative AI, which has an emotion engine and prepares the text for analysis.

[1124] Generative AI analyzes text and determines emotions

[1125] Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in input text. For example, it can determine positive emotions from the word "fun." This analysis uses natural language processing technology and machine learning algorithms.

[1126] The determined emotion is returned to the server

[1127] The generative AI returns the analysis results to the server, which then receives the results and proceeds to the next step.

[1128] The server sends the emotion determination result to the app.

[1129] The server sends the emotion determination results to the user's device, which receives the results and prepares them for display in the app.

[1130] The device displays the emotion determination result in the app.

[1131] The device displays the received emotion determination result in the app. The user can check the emotion determination result through the app. For example, the app may display "positive emotion."

[1132] Users choose when to communicate

[1133] The user can choose the timing of communication based on the emotion assessment results displayed on the app. For example, if the other person is showing positive emotions, they can choose to reply at that time.

[1134] Specific examples

[1135] A user types "Today was such a fun day!" into Google Chat and presses the send button. This message is sent to the server, which receives the text via the Google Chat API. The server connects the received text to a generative AI, which analyzes the word "fun" and determines the positive emotion. The generative AI returns the positive emotion determination result to the server, which then sends the emotion determination result to the user's device. The emotion determination result received by the device is displayed in the app, and is displayed as "positive emotion." The user checks the "positive emotion" displayed in the app and chooses to reply at that time.

[1136] Prompt Sentence Examples

[1137] "Create a generative AI program that analyzes text entered in Google Chat and determines the emotion. Include a function that uses an emotion engine to infer emotion from words, phrases, emojis, etc. in the text, and visualize the results in the app."

[1138] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1139] Step 1: User types a sentence into Google Chat

[1140] What happens: A user opens the Google Chat app and types a message, for example, "Today was such a fun day!"

[1141] Input: Text entered by the user

[1142] Output: Message sent to Google Chat

[1143] Step 2: The input text is sent to the server

[1144] How it works: When a user sends a message, the text is sent over the Internet to a server, which then receives the message using the Google Chat API.

[1145] Input: Message sent from Google Chat

[1146] Output: Message received by server

[1147] Step 3: The server connects the text to the generative AI

[1148] Specific operation: The server sends the received text to the generative AI, which has an emotion engine and prepares to analyze the text.

[1149] Input: Message received by the server

[1150] Output: Message sent to the generative AI

[1151] Step 4: Generative AI analyzes the text and determines the sentiment

[1152] How it works: Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in the input text. For example, it determines positive emotions from the word "fun."

[1153] Input: Message sent to the generative AI

[1154] Output: Emotion determination result (e.g., positive emotion)

[1155] Step 5: The determined emotion is returned to the server

[1156] Specific operation: The generative AI returns the analysis results to the server, which receives the results and proceeds to the next step.

[1157] Input: Emotion determination result

[1158] Output: Emotion judgment result returned to the server

[1159] Step 6: The server sends the emotion result to the app

[1160] Specific operation: The server sends the emotion determination result to the user's device. The device receives the result and prepares to display it in the app.

[1161] Input: Emotion judgment result returned to the server

[1162] Output: Emotion judgment result sent to the user's device

[1163] Step 7: The device displays the emotion detection result in the app.

[1164] Specific operation: The device displays the received emotion judgment result in the app. The user can check the emotion judgment result through the app. For example, the app will display "positive emotion."

[1165] Input: Emotion judgment result sent to the user's device

[1166] Output: Emotion judgment results displayed on the app

[1167] Step 8: Users choose when to communicate

[1168] Specific operation: The user chooses the timing of communication based on the emotion judgment results displayed in the app. For example, if the other person is showing positive emotions, they choose to reply at that time.

[1169] Input: Emotion judgment result displayed on the app

[1170] Output: User-selected timing of communication

[1171] (Application example 1)

[1172] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1173] With conventional communication tools, it was difficult for users to accurately grasp the emotions of the other party, making it difficult to communicate at the right time. Also, in physical stores, store staff are required to understand customers' emotions in real time and respond optimally, but there was a lack of effective means to do so. This led to the issue of not being able to improve customer satisfaction or provide efficient customer service.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1175] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an app, means for converting voice data into text, means for displaying the emotion determination result on a display device, and means for providing a response guide according to the emotion. This allows users to understand the other person's emotion in real time and communicate and respond to customers at the optimal time.

[1176] A "communication tool" is software or hardware that allows users to exchange information with each other through text or voice.

[1177] "Generative AI" is an artificial intelligence system that generates new information based on input data.

[1178] "Emotion determination" is the process of identifying and classifying the emotions contained in the content of input text or audio data.

[1179] "Visualization" is a method of visually displaying data or information so that users can easily understand it.

[1180] "Audio data" refers to data in which audio is recorded in digital format.

[1181] "Text conversion" is the process of converting audio data into text data.

[1182] A "display device" is a device for visually displaying information, and includes displays and screens.

[1183] A "response guide" is a set of instructions or advice that suggests appropriate actions or words to use in a given situation or emotion.

[1184] As an embodiment of the present invention, a system for supporting customer service in brick-and-mortar stores will be described as an example. This system is configured using hardware and software such as smart glasses, a microphone, a generative AI model, an emotion engine, and a display device.

[1185] First, a microphone built into the smart glasses captures what the customer says. The captured voice data is converted into text using the Google Cloud Speech-to-Text API. The converted text is then sent to a generative AI model (e.g., OpenAI GPT-4) and emotion is determined using an emotion engine (e.g., IBM Watson Tone Analyzer).

[1186] The detected emotion is displayed on the smart glasses' display. For example, positive emotions (such as joy and relief) are displayed in green, while negative emotions (such as anger and anxiety) are displayed in red. A guide to how to respond based on the emotion is also displayed. For example, if a customer is feeling anxious, phrases or actions to reassure them are suggested.

[1187] This system allows store staff to understand customer emotions in real time and respond optimally, thereby improving customer satisfaction and enabling more efficient customer service.

[1188] For example, the following prompt is input to a generative AI model:

[1189] Customer says: "I'd like to learn more about this product, but I'm a little unsure."

[1190] Prompt: "Determine the customer's sentiment from this statement."

[1191] Based on this prompt, the generative AI model determines the customer's emotions and displays the results on the smart glasses' display, allowing the store clerk to respond optimally based on the customer's emotions.

[1192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1193] Step 1:

[1194] A microphone in the smart glasses captures what the customer says.

[1195] Input: Customer voice

[1196] Output: Audio data

[1197] Specific operation: The microphone built into the smart glasses records the customer's speech in real time and saves it as audio data.

[1198] Step 2:

[1199] The server converts the audio data into text.

[1200] Input: Audio data

[1201] Output: Text data

[1202] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1203] Step 3:

[1204] The server sends the text data to a generative AI model to determine the emotion.

[1205] Input: Text data

[1206] Output: Emotion determination result

[1207] Specific operation: The server sends text data to a generative AI model (e.g., OpenAI GPT-4), which then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to determine the emotion.

[1208] Step 4:

[1209] The server displays the emotion determination result on the display of the smart glasses.

[1210] Input: Emotion determination result

[1211] Output: Display

[1212] Specific operation: The server sends the emotion judgment result to the smart glasses display, and positive emotions are displayed in green and negative emotions in red.

[1213] Step 5:

[1214] The server provides a response guide based on the emotion.

[1215] Input: Emotion determination result

[1216] Output: Corresponding guide

[1217] Specific operation: Based on the emotion determination result, the server generates a response guide and displays it on the smart glasses display. For example, if the customer is feeling anxious, it will suggest phrases or actions to reassure them.

[1218] Example 2

[1219] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1220] With conventional communication tools, it was difficult to accurately assess a user's emotions and provide optimal communication timing based on the results. In addition, there was a lack of means for users to use past communication data to understand patterns of emotional fluctuations and communicate effectively. This meant that users were unable to communicate at the appropriate time, resulting in a decline in the quality of communication.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1222] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for collecting past communication data, means for extracting patterns of emotional fluctuation from the collected data, means for predicting the optimal communication timing based on the extracted patterns, and means for presenting the prediction result in the application. This enables the user to understand patterns of emotional fluctuation and communicate at the optimal timing.

[1223] "Communication Tools" means software or applications that allow users to exchange messages or information with other users.

[1224] "Generative AI" is a system that uses artificial intelligence technology to generate new information and results from input data.

[1225] "Emotion determination" is the process of analyzing a user's emotions from input text and data and identifying the type and intensity of those emotions.

[1226] An "application" is a software program that a user uses to perform a specific task.

[1227] "Data collection" is the process of collecting and storing a user's past communication data.

[1228] "Emotional fluctuation patterns" are trends or patterns in data that show how a user's emotions change over time and in different situations.

[1229] "Communication timing" refers to the optimal time and situation for a user to communicate with other users.

[1230] "Forecasting" is the process of predicting future events or outcomes based on past data and patterns.

[1231] "Visualization" is the process of visually displaying data and information and providing it in a form that is easy for users to understand.

[1232] "Feedback" refers to the evaluation and advice the system provides to the user.

[1233] The present invention is a system for determining a user's emotions and providing optimal communication timing. Specific embodiments of this system will be described below.

[1234] Hardware and software used

[1235] Hardware: Servers (with high-performance data processing capabilities), user devices (smartphones, tablets, PCs, etc.)

[1236] Software: Emotion engine, generative AI model, application (user interface)

[1237] Program processing

[1238] Server Processing

[1239] The server collects users' past communication data, including message content, sending time, receiving time, reply time, etc. The server stores this data in a database for later processing.

[1240] Next, the server uses the collected communication data to determine emotions in the emotion engine. The emotion engine uses natural language processing (NLP) technology to extract emotions from the content of the message. Emotions include joy, sadness, anger, surprise, etc.

[1241] The server then uses the emotion determination results to train a generative AI model. It learns past patterns of emotion fluctuations and predicts the optimal timing for communication. This prediction is made using machine learning algorithms (e.g., LSTM, Random Forest).

[1242] Finally, the server sends the prediction results to the application, which then suggests optimal communication timings to the user via notifications or dashboard displays.

[1243] Terminal handling

[1244] The device receives the emotion determination results and the prediction results for the optimal communication timing sent from the server. This information is displayed to the user on the application. The user can refer to this information to select the best timing for communication.

[1245] User Action

[1246] The user can check the emotion assessment results and the optimal communication timing predictions provided by the server through the application, which allows the user to communicate at the optimal timing.

[1247] Specific examples

[1248] The server collects communication data from user A over the past month. The server uses an emotion engine to extract patterns of emotional fluctuations from this data and trains it using a generative AI model. Based on the learning results, the server predicts the optimal timing for communication for user A and presents the results to user A via the application. User A uses this information to decide when to send a message to friend B.

[1249] Prompt Sentence Examples

[1250] "Based on communication data from the past month, please learn User A's emotional fluctuation patterns and predict the optimal timing for communication."

[1251] In this way, by clarifying the roles of the server, terminal, and user, and explaining how to use specific hardware and software, it becomes easier to understand the overall picture of the system.

[1252] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1253] Step 1: Data collection

[1254] Subject: Server

[1255] The server collects users' past communication data, including message content, sending time, receiving time, reply time, etc. The server stores this data in a database.

[1256] Input: User's message data

[1257] Output: Communication data stored in a database

[1258] What it does: The server retrieves data from the user's messaging application through an API, such as the content and timestamps of messages the user has sent and received in the past month.

[1259] Step 2: Emotion determination

[1260] Subject: Server

[1261] The server uses the collected communication data to determine emotions using an emotion engine. The emotion engine uses natural language processing (NLP) technology to extract emotions from the content of messages. Emotions include joy, sadness, anger, and surprise.

[1262] Input: Communication data stored in a database

[1263] Output: Emotion determination result

[1264] What it does: The server uses an NLP library (e.g., spaCy or NLTK) to analyze the text of the message and determine its sentiment. For example, it extracts the sentiment "joy" from the message "I'm so happy today!"

[1265] Step 3: Learning and Prediction

[1266] Subject: Server

[1267] The server uses the emotion determination results to train a generative AI model. It learns past patterns of emotion fluctuations and predicts the optimal timing for communication. This prediction is made using machine learning algorithms (e.g., LSTM, Random Forest).

[1268] Input: Emotion judgment result

[1269] Output: Prediction of optimal communication timing

[1270] How it works: The server inputs the emotion determination results as features into a generative AI model and trains the model. For example, it learns a pattern from past data that "users' emotions are stable on Monday afternoons."

[1271] Step 4: Presenting the results

[1272] Subject: Server

[1273] The server sends the prediction results to the application, which then suggests optimal communication timings to the user. This can be done via notifications or dashboard displays.

[1274] Input: Prediction results of optimal communication timing

[1275] Output: The prediction results displayed in the application

[1276] What it does: The server sends the prediction results in JSON format to the application. The application receives this data and displays it in a user interface, for example, by notifying the user that "next Monday at 3:00 PM is the best time."

[1277] Step 5: User Actions

[1278] Subject: User

[1279] The user checks the optimal communication timings suggested by the application and uses this information to choose when to send a message.

[1280] Input: The prediction result displayed in the application

[1281] Output: User's communication actions

[1282] Specific behavior: The user checks the application's notification and sends a message to a friend. For example, if the application notifies the user that "next Monday at 3:00 PM is the best time," the user sends a message to a friend at that time.

[1283] (Application example 2)

[1284] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1285] Conventional communication tools have had the problem of being unable to accurately assess a user's emotions and communicate at the optimal timing. In particular, when serving customers in physical stores, it is necessary to analyze customer emotions and behavioral patterns in real time and communicate at the optimal timing, but there has been no effective means to achieve this.

[1286] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1287] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning patterns of emotional fluctuations using past communication data, means for predicting the optimal timing for communication based on new data, and means for notifying the user of the predicted timing. This makes it possible to analyze customer emotions and behavioral patterns in real time when serving customers in physical stores, and to serve customers at the optimal timing.

[1288] "Communication Tools" means software or hardware that a User uses to exchange information with other Users.

[1289] "Generative AI" refers to artificial intelligence that generates new data based on input data.

[1290] "Emotion determination" refers to the process of identifying a user's emotional state from their input data.

[1291] "Visualization" is the process of displaying data or information visually to make it easier for users to understand.

[1292] "Past communication data" refers to recorded data relating to previous communications.

[1293] "Emotional fluctuation patterns" refer to the tendency or characteristics of emotions that change over time.

[1294] The "optimal communication timing" is the time when users are predicted to be able to communicate most effectively.

[1295] A "notification" is a message or alert that informs the user of specific information.

[1296] A system for carrying out this invention is configured as follows: The server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning patterns of emotional fluctuations using past communication data, means for predicting the optimal timing for communication based on new data, and means for notifying the user of the predicted timing.

[1297] Specifically, the server uses Python and the scikit-learn library to analyze past communication data and learn patterns of emotional fluctuations. This allows it to predict the optimal timing for communication based on new data entered by the user. The server then notifies the smart glasses of the predicted timing, helping store clerks to communicate with customers at the optimal time.

[1298] For example, in a brick-and-mortar store, if a salesperson is wearing smart glasses, they can analyze the customer's behavioral patterns and facial expressions in real time when they pick up a product. The server compares this with past data to predict when the customer will be most likely to accept service and notify the salesperson. This can improve customer satisfaction.

[1299] An example of a prompt sentence is, "Please predict the best time to speak to the customer after they have picked up the product." By inputting this prompt sentence into the generative AI model, it is possible to predict the best timing for communication and notify the store clerk.

[1300] This system will make it possible to improve the quality of customer service in physical stores and increase customer satisfaction.

[1301] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1302] Step 1:

[1303] The server receives text entered into the communication tool. The input data is a text message entered by the user. This data is converted into an appropriate format for linking with the generative AI. Specifically, the text data is tokenized and converted into a format that can be processed by the generative AI.

[1304] Step 2:

[1305] The server uses generative AI to determine the user's emotion from the input text. The input data is tokenized text, and the output data is emotion labels (e.g., joy, sadness, anger, etc.). The generative AI uses a pre-trained model to extract emotions from the text data.

[1306] Step 3:

[1307] The server visualizes the determined emotions in an application. The input data is emotion labels, and the output data is a graphical representation of the emotion displayed on the user interface. Specifically, the emotional state is visually shown to the user using icons and colors corresponding to the emotion labels.

[1308] Step 4:

[1309] The server uses past communication data to learn patterns of emotional fluctuations. The input data are past text messages and their corresponding emotion labels, and the output data is a pattern model of emotional fluctuations. Specifically, a machine learning algorithm is used to analyze trends in emotions that change over time and build the model.

[1310] Step 5:

[1311] The server predicts the optimal timing for communication based on new data. The input data are text messages and emotion labels captured in real time, and the output data is the optimal timing prediction. Specifically, the server uses a trained emotion fluctuation pattern model to calculate the optimal timing based on the current emotional state.

[1312] Step 6:

[1313] The server notifies the user of the predicted optimal communication timing. The input data is the prediction result, and the output data is a notification message to be displayed on the user's smart glasses or smartphone. Specifically, the server generates the notification message and sends it to the user's device.

[1314] Step 7:

[1315] The user receives the notification and communicates with the customer at the optimal time. The input data is the notification message from the server, and the output data is the user's actions (e.g., when to speak to the customer). Specifically, the user serves the customer at the appropriate time according to the notification message.

[1316] Example 3

[1317] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1318] With conventional communication tools, it was difficult to accurately assess a user's emotions and provide effective feedback based on those results. Furthermore, there was a lack of concrete methods for utilizing past communication data to improve users' communication skills. This made it difficult for users to understand the relationship between their own emotions and communication and improve their skills.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1320] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an application, means for analyzing past communication data and the determination results of emotions at that time, means for generating feedback for the user based on the analysis results, and means for providing the generated feedback to the user. This enables users to understand the relationship between their own emotions and communication and improve their skills.

[1321] "Communication Tools" means software or applications that allow users to exchange messages or information with other users.

[1322] "Generative AI" is an artificial intelligence technology that generates new information and results based on input data.

[1323] "Emotion determination" is the process of analyzing a user's emotions from input text or data and assigning a specific emotion label.

[1324] An "application" is a software program designed to provide a particular function or service.

[1325] A "database" is a system for efficiently storing, managing, and retrieving data.

[1326] "Feedback" refers to evaluations and advice provided to users.

[1327] "Analysis" is the process of examining data in detail to find meaning and patterns.

[1328] "User" means any individual or entity that uses the system or application.

[1329] The present invention is a system for determining a user's emotion and providing feedback based on the result. A specific embodiment of this system will be described below.

[1330] Hardware and software used

[1331] Hardware: Servers, devices (PCs, smartphones, etc.)

[1332] Software: Emotion engine, generative AI model, database management system (DBMS)

[1333] System Overview

[1334] This system collects communication data when a user communicates using a device and uses an emotion engine to determine the user's emotions. The results are stored in a database, and a generative AI model is used to analyze past communication data and the emotion determination results. The system extracts the situations in which communication was most effective and generates and provides feedback to the user based on the results.

[1335] Specific examples

[1336] If the emotion engine determines that User A felt "joy" based on past communication data, then that communication was the most effective. The generative AI model analyzes this information and provides feedback to User A, such as, "The communication you received when you felt joy was the most effective. Try to keep this emotion in mind next time."

[1337] Prompt Sentence Examples

[1338] "Analyze the emotions the user has expressed in the past and the effectiveness of communication at that time to extract the most effective situations. Then, generate feedback for the user based on the results."

[1339] This system enables users to understand the relationship between their own emotions and communication and improve their skills. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1340] Step 1:

[1341] Users communicate using terminals.

[1342] Typed: Messages typed by users into communication tools such as chat apps or emails.

[1343] Specific action: For example, a user sends a message saying, "I'm so happy today!"

[1344] Output: Message data sent from the device to the server.

[1345] Step 2:

[1346] The server inputs the received message data into an emotion engine to determine the user's emotion.

[1347] Input: Message data sent from the terminal.

[1348] Specific behavior: The server inputs the message "I'm so happy today!" into the emotion engine.

[1349] Data processing: The emotion engine uses natural language processing technology to analyze the sentiment of the message.

[1350] Output: An emotion label, such as "joy."

[1351] Step 3:

[1352] The server stores the emotion determination results in a database.

[1353] Input: The emotion labels output by the emotion engine and the original message data.

[1354] Specific behavior: The server stores the message "I'm so happy today!" and the emotion label "joy" in a database.

[1355] Output: Message data and emotion labels stored in a database.

[1356] Step 4:

[1357] The server inputs past communication data and the emotional assessment results at that time into the generative AI model for analysis.

[1358] Input: Historical communication data and emotion labels obtained from the database.

[1359] Specific operation: The server inputs past messages labeled with the emotion "joy" into the generative AI model.

[1360] Data computation: Generative AI models analyze this data and extract the most effective communication situations.

[1361] Output: Patterns of effective communication.

[1362] Step 5:

[1363] The server generates feedback for the user based on the analysis results.

[1364] Input: Effective communication patterns output from a generative AI model.

[1365] Specific action: The server generates a feedback message saying, "The communication was most effective when you felt joy. Please try to keep this emotion in mind next time."

[1366] Output: The feedback message.

[1367] Step 6:

[1368] The server generates feedback and sends it to the terminal for display to the user.

[1369] Input: The server-generated feedback message.

[1370] Specific operation: The server sends a feedback message to the user's device.

[1371] Output: Feedback messages displayed on the terminal.

[1372] (Application example 3)

[1373] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1374] Conventional communication tools have difficulty accurately assessing users' emotions and providing optimal feedback based on those emotions. Furthermore, because they cannot assess emotions and provide feedback in real time, they are unable to contribute to improving users' communication skills. Furthermore, there is a lack of a way to provide effective feedback using past communication data.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1376] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for analyzing past communication data and the emotion determination results, means for providing feedback to the user based on the analysis results, and means for determining emotions in real time and displaying optimal feedback. This makes it possible to determine the user's emotions in real time and provide optimal feedback by utilizing past data.

[1377] "Communication Tools" means software or hardware that a User uses to exchange information with others.

[1378] "Generative AI" refers to artificial intelligence that generates new data based on input data.

[1379] "Emotion determination" refers to the process of identifying the user's current emotion from their facial expressions, writing, etc.

[1380] "Visualization" is the process of displaying data or information visually to make it easier for users to understand.

[1381] "Past Communication Data" means records of communications previously conducted by a User.

[1382] "Analysis" is the process of examining data in detail to find meaning and patterns.

[1383] "Feedback" refers to providing reactions and advice to the user's actions or state.

[1384] "Real-time" refers to data processing and information delivery occurring immediately, without delay.

[1385] A "smart device" is an electronic device that has internet connectivity and advanced computing capabilities.

[1386] The following system configuration will be described as an embodiment of the present invention.

[1387] System Configuration

[1388] The system includes a communication tool, generative AI, an emotion determination engine, a data analysis module, a feedback generation module, and a smart device (smart glasses).

[1389] Program processing

[1390] The server connects the text entered into the communication tool to the generative AI. The generative AI analyzes the entered text and determines the emotion of the person who wrote it. The emotion determination engine analyzes past communication data and the emotion determination results based on the determined emotion. The data analysis module extracts the situations in which the most effective communication took place and sends the results to the feedback generation module. The feedback generation module provides feedback to the user based on the analysis results.

[1391] The smart device (smart glasses) detects emotions in real time and displays appropriate feedback, allowing users to learn optimal responses in real time and improve their communication skills.

[1392] Hardware and software used

[1393] Hardware: Smart glasses (with camera and display)

[1394] Software: OpenCV (image processing), emotion detection engine, feedback generation module

[1395] Specific examples

[1396] For example, if a customer looks dissatisfied while a waiter is wearing smart glasses, the emotion recognition model will determine that emotion. Based on past data, feedback will be generated that "when a customer looks dissatisfied, a quick response and apology is effective," and the smart glasses will display the message "Please respond quickly and apologize."

[1397] Prompt Sentence Examples

[1398] Examples of prompts to input to a generative AI model include:

[1399] If a customer looks unhappy, learn from past data how to best respond and provide real-time feedback.

[1400] In this way, customer service staff can learn optimal responses in real time, improving customer satisfaction.

[1401] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1402] Step 1:

[1403] The server receives the text entered into the communication tool and connects to the generative AI. The input is the text entered by the user into the communication tool, and the output is the text to be analyzed that is passed to the generative AI. Specifically, the server sends the text data to the generative AI via an API.

[1404] Step 2:

[1405] The generative AI analyzes the received text and determines the emotion of the person who wrote it. The input is the text received in step 1, and the output is the emotion determination result. Specifically, the generative AI uses natural language processing technology to analyze the emotion of the text and assigns an emotion label such as positive, negative, or neutral.

[1406] Step 3:

[1407] The server analyzes past communication data and emotion determination results based on the emotions determined using the emotion determination engine. The input is the emotion determination results and past communication data, and the output is the analysis results. Specifically, the server retrieves past communication data from the database, compares it with the emotion labels obtained by the emotion determination engine, and extracts the most effective communication pattern.

[1408] Step 4:

[1409] The data analysis module extracts the situations in which the most effective communication took place and sends the results to the feedback generation module. The input is the analysis results obtained in step 3, and the output is the feedback data sent to the feedback generation module. Specifically, the data analysis module generates feedback that is optimal for the current situation based on past success stories.

[1410] Step 5:

[1411] The feedback generation module provides feedback to the user based on the analysis results. The input is the feedback data obtained in step 4, and the output is a feedback message to be provided to the user. Specifically, the feedback generation module generates a text message or a voice message and presents it to the user.

[1412] Step 6:

[1413] The smart device (smart glasses) determines emotions in real time and displays appropriate feedback. The input is a feedback message from the feedback generation module, and the output is feedback information displayed on the smart glasses display. Specifically, a message such as "Please respond promptly and apologize" is displayed on the smart glasses display.

[1414] In this way, users can learn optimal responses in real time and improve their communication skills.

[1415] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1417] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1418] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1419] [Third embodiment]

[1420] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1421] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1422] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1423] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1424] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1425] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1426] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1427] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1430] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1431] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1432] "Example 1"

[1433] One embodiment of the present invention is to use Google Chat as a communication tool. Text entered into Google Chat is linked to a generative AI, which determines human emotions from the text. The results are visualized in the app, and the user can use these results to select the timing of communication. Specifically, if the other person is showing positive emotions such as joy or relief, it is recommended that they communicate at that time.

[1434] "Example 2"

[1435] The generative AI also determines the optimal timing for communication based on the emotion assessment results. Specifically, the AI ​​learns past communication data and patterns of emotional fluctuations, and predicts the optimal timing for communication based on that. This prediction result is also presented to the user on the app, allowing the user to use it as a reference when selecting the timing for communication.

[1436] "Example 3"

[1437] Furthermore, the generative AI provides feedback to the user based on the emotion assessment results. Specifically, the AI ​​analyzes which emotions the user has displayed in the past that have resulted in the most effective communication, and provides feedback based on the results. This feedback allows the user to improve their own communication skills.

[1438] The processing flow of each embodiment will be described below.

[1439] "Example 1"

[1440] Step 1: User types a message into Google Chat.

[1441] Step 2: The input text is linked to the generative AI.

[1442] Step 3: The generative AI determines human emotions from the text.

[1443] Step 4: The results are visualized in the app.

[1444] Step 5: The user selects the timing of communication based on the visualized results.

[1445] "Example 2"

[1446] Step 1: The generative AI learns past communication data and patterns of emotional fluctuations.

[1447] Step 2: The AI ​​predicts the optimal timing for communication based on the learned data.

[1448] Step 3: The prediction results are presented to the user on the app.

[1449] Step 4: The user selects the timing of communication based on the presented prediction results.

[1450] "Example 3"

[1451] Step 1: Generative AI analyzes which emotions the user has displayed in the past that resulted in the most effective communication.

[1452] Step 2: The AI ​​provides feedback to the user based on the analysis results.

[1453] Step 3: Users use the provided feedback to improve their communication skills.

[1454] Example 1

[1455] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1456] With conventional communication tools, it was difficult to accurately grasp the emotions of the other person, making it difficult to communicate at the right time. In addition, there was a lack of means to provide users with the results of emotion assessment, and users were not given enough support to communicate effectively.

[1457] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1458] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for displaying the determination result on the user's device, and means for providing information for the user to select the timing of communication. This enables the user to accurately grasp the emotion of the other person and communicate effectively at the appropriate time.

[1459] A "communication tool" is software or an application that allows users to send and receive messages.

[1460] "Generative AI" is an artificial intelligence model that uses natural language processing technology to analyze input text and perform specific tasks (such as determining emotions).

[1461] "Emotion determination" is the process of identifying the emotions (e.g., joy, relief, anger, sadness, etc.) of the person who wrote the input text.

[1462] An "application" is a software program available to users that provides specific functions or services.

[1463] "User device" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[1464] The "judgment result" is the emotion identification result obtained by the generative AI analyzing the input text.

[1465] "Visualization" is the process of visually displaying data or information, providing it in a way that is easy for users to understand.

[1466] "Information for selecting timing" refers to information such as emotion determination results that are provided to the user to determine the optimal timing for communicating with the other person.

[1467] This invention is a system that links text entered into a communication tool with generative AI, determines emotions, and provides the results to the user. Specifically, the server, terminal, and user elements work together.

[1468] The server receives the text entered by the user into the communication tool. For example, if the user sends the message "I had so much fun today!", this message arrives at the server. The server then sends the received message to a generative AI model (e.g., OpenAI's GPT-4). This generative AI model uses natural language processing technology to analyze the message and determine its sentiment.

[1469] The generative AI model determines the emotion from the message sent. For example, it determines the emotion "joy" from the message "I had so much fun today!" This process uses a machine learning algorithm. After the generative AI model determines the emotion, it sends the result back to the server. The server then sends this determination result to the user's device.

[1470] The user's device visualizes the emotion determination results received from the server. For example, the user's device might display "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[1471] As a concrete example, consider the case where a user sends a message to Google Chat saying, "I had so much fun today!" This message is sent to the server and passed to a generative AI model. The generative AI model determines the emotion "joy" from this message. The result of this determination is sent to the application, and the message "The other person is feeling happy" is displayed on the user's device. The user can use this information to decide when to communicate with the other person.

[1472] An example of a prompt might be:

[1473] Determine the emotion from the message, "Today was so much fun!"

[1474] By inputting this prompt into the generative AI model, the model determines the emotion "joy," thus providing support for users to communicate effectively.

[1475] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1476] Step 1:

[1477] The server receives the text entered by the user into the communication tool.

[1478] Specifically, when a user sends a message such as "Today was so much fun!", this message is received by the server.

[1479] Input: Text entered by the user into the communication tool

[1480] Output: The text received by the server

[1481] Step 2:

[1482] The server sends the received sentences to the generative AI model.

[1483] Specifically, the server sends the received message to a generative AI model (e.g., GPT-4), which then analyzes the message using natural language processing techniques.

[1484] Input: The text received by the server

[1485] Output: The sentence sent to the generative AI model

[1486] Step 3:

[1487] The generative AI model determines the sentiment from the text sent.

[1488] Specifically, the generative AI model analyzes the message "I had so much fun today!" and determines the emotion "joy." This process uses machine learning algorithms.

[1489] Input: The sentence sent to the generative AI model

[1490] Output: Determined emotion (e.g., "joy")

[1491] Step 4:

[1492] The generative AI model returns the emotion determination results to the server.

[1493] Specifically, the generative AI model determines the emotion of "joy" and then sends the result back to the server.

[1494] Input: Determined emotion

[1495] Output: Emotion judgment result sent back to the server

[1496] Step 5:

[1497] The server transmits the emotion determination result to the user's terminal.

[1498] Specifically, the server sends the emotion determination results received from the generative AI model to the user's device.

[1499] Input: Emotion judgment result returned to the server

[1500] Output: Emotion judgment result sent to the user's device

[1501] Step 6:

[1502] The user's device visualizes the emotion determination results.

[1503] Specifically, the user's device will display a message saying, "The other person is feeling happy." Based on this information, the user can decide when to communicate with the other person.

[1504] Input: Emotion judgment result sent to the user's device

[1505] Output: Visualized emotion judgment result (e.g., "The other person is feeling happy")

[1506] In this way, the system provides support for users to communicate effectively.

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1509] In physical stores, in order to communicate smoothly with customers, it is important to properly understand the customer's emotions and provide customer service at the optimal timing. However, with conventional methods, it is difficult for store staff to accurately read the customer's emotions, which can result in a decline in the quality of customer service. To solve this problem, a system is needed that analyzes customer emotions in real time and provides appropriate feedback.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes a means for linking text entered into the communication tool to a generative AI, a means for using the generative AI to determine the emotions of the person who wrote the text, and a means for visualizing the determination results in an application. This makes it possible to input the content of conversations with customers in a physical store and analyze their emotions in real time. By including a means for recommending the timing of customer service based on the analysis results, store staff can properly understand customer emotions and provide customer service at the optimal time.

[1512] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[1513] "Generative AI" is an artificial intelligence technology that analyzes input data and generates appropriate output for a specific task.

[1514] "Means of collaboration" are methods and technologies for exchanging data between different systems and software, making it mutually usable.

[1515] A "means for determining" is a method or technology for analyzing input data and evaluating the results based on specific criteria.

[1516] "Visualization means" refers to methods and techniques for displaying analysis results and data in a way that is easy for users to understand.

[1517] A "brick and mortar store" is a store that exists in a physical location and where customers can visit in person to purchase goods or services.

[1518] "Customers" are people who visit a physical store to purchase goods or services.

[1519] "Conversation content" refers to the words and sentences exchanged between the customer and the store clerk.

[1520] "Means for analyzing emotions in real time" refers to methods or technologies for instantly analyzing input conversation content and determining customer emotions.

[1521] A "means for recommending the timing of customer service" is a method or technology for instructing store staff to respond to customers at the optimal timing based on the analysis results.

[1522] In order to implement the present invention, the following system configuration and program are required.

[1523] System Configuration

[1524] 1. Hardware:

[1525] Smartphone: Used by store clerks to input conversations with customers and check the analysis results.

[1526] Server: Provides the computational resources to run generative AI models and perform sentiment analysis.

[1527] 2. Software:

[1528] Communication tool: An application that allows store clerks to enter conversations with customers.

[1529] Generative AI model: An artificial intelligence technology that analyzes input text and determines emotions.

[1530] Google Cloud Natural Language API: An API for analyzing the sentiment score of a sentence.

[1531] OpenAI API: An API for running generative AI models and determining emotions.

[1532] Program processing

[1533] The server processes the data in the following steps:

[1534] 1. Data Entry:

[1535] The store clerk enters the details of the conversation with the customer into the communication tool on their smartphone.

[1536] 2. Emotion analysis:

[1537] The server sends the input conversation content to the Google Cloud Natural Language API and obtains an emotion score.

[1538] In addition, it uses the OpenAI API to perform detailed sentiment determination of the input text.

[1539] 3. Visualizing the results:

[1540] The server sends the analysis results to a smartphone application and displays the visualized results to the store clerk.

[1541] Based on the analysis, store staff are encouraged to continue serving customers if they are displaying positive emotions.

[1542] Specific examples

[1543] For example, consider the case where a store clerk inputs the following conversation content:

[1544] Type: "It's a beautiful day today. Can I see your new items?"

[1545] The server sends this input to the Google Cloud Natural Language API to obtain an emotion score, then uses the OpenAI API to generate the following prompt sentence and perform emotion determination:

[1546] Example prompt sentence:

[1547] Determine the sentiment of the following sentences: The weather is very nice today. Can you show me your new products?

[1548] If the analysis results in a positive emotion, the smartphone application will display the message, "The customer is expressing a positive emotion. Please continue serving the customer."

[1549] In this way, store staff can properly understand the customer's emotions and provide service at the optimal time.

[1550] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1551] Step 1:

[1552] The user (store clerk) inputs the contents of the conversation with the customer into the communication tool on the smartphone. The inputted contents of the conversation are saved on the smartphone as text data.

[1553] Step 2:

[1554] The device (smartphone) sends the entered conversation content to the server. The data sent is the conversation content in text format.

[1555] Step 3:

[1556] The server sends the received conversation content to the Google Cloud Natural Language API and obtains the sentiment score. The input data is the conversation content in text format, and the output data is the sentiment score (positive, negative, neutral, etc.).

[1557] Step 4:

[1558] The server uses the OpenAI API to perform detailed emotion determination based on the emotion score. The server generates a prompt from the conversation content and sends it to the OpenAI API. The input data is the prompt, and the output data is the detailed emotion determination result.

[1559] Step 5:

[1560] The server integrates the emotion analysis results obtained from the Google Cloud Natural Language API and the OpenAI API to generate a final emotion judgment result. The input data is the emotion score and detailed emotion judgment result, and the output data is the integrated emotion judgment result.

[1561] Step 6:

[1562] The server then transmits the integrated emotion determination results to the smartphone. The transmitted data is a visualized version of the emotion determination results.

[1563] Step 7:

[1564] The device (smartphone) receives the emotion assessment results and displays them in the application. The displayed data is information about the customer's emotional state and the recommended timing for serving the customer.

[1565] Step 8:

[1566] The user (store clerk) refers to the displayed emotion determination result and adjusts the timing of serving the customer. Specifically, if the customer shows a positive emotion, the store clerk continues serving the customer, and if the customer shows a negative emotion, the store clerk reconsiders the timing of serving the customer.

[1567] Example 2

[1568] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1569] Conventional communication tools have difficulty accurately assessing users' emotions and providing optimal communication timing based on those emotions. Furthermore, there are also insufficient ways for users to utilize past communication data to obtain feedback for effective communication. This has resulted in users being unable to communicate effectively, leading to a decline in the quality of communication.

[1570] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for linking a sentence input to a communication tool to a generative AI, means for determining the emotion of the person who wrote the sentence using the generative AI, means for visualizing the determination result using an application, means for collecting past communication data, means for preprocessing the collected data, means for learning emotion fluctuation patterns, means for predicting the optimal communication timing, and means for presenting the prediction result to the user. This allows the user to know the optimal communication timing based on the emotion fluctuation pattern, enabling effective communication.

[1571] "Communication Tools" refers to software or hardware that allows users to exchange information with other users through messages, voice, video, etc.

[1572] "Generative AI" refers to artificial intelligence that uses natural language processing and machine learning techniques to generate and analyze data such as text and voice.

[1573] "Emotion determination" refers to the process of analyzing a user's emotions from text or voice data entered by the user and classifying them into specific emotional categories.

[1574] "Visualization" refers to displaying analysis results and data in a way that is easy for users to understand, such as in graphs, charts, or text format.

[1575] "Past communication data" refers to records of messages, voice and video calls, etc. that a user has previously made.

[1576] "Preprocessing" refers to a series of operations performed on collected data to transform it into an analyzable format, including denoising, tokenization, and normalization.

[1577] "Emotional patterns" refer to trends and patterns in data that show how users' emotions change over time and across situations.

[1578] "Optimal communication timing" refers to the time period or timing when a user is predicted to be able to communicate most effectively with other users.

[1579] "Prediction results" refer to the conclusions and estimates that generative AI reaches through analysis and learning.

[1580] MODE FOR CARRYING OUT THE INVENTION

[1581] This invention provides a system for enabling users to communicate effectively with other users. The system uses a generative AI model to determine the user's emotions and predict the optimal timing for communication. A specific embodiment of this system is described below.

[1582] 1. Program Generation

[1583] The server uses a generative AI model to generate a program that determines emotions and optimizes communication timing. This program is built using machine learning libraries such as Python and TensorFlow.

[1584] 2. Program Processing

[1585] The server first collects the user's past communication data, including text messages, voice data, and facial expression data. For example, it collects the user's message history with friends and facial expression data during video calls.

[1586] The server then preprocesses the collected data. For text data, unnecessary characters and symbols are removed and the data is tokenized. For audio data, noise is removed and the audio is converted into text. For facial expression data, facial recognition technology is used to extract facial expression features.

[1587] The preprocessed data is then input into an emotion detection model, which uses generative AI models such as BERT and GPT-3 to analyze user emotions. For example, it can detect emotions such as "joy," "sadness," and "anger" from text messages.

[1588] Once emotion determination is complete, the server performs time series analysis to learn the emotional fluctuation patterns. This is done using recurrent neural networks (RNNs) such as LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit). After learning is complete, the server predicts the optimal timing for communication.

[1589] The prediction results are presented to the user through the application. The user can use this information to communicate at the optimal time. For example, the app may notify the user, "Now is the best time to message a friend."

[1590] 3. Examples of concrete examples and prompts

[1591] For example, when a user is messaging a friend, the server analyzes past message data to learn the user's emotional fluctuation patterns.The server then predicts when the friend is most likely to respond and notifies the user of that timing.

[1592] Specific prompt examples:

[1593] "Analyze message exchange data with friends and predict the best timing for communication."

[1594] In this way, users can communicate more effectively with the help of generative AI.

[1595] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1596] Step 1: Data collection

[1597] The server collects the user's past communication data. Specifically, it acquires message history, voice data, and facial expression data during video calls from the user's smartphone or PC. The input is data from the user's device, and the output is the collected raw data.

[1598] Step 2: Data Preprocessing

[1599] The server preprocesses the collected data. In the case of text data, unnecessary characters and symbols are removed and the data is tokenized. In the case of audio data, noise is removed and the audio is converted into text. In the case of facial expression data, facial expression features are extracted using facial recognition technology. The input is the collected raw data, and the output is the preprocessed data.

[1600] Step 3: Emotion determination

[1601] The server inputs the preprocessed data into an emotion detection model. The emotion detection model uses generative AI models such as BERT and GPT-3 to analyze the user's emotions. For example, it detects emotions such as "joy," "sadness," and "anger" from a text message. The input is the preprocessed data, and the output is the emotion detection result.

[1602] Step 4: Learn emotional patterns

[1603] The server learns patterns of emotional fluctuations based on the results of emotion determination. For this, it uses recurrent neural networks (RNNs) such as LSTM and GRU. For example, it learns whether a user tends to show the emotion of "happiness" at a certain time of day. The input is the result of emotion determination, and the output is the learned pattern of emotional fluctuations.

[1604] Step 5: Anticipate communication timing

[1605] The server predicts the optimal timing for communication based on the learned emotional fluctuation patterns. For example, it predicts the time of day when a user can communicate most effectively with friends. The input is the learned emotional fluctuation patterns, and the output is the predicted optimal timing for communication.

[1606] Step 6: Presenting the results

[1607] The server presents the prediction results to the user through the application. The user can use this information to communicate at the optimal time. For example, the application may notify the user, "Now is the best time to send a message to a friend." The input is the predicted optimal communication timing, and the output is the notification to the user.

[1608] (Application example 2)

[1609] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1610] With conventional communication tools, it was difficult to accurately assess users' emotions and support communication at the optimal timing. In addition, in physical stores, it was difficult for store staff to grasp customers' emotions in real time and determine the optimal timing to interact with them. This resulted in inconsistent customer service quality, making it difficult to improve customer satisfaction.

[1611] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1612] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an app, means for learning past communication data and patterns of emotional fluctuations, means for predicting the optimal timing for communication, means for presenting the prediction results to a user, means for the user to select the timing of communication based on the presented prediction results, and means for supporting customer service in a physical store. This enables the user to select the optimal timing for communication based on emotions, and enables the emotions of customers in a physical store to be grasped in real time and the optimal timing for customer service to be determined.

[1613] A "communication tool" is software or an application that allows a user to input text or messages and exchange information with other users.

[1614] "Generative AI" is an artificial intelligence technology that generates new data and information based on input data.

[1615] "Emotion determination" is the process of analyzing a user's emotional state from input text or data and identifying a specific emotion.

[1616] "Visualization" refers to displaying data and information visually and providing it in a form that is easy for users to understand.

[1617] "Past communication data" refers to records or histories of previous communications, including message content and timing.

[1618] "Emotional fluctuation patterns" refer to emotional tendencies or patterns that change over time or in response to specific situations.

[1619] "Optimal communication timing" refers to the timing for the most effective communication based on the user's emotional state and past data.

[1620] "Predicted results" refers to future results and trends derived by generative AI through analysis and learning.

[1621] A "brick and mortar store" is a store that exists in a physical location and where customers visit in person to purchase goods or services.

[1622] "Customer service" refers to the service and response provided by store staff to customers.

[1623] As an embodiment of the present invention, a system for supporting customer service in a brick-and-mortar store will be described as an example.

[1624] First, the server provides a means for linking text entered into the communication tool to the generative AI. Specifically, text entered by the user using a device such as a smartphone or tablet is sent to the server via the internet. The server then passes this text to the generative AI, which determines its emotion.

[1625] Next, generative AI analyzes the input text and determines the user's emotions. Natural language processing techniques and machine learning algorithms are used to determine emotions. For example, a sentiment analysis model can be built using the Python libraries scikit-learn and TensorFlow.

[1626] The server visualizes the emotion results in an application. The application displays the emotion results on the screen of a smartphone or tablet, allowing the user to understand them intuitively. For example, the emotion score could be displayed as a graph or icon.

[1627] Furthermore, the server provides a means to learn past communication data and emotional fluctuation patterns, using past communication history stored in a database. Using this data, the server trains a model for the generative AI to predict optimal communication timing.

[1628] To predict the optimal timing for communication, the generative AI receives the current time and emotion score as input and generates a prediction result. This prediction result is sent to the application via the server and presented to the user. The user can refer to this prediction result to communicate at the optimal time.

[1629] In physical stores, store clerks use their smartphones to capture customers' facial expressions. The application calculates an emotion score in real time and suggests the optimal timing for customer interaction. This allows store clerks to understand the emotional state of customers and provide assistance at the optimal time.

[1630] As a concrete example, the following prompt might be input to a generative AI:

[1631] "It is currently 10 o'clock. The customer's sentiment score is 0.7. Please predict the optimal time to serve them."

[1632] In this way, users can choose the optimal timing for communication based on their emotions, and even in physical stores, they can grasp the emotions of customers in real time and determine the optimal timing for customer service.

[1633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1634] Step 1:

[1635] A user inputs text into a communication tool using a device such as a smartphone or tablet.

[1636] Input: Text entered by the user

[1637] Output: The input text is sent from the terminal to the server.

[1638] Specific actions: The user opens an application on the device, enters text in the text box, and presses the send button.

[1639] Step 2:

[1640] The server receives the text sent from the device and connects it to the generative AI.

[1641] Input: Text sent from the terminal

[1642] Output: Text passed to the generative AI

[1643] Specific operation: The server receives an HTTP request, analyzes the text data, and passes it to the generative AI API.

[1644] Step 3:

[1645] The generative AI analyzes the received text and determines the user's emotions.

[1646] Input: Text given to the generative AI

[1647] Output: Sentiment score (e.g. 0.7)

[1648] How it works: Generative AI uses natural language processing techniques to analyze text and calculates a sentiment score using a sentiment analysis model (e.g., scikit-learn or TensorFlow).

[1649] Step 4:

[1650] The server visualizes the emotion scores received from the generative AI in the application.

[1651] Input: Sentiment score

[1652] Output: Visualized sentiment scores (e.g., graphs and icons)

[1653] Specific operation: The server receives the emotion score, generates data to reflect in the application UI, and sends it to the device.

[1654] Step 5:

[1655] The server learns past communication data and patterns of emotional fluctuations.

[1656] Input: Historical communication data

[1657] Output: The trained model

[1658] Specific operation: The server retrieves past communication history from the database and passes it to the generative AI to train the model.

[1659] Step 6:

[1660] The generative AI takes the current time and emotion score as input and predicts the optimal timing for communication.

[1661] Input: Current time, Sentiment score

[1662] Output: Prediction of optimal communication timing

[1663] How it works: The generative AI uses a trained model to predict the optimal timing using the current time and emotion score as input.

[1664] Step 7:

[1665] The server presents the prediction results to the user.

[1666] Input: Prediction results of optimal communication timing

[1667] Output: Predictions presented to the user

[1668] Specific operation: The server receives the prediction results, generates data to be reflected in the application UI, and sends it to the device.

[1669] Step 8:

[1670] The user refers to the presented prediction results and communicates at the optimal time.

[1671] Input: Predicted results

[1672] Output: User-selected timing of communication

[1673] Specific behavior: The user checks the results presented by the application and sends a message at the optimal time.

[1674] Step 9:

[1675] When providing customer service support in a physical store, store staff carry smartphones and use the camera to capture the facial expressions of customers.

[1676] Input: Customer facial expression data

[1677] Output: Real-time emotion score

[1678] How it works: A store associate uses a smartphone camera to capture a customer's facial expression, and the application determines an emotion score in real time.

[1679] Step 10:

[1680] The application suggests the best time to serve customers.

[1681] Input: Real-time emotion score

[1682] Output: Recommendation of optimal timing for customer service

[1683] Specific operation: The application calculates the optimal timing to serve the customer based on the emotion score and notifies the store clerk.

[1684] Example 3

[1685] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1686] Conventional communication tools have difficulty accurately assessing users' emotions and providing feedback based on them. Furthermore, there have been no systems that analyze past emotional data and the effectiveness of communication to provide effective advice to users. As a result, users have been unable to obtain specific guidance on how to improve their communication skills.

[1687] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1688] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotions of the person who wrote the text using the generative AI, means for visualizing the determination results in an application, means for analyzing past emotion data and the effectiveness of communication, and means for providing feedback to the user based on the analysis results. This allows the user to receive effective feedback based on their own emotions and obtain specific guidelines for improving their communication skills.

[1689] "Communication tools" is a general term for software or hardware that allows users to exchange information with others through text, voice, etc.

[1690] "Generative AI" is a general term for artificial intelligence technology that generates new information based on input data.

[1691] "Means for determining emotions" is a general term for algorithms and technologies that analyze a user's emotions from input text or data and identify specific emotions.

[1692] "Application" is a general term for a software program designed to provide a specific function or service.

[1693] "Visualization means" is a general term for techniques and methods for visually displaying data and information so that users can easily understand it.

[1694] "Past emotion data" is a general term for records and data relating to emotions expressed by a user in the past.

[1695] "Communication effectiveness" is a general term for the impact or results that a particular communication method has on users.

[1696] "Analytical methods" is a general term for techniques and methods used to analyze data and find specific patterns and trends.

[1697] "Means for providing feedback" is a general term for techniques and methods for providing advice and information to users based on analysis results.

[1698] This invention is a system that links text entered by a user through a communication tool to a generative AI, determines emotions, and provides feedback based on the results. A specific embodiment of this system is described below.

[1699] Hardware and software used

[1700] server

[1701] The server provides the computational resources to run the generative AI model. Specifically, it uses the following software and hardware:

[1702] Generative AI models: Natural language processing models such as OpenAI's GPT-4

[1703] Database Management System: Relational database such as MySQL

[1704] Machine learning algorithms: Machine learning libraries such as scikit-learn

[1705] Terminal

[1706] A terminal is a device for users to provide input data and receive feedback. Specifically, it uses the following software and hardware:

[1707] Browser: A web browser such as Google Chrome

[1708] Communication protocol: HTTP / HTTPS

[1709] User

[1710] A user uses a communication tool to input text data and receive feedback.

[1711] Program processing

[1712] server

[1713] The server receives text data sent from the device and uses the generative AI model to determine the emotion. For example, if a user enters "Recently, communication at work has been going poorly," the server analyzes the text data and determines the emotion as "anxiety."

[1714] The server then uses a database management system to analyze the user's past emotional data and the effectiveness of communication. For example, it might determine that presenting specific solutions to problems was effective when the user expressed anxiety.

[1715] The server then uses the generative AI model to generate feedback for the user, such as "By providing specific solutions to the problem, you may be able to improve communication in the workplace."

[1716] Terminal

[1717] The device displays the feedback received from the server to the user, who can then check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[1718] Examples of specific examples and prompts

[1719] As a concrete example, consider the case where a user inputs, "Recently, communication at work has not been going well." The server receives this input and uses a generative AI model to determine the user's emotion as "anxiety." It then analyzes past data to identify the most effective communication method when the user expressed "anxiety." For example, suppose "presenting specific solutions to the problem" was effective. Based on this information, the server provides the user with feedback such as, "Providing specific solutions to the problem may improve communication at work."

[1720] Example prompt sentence:

[1721] User: Communication at work has been difficult lately.

[1722] Server: Determines user sentiment and analyzes the most effective communication method based on past data.

[1723] Server: When users expressed anxiety, the most effective thing we did was to offer concrete solutions to their problems.

[1724] Server: Providing concrete solutions to problems can improve communication in the workplace.

[1725] In this way, a system is realized in which the server, the terminal, and the user work together to provide effective feedback to the user. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1726] Step 1:

[1727] The user provides the input data.

[1728] A user inputs text data through the interface of a communication tool. For example, the user inputs, "Recently, communication at work has not been going well." The input data is in text format and expresses the user's feelings and situation.

[1729] Step 2:

[1730] The terminal sends the input data to the server.

[1731] The terminal converts the text data entered by the user into JSON format and sends it to the server as an HTTP POST request. The input data is a text document, and the output is the JSON format data sent to the server.

[1732] Step 3:

[1733] The server uses a generative AI model to determine emotions.

[1734] The server inputs the received text data into a generative AI model (e.g., GPT-4) to determine the user's emotion. The input data is text data in JSON format, and the output is the emotion determination result (e.g., "anxiety"). The server invokes the generative AI model, analyzes the text data, and identifies the emotion.

[1735] Step 4:

[1736] The server analyzes past data and identifies effective communication methods.

[1737] The server uses a database management system (e.g., MySQL) to analyze the user's past emotional data and the effectiveness of communication. The input data is the emotional judgment result and past emotional data, and the output is an effective communication method (e.g., "present a specific solution to the problem"). The server analyzes the data using a machine learning algorithm (e.g., scikit-learn) and identifies the optimal communication method.

[1738] Step 5:

[1739] The server generates the feedback.

[1740] The server generates feedback using a generative AI model based on the identified effective communication methods. The input data are effective communication methods, and the output is feedback to the user (e.g., "Providing specific solutions to problems may improve communication in the workplace"). The server then invokes the generative AI model again to generate feedback in natural language.

[1741] Step 6:

[1742] The device displays the feedback to the user.

[1743] The device displays the feedback received from the server to the user. The input data is the feedback text, and the output is the feedback displayed on the user's screen. The user can check the feedback displayed on their smartphone or PC screen and use it in their next communication.

[1744] (Application example 3)

[1745] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1746] Conventional communication tools have difficulty accurately determining a user's emotions and providing appropriate feedback based on them. Security services are required to determine a user's emotions in real time and provide appropriate responses, but there has been a lack of effective means to achieve this. Furthermore, there has been no system that utilizes past emotion data to suggest the most effective communication method.

[1747] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence using the generative AI, a means for visualizing the determination result in an app, a means for analyzing past emotion data and proposing the most effective communication method, and a means for determining emotion in real time and providing feedback according to the security situation. This makes it possible to accurately determine a user's emotion and provide appropriate feedback. Furthermore, in security services, emotions can be determined in real time and appropriate responses can be taken. Furthermore, the most effective communication method can be proposed by utilizing past emotion data.

[1748] A "communication tool" is software or hardware that allows a user to exchange information with others through text or voice.

[1749] "Generative AI" is an artificial intelligence technology that generates new data based on input data, and is particularly used in natural language processing and image generation.

[1750] "Emotion determination" is the process of identifying a user's emotional state from the text or voice input they provide.

[1751] "Visualization" is a method of visually displaying data and information to make it easier for users to understand.

[1752] "Past emotion data" is a record of emotions a user has expressed in the past, and analyzing this data can reveal patterns and trends.

[1753] The "most effective communication method" is the method that provides the best response and feedback to the user's emotional state.

[1754] "Real-time" refers to data processing and feedback occurring immediately, without delay.

[1755] "Security situation" refers to the safety-related state and risks faced by users and systems.

[1756] "Feedback" is a response or advice provided to a user's actions or status.

[1757] The system for implementing this invention mainly consists of three elements: a server, a terminal, and a user. The function of each element will be explained below.

[1758] server

[1759] The server has a means of linking text entered into the communication tool to the generative AI. Specifically, it receives text data entered by the user into the device and sends it to the generative AI. The generative AI analyzes the entered text data and determines the user's emotions. The results of this emotion determination are saved by the server and used for later analysis.

[1760] Furthermore, the server has a means of analyzing past emotional data and proposing the most effective communication method, which allows it to learn which emotions the user expressed in the past resulted in the most effective feedback, and use this knowledge for future communications.

[1761] Terminal

[1762] The device is responsible for sending the text data entered by the user to the server. It also has a means of visualizing the emotion assessment results and feedback received from the server to the user, allowing the user to check their own emotional state in real time.

[1763] The device also has a means of detecting emotions in real time and providing feedback according to the security situation. For example, if a user inputs "I'm very anxious right now," the device will provide advice such as "Take a deep breath and calm down" based on the feedback received from the server.

[1764] User

[1765] Users input text data through their devices and receive feedback from the server. This allows users to understand their own emotional state and take appropriate action. Furthermore, by receiving suggestions on the most effective communication methods based on past emotional data, users can improve their communication skills.

[1766] Hardware and software used

[1767] Hardware: Smartphone (iOS or Android)

[1768] Software: Python, scikit-learn (machine learning library), joblib (saving and loading models)

[1769] Specific examples

[1770] For example, if a user types "I'm very anxious right now," the server sends this text data to the generative AI, which determines the emotion as "anxiety." The server then analyzes past emotion data and suggests the most effective feedback: "Take a deep breath and calm down." The device displays this feedback to the user, who can then act accordingly.

[1771] Prompt Sentence Examples

[1772] "Write a Python program that determines the emotion of text entered by a user and provides the most effective feedback. Use a pre-trained model to determine the emotion, and a model trained on past data to provide the feedback."

[1773] In this way, the present invention accurately determines the user's emotions and provides appropriate feedback, enabling the user to improve their communication skills and respond appropriately to security situations.

[1774] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1775] Step 1:

[1776] The user inputs text data into the terminal. The input text data is used as initial data for determining the user's emotion. Input: Text data input by the user. Output: Text data sent to the terminal.

[1777] Step 2:

[1778] The device sends the input text data to the server. The server receives this data and connects it to the generative AI. Input: Text data sent from the device. Output: Text data sent to the server.

[1779] Step 3:

[1780] The server sends the received text data to the generative AI, which determines the emotion. The generative AI analyzes the text data and determines the user's emotion. Input: Text data sent to the server. Output: Emotion data determined by the generative AI.

[1781] Step 4:

[1782] The server stores the emotional data received from the generative AI and analyzes it together with past emotional data. This generates data to suggest the most effective communication methods. Input: Emotional data received from the generative AI. Output: Analyzed communication method data.

[1783] Step 5:

[1784] The server generates the most effective feedback based on the analysis results and sends it to the device. Input: Analyzed communication method data. Output: Feedback data sent to the device.

[1785] Step 6:

[1786] The device visualizes the feedback data received from the server to the user, allowing the user to check their own emotional state and the appropriate feedback in real time. Input: Feedback data received from the server. Output: Feedback information displayed to the user.

[1787] Step 7:

[1788] The user adjusts their behavior based on the feedback displayed on the device. This allows the user to take the optimal action based on their emotional state. Input: Feedback information displayed on the device. Output: User behavior adjustment.

[1789] In this way, a system is realized that can accurately determine a user's emotions and provide appropriate feedback, enabling them to improve their communication skills and respond appropriately to security situations.

[1790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1791] "Example 1"

[1792] In one embodiment of the present invention, when a user inputs a sentence into a communication tool, the sentence is linked to a generative AI. The generative AI uses an emotion engine to determine the user's emotion from the sentence. The emotion engine infers the emotion from, for example, words, phrases, emojis, etc. in the sentence. The determined emotion is visualized in the app, and the user can use the results as a reference when selecting the timing of communication.

[1793] "Example 2"

[1794] In another embodiment, the generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns patterns of emotional fluctuation from past communication data. It then predicts the optimal timing for communication based on the learned data. The predicted timing is presented to the user on the app, allowing the user to select the timing of communication based on this information.

[1795] "Example 3"

[1796] In yet another embodiment, the generative AI uses an emotion engine to analyze what emotions the user displayed in the past to achieve the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotion assessment results at that time. It then provides feedback to the user based on these results. The user can use this feedback to improve their own communication skills.

[1797] The processing flow of each embodiment will be described below.

[1798] "Example 1"

[1799] Step 1: The user enters text into the communication tool.

[1800] Step 2: The input text is linked to the generative AI.

[1801] Step 3: The generative AI uses an emotion engine to determine the user's emotion from the text. The emotion engine infers emotion from words, phrases, emojis, etc. in the text.

[1802] Step 4: The determined emotions are visualized in the app, and the user can use this result to choose the timing of communication.

[1803] "Example 2"

[1804] Step 1: The generative AI uses an emotion engine to learn the user's emotional fluctuation patterns. Specifically, it extracts and learns emotional fluctuation patterns from past communication data.

[1805] Step 2: The generative AI predicts the optimal timing for communication based on the learned data.

[1806] Step 3: The predicted timing is presented to the user on the app, and the user uses this information to choose the timing of communication.

[1807] "Example 3"

[1808] Step 1: The generative AI uses an emotion engine to analyze what emotions the user displayed in the past that resulted in the most effective communication. Specifically, it extracts the most effective communication situations from past communication data and the emotional assessment results at that time.

[1809] Step 2: The generative AI provides feedback to the user based on the results. Step 3: The user uses this feedback to improve their communication skills.

[1810] Example 1

[1811] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1812] Conventional communication tools have had the problem that it is difficult for users to accurately grasp the emotions of others, making it difficult to communicate at the right time. In addition, there are insufficient means for providing users with the emotion assessment results, and it is unclear how users should use the results.

[1813] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for linking a sentence input to a communication tool to a generative AI, a means for determining the emotion of the person who wrote the sentence by the generative AI, a means for visualizing the determination result in an app, and a means for the user to select the timing of communication based on the emotion determination result. This allows the user to accurately understand the other person's emotion and communicate at an appropriate time.

[1814] A "communication tool" is software or a platform that allows users to exchange messages with other users.

[1815] "Generative AI" is an artificial intelligence technology that analyzes input data and generates specific results.

[1816] The "means for determining emotion" is an algorithm or system for analyzing emotion from input text and identifying that emotion.

[1817] "Visualization means" refers to an interface or function that displays the analysis results in a form that is easy for the user to understand.

[1818] The "means for the user to select the timing of communication based on the emotion determination result" is a guideline or function that allows the user to determine the optimal timing of communication based on the displayed emotion determination result.

[1819] The "means for providing feedback" is a function for providing advice and information to the user based on the emotion determination result.

[1820] MODE FOR CARRYING OUT THE INVENTION

[1821] This invention is a system that links sentences entered by users into a communication tool with generative AI, determines emotions from the sentences, and visualizes the determination results in an app. A specific embodiment of this system is described below.

[1822] A user types a sentence into Google Chat

[1823] A user uses a communication tool such as Google Chat to type a message, for example, "Today was a great day!" This message is then sent over the Internet to a server.

[1824] The server connects the text to a generative AI

[1825] The server receives messages sent by users using the Google Chat API. The received messages are sent to the generative AI, which has an emotion engine and prepares the text for analysis.

[1826] Generative AI analyzes text and determines emotions

[1827] Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in input text. For example, it can determine positive emotions from the word "fun." This analysis uses natural language processing technology and machine learning algorithms.

[1828] The determined emotion is returned to the server

[1829] The generative AI returns the analysis results to the server, which then receives the results and proceeds to the next step.

[1830] The server sends the emotion determination result to the app.

[1831] The server sends the emotion determination results to the user's device, which receives the results and prepares them for display in the app.

[1832] The device displays the emotion determination result in the app.

[1833] The device displays the received emotion determination result in the app. The user can check the emotion determination result through the app. For example, the app may display "positive emotion."

[1834] Users choose when to communicate

[1835] The user can choose the timing of communication based on the emotion assessment results displayed on the app. For example, if the other person is showing positive emotions, they can choose to reply at that time.

[1836] Specific examples

[1837] A user types "Today was such a fun day!" into Google Chat and presses the send button. This message is sent to the server, which receives the text via the Google Chat API. The server connects the received text to a generative AI, which analyzes the word "fun" and determines the positive emotion. The generative AI returns the positive emotion determination result to the server, which then sends the emotion determination result to the user's device. The emotion determination result received by the device is displayed in the app, and is displayed as "positive emotion." The user checks the "positive emotion" displayed in the app and chooses to reply at that time.

[1838] Prompt Sentence Examples

[1839] "Create a generative AI program that analyzes text entered in Google Chat and determines the emotion. Include a function that uses an emotion engine to infer emotion from words, phrases, emojis, etc. in the text, and visualize the results in the app."

[1840] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1841] Step 1: User types a sentence into Google Chat

[1842] What happens: A user opens the Google Chat app and types a message, for example, "Today was such a fun day!"

[1843] Input: Text entered by the user

[1844] Output: Message sent to Google Chat

[1845] Step 2: The input text is sent to the server

[1846] How it works: When a user sends a message, the text is sent over the Internet to a server, which then receives the message using the Google Chat API.

[1847] Input: Message sent from Google Chat

[1848] Output: Message received by server

[1849] Step 3: The server connects the text to the generative AI

[1850] Specific operation: The server sends the received text to the generative AI, which has an emotion engine and prepares to analyze the text.

[1851] Input: Message received by the server

[1852] Output: Message sent to the generative AI

[1853] Step 4: Generative AI analyzes the text and determines the sentiment

[1854] How it works: Generative AI uses an emotion engine to analyze words, phrases, emojis, etc. in the input text. For example, it determines positive emotions from the word "fun."

[1855] Input: Message sent to the generative AI

[1856] Output: Emotion determination result (e.g., positive emotion)

[1857] Step 5: The determined emotion is returned to the server

[1858] Specific operation: The generative AI returns the analysis results to the server, which receives the results and proceeds to the next step.

[1859] Input: Emotion determination result

[1860] Output: Emotion judgment result returned to the server

[1861] Step 6: The server sends the emotion result to the app

[1862] Specific operation: The server sends the emotion determination result to the user's device. The device receives the result and prepares to display it in the app.

[1863] Input: Emotion judgment result returned to the server

[1864] Output: Emotion judgment result sent to the user's device

[1865] Step 7: The device displays the emotion detection result in the app.

[1866] Specific operation: The device displays the received emotion judgment result in the app. The user can check the emotion judgment result through the app. For example, the app will display "positive emotion."

[1867] Input: Emotion judgment result sent to the user's device

[1868] Output: Emotion judgment results displayed on the app

[1869] Step 8: Users choose when to communicate

[1870] Specific operation: The user chooses the timing of communication based on the emotion judgment results displayed in the app. For example, if the other person is showing positive emotions, they choose to reply at that time.

[1871] Input: Emotion judgment result displayed on the app

[1872] Output: User-selected timing of communication

[1873] (Application example 1)

[1874] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1875] With conventional communication tools, it was difficult for users to accurately grasp the emotions of the other party, making it difficult to communicate at the right time. Also, in physical stores, store staff are required to understand customers' emotions in real time and respond optimally, but there was a lack of effective means to do so. This led to the issue of not being able to improve customer satisfaction or provide efficient customer service.

[1876] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1877] In this invention, the server includes means for linking text entered into the communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an app, means for converting voice data into text, means for displaying the emotion determination result on a display device, and means for providing a response guide according to the emotion. This allows users to understand the other person's emotion in real time and communicate and respond to customers at the optimal time.

[1878] A "communication tool" is software or hardware that allows users to exchange information with each other through text or voice.

[1879] "Generative AI" is an artificial intelligence system that generates new information based on input data.

[1880] "Emotion determination" is the process of identifying and classifying the emotions contained in the content of input text or audio data.

[1881] "Visualization" is a method of visually displaying data or information so that users can easily understand it.

[1882] "Audio data" refers to data in which audio is recorded in digital format.

[1883] "Text conversion" is the process of converting audio data into text data.

[1884] A "display device" is a device for visually displaying information, and includes displays and screens.

[1885] A "response guide" is a set of instructions or advice that suggests appropriate actions or words to use in a given situation or emotion.

[1886] As an embodiment of the present invention, a system for supporting customer service in brick-and-mortar stores will be described as an example. This system is configured using hardware and software such as smart glasses, a microphone, a generative AI model, an emotion engine, and a display device.

[1887] First, a microphone built into the smart glasses captures what the customer says. The captured voice data is converted into text using the Google Cloud Speech-to-Text API. The converted text is then sent to a generative AI model (e.g., OpenAI GPT-4) and emotion is determined using an emotion engine (e.g., IBM Watson Tone Analyzer).

[1888] The detected emotion is displayed on the smart glasses' display. For example, positive emotions (such as joy and relief) are displayed in green, while negative emotions (such as anger and anxiety) are displayed in red. A guide to how to respond based on the emotion is also displayed. For example, if a customer is feeling anxious, phrases or actions to reassure them are suggested.

[1889] This system allows store staff to understand customer emotions in real time and respond optimally, thereby improving customer satisfaction and enabling more efficient customer service.

[1890] For example, the following prompt is input to a generative AI model:

[1891] Customer says: "I'd like to learn more about this product, but I'm a little unsure."

[1892] Prompt: "Determine the customer's sentiment from this statement."

[1893] Based on this prompt, the generative AI model determines the customer's emotions and displays the results on the smart glasses' display, allowing the store clerk to respond optimally based on the customer's emotions.

[1894] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1895] Step 1:

[1896] A microphone in the smart glasses captures what the customer says.

[1897] Input: Customer voice

[1898] Output: Audio data

[1899] Specific operation: The microphone built into the smart glasses records the customer's speech in real time and saves it as audio data.

[1900] Step 2:

[1901] The server converts the audio data into text.

[1902] Input: Audio data

[1903] Output: Text data

[1904] Specific operation: The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1905] Step 3:

[1906] The server sends the text data to a generative AI model to determine the emotion.

[1907] Input: Text data

[1908] Output: Emotion determination result

[1909] Specific operation: The server sends text data to a generative AI model (e.g., OpenAI GPT-4), which then uses an emotion engine (e.g., IBM Watson Tone Analyzer) to determine the emotion.

[1910] Step 4:

[1911] The server displays the emotion determination result on the display of the smart glasses.

[1912] Input: Emotion determination result

[1913] Output: Display

[1914] Specific operation: The server sends the emotion judgment result to the smart glasses display, and positive emotions are displayed in green and negative emotions in red.

[1915] Step 5:

[1916] The server provides a response guide based on the emotion.

[1917] Input: Emotion determination result

[1918] Output: Corresponding guide

[1919] Specific operation: Based on the emotion determination result, the server generates a response guide and displays it on the smart glasses display. For example, if the customer is feeling anxious, it will suggest phrases or actions to reassure them.

[1920] Example 2

[1921] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1922] With conventional communication tools, it was difficult to accurately assess a user's emotions and provide optimal communication timing based on the results. In addition, there was a lack of means for users to use past communication data to understand patterns of emotional fluctuations and communicate effectively. This meant that users were unable to communicate at the appropriate time, resulting in a decline in the quality of communication.

[1923] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1924] In this invention, the server includes means for linking text entered into a communication tool to a generative AI, means for determining the emotion of the person who wrote the text using the generative AI, means for visualizing the determination result in an application, means for collecting past communication data, means for extracting patterns of emotional fluctuation from the collected data, means for predicting the optimal communication timing based on the extracted patterns, and means for presenting the prediction result in the application. This enables the user to understand patterns of emotional fluctuation and communicate at the optimal timing.

[1925] "Communication Tools" means software or applications that allow users to exchange messages or information with other users.

[1926] "Generative AI" is a system that uses artificial intelligence technology to generate new information and results from input data.

[1927] "Emotion determination" is the process of analyzing a user's emotions from input text and data and identifying the type and intensity of those emotions.

[1928] An "application" is a software program that a user uses to perform a specific task.

[1929] "Data collection" is the process of collecting and storing a user's past communication data.

[1930] "Emotional fluctuation patterns" are trends or patterns in data that show how a user's emotions change over time and in different situations.

[1931] "Communication timing" refers to the optimal time and situation for a user to communicate with other users.

[1932] "Forecasting" is the process of predicting future events or outcomes based on past data and patterns.

[1933] ...

Claims

1. A means for linking text input into a communication tool to a generative AI; A means for determining the emotion of a user who created the text using the generative AI; means for analyzing past communication data and the user's emotions corresponding to the communication data; means for generating feedback for the text based on the determined user's sentiment and an analysis result by the analyzing means; means for providing the generated feedback to the user; A system including:

2. The generative AI further includes a means for determining an optimal timing for communication based on the determined emotion of the user. The system of claim 1 .

3. A means for visualizing the determined emotion of the user; and a means for displaying the visualized emotion of the user on a display device. The system of claim 1 .

Images