System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting in-house text-based communication data; means for feeding the collected data into a natural language processing engine and performing sentiment analysis; means for storing the analysis results and calculating a sentiment score; means for generating and notifying an alert when the sentiment score falls below a set threshold; and means for generating and displaying periodic reports based on the sentiment score and analysis results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, the spread of remote work and flexible work has reduced face-to-face communication, making it difficult to detect employee stress and dissatisfaction early within an organization and take appropriate measures. There is a need for effective tools that can grasp employees' psychological health in real time and lead to improvements in productivity and workplace atmosphere. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for collecting text-based communication data within a company, a means for supplying the collected data to a natural language processing engine and performing sentiment analysis, a means for saving the analysis results and calculating a sentiment score, a means for generating and notifying an alert when the sentiment score falls below a set threshold, and a means for generating and displaying periodic reports based on the sentiment score and analysis results. Furthermore, by adding a means for collecting data from multiple internal communication tools and a means for tracking fluctuations in sentiment scores and analyzing trends, it is possible to accurately monitor the emotional health of an organization and improve the work environment.

[0006] "Internal text-based communication data" refers to text messages sent by employees through messaging platforms, emails, chat tools, etc. used within the company.

[0007] "Collection Method" means a software or hardware mechanism configured to automatically and efficiently collect internal company text-based communications data.

[0008] A "natural language processing engine" refers to a machine learning algorithm or model that analyzes collected text data and understands its emotions and intent.

[0009] "Sentiment analysis" refers to the process of identifying emotions or attitudes (e.g., positive, negative, neutral) within text data and calculating a corresponding score.

[0010] "Means for storage" refers to a mechanism for storing analysis results and emotion scores in a database or other storage system so that they can be referenced later.

[0011] An "emotion score" is a numerical indicator of the emotional state of text data, usually on a scale of positive, negative, or neutral.

[0012] "Alert generation and notification means" refers to the systems and processes for sending alerts and notifications to designated recipients (e.g., HR or managers) when sentiment scores fall below established thresholds.

[0013] "Means for generating and displaying periodic reports" refers to a mechanism for aggregating and organizing the results of sentiment analysis at regular intervals and displaying them in a visually easy-to-understand format.

[0014] "Multiple internal communication tools" refers to multiple messaging platforms, email systems, and chat applications used within the company.

[0015] "Means for tracking changes in emotion scores and analyzing trends" refers to a mechanism for recording and tracking changes in emotion scores over time and using that data to analyze trends in emotional states. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] As an embodiment of the present invention, the configuration of the EmoSync system and program processing will be described in detail below.

[0038] System Configuration

[0039] 1. Server

[0040] The server has an API interface for collecting text-based message data from internal communication tools.

[0041] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[0042] The server stores the analysis results in a database and calculates emotion scores in real time.

[0043] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[0044] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[0045] 2. Device (user's PC or smartphone)

[0046] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[0047] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0048] 3. Users (managers, human resources department)

[0049] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0050] Program processing

[0051] The program of this system performs the following main processes.

[0052] 1. Data Collection

[0053] The device collects data on messages sent by users using internal communication tools and sends it to a server via an API.

[0054] Example: When a user sends a message on Slack saying, "The project is really tough right now," that message is collected on the server.

[0055] 2. Data Preprocessing

[0056] The server cleanses the received message data and formats it into the required format.

[0057] Example: Cleaning up text data by removing unnecessary HTML tags and special characters from messages.

[0058] 3. Sentiment analysis

[0059] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0060] Example: An NLP engine analyzes the text "The project is really tough right now" and calculates a negative sentiment score (e.g., -0.5).

[0061] 4. Storing and calculating sentiment scores

[0062] The server stores the analysis results in a database and calculates an emotion score.

[0063] Example: Record the sentiment score of each message in a database and calculate the average sentiment score for each user or team.

[0064] 5. Alerting and Notification

[0065] The server generates an alert if the emotion score falls below a set threshold and sends a notification to the relevant manager or HR department.

[0066] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[0067] 6. Report Generation and Viewing

[0068] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[0069] Example: Create a monthly report containing company-wide and departmental sentiment scores and trend graphs and provide it to managers.

[0070] 7. Feedback and Actions

[0071] Users check the reports and alerts provided by the server, consider and implement the necessary measures.

[0072] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[0073] In this way, the EmoSync system monitors the emotional climate within an organization in real time and takes appropriate measures at the right time to improve the work environment.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The device receives text messages sent by the user using internal communication tools (e.g., messaging apps, email systems).

[0077] Example: An employee sends a message on a messaging app saying, "I'm stressed because the project isn't progressing."

[0078] Step 2:

[0079] The terminal transmits the acquired text message to the server.

[0080] Example: Sending text message data to a server through a messaging app's API.

[0081] Step 3:

[0082] The server receives the text message sent from the terminal and temporarily stores it in a database.

[0083] Example: Recording received messages in a database and preparing them for cleansing processing.

[0084] Step 4:

[0085] The server cleans the received message data, removing unnecessary HTML tags and special characters.

[0086] Example: Removing redundant tags and symbols from the text "I'm stressed because the project isn't progressing."

[0087] Step 5:

[0088] The server passes the cleaned text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0089] Example: An NLP engine analyzes the text "I'm stressed because the project isn't progressing" and calculates a sentiment score (e.g., -0.6).

[0090] Step 6:

[0091] The server stores the calculated emotion scores in a database.

[0092] Example: Analysis results and sentiment scores are recorded in a database for each user and message.

[0093] Step 7:

[0094] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[0095] Example: Aggregate each user's sentiment score from a week's worth of message data and calculate the average value.

[0096] Step 8:

[0097] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[0098] Example: Alert if a team's sentiment score falls below -0.3.

[0099] Step 9:

[0100] The server notifies the generated alerts to the HR department and relevant managers.

[0101] Example: Sending alert messages to HR and managers via email and push notifications.

[0102] Step 10:

[0103] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[0104] Example: Create and distribute a monthly report containing sentiment scores and trend graphs for each department.

[0105] Step 11:

[0106] The terminal receives reports and alerts sent from the server and displays them to the user.

[0107] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[0108] Step 12:

[0109] Users check the alerts and reports provided by the server and take appropriate action.

[0110] Example: HR receives an alert and provides mental health counseling to a specific employee.

[0111] In this way, the EmoSync system can monitor employees' emotional states in real time, detect problems early, and take countermeasures.

[0112] Example 1

[0113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Conventional internal communication tools do not analyze the emotional content of messages, making it difficult to provide the feedback and early action needed to improve the work environment. The accumulation of negative emotions, in particular, can lead to lower employee morale, poor performance, and even turnover. Furthermore, the lack of reports based on concrete data makes it difficult for managers to implement effective countermeasures.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0116] In this invention, the server includes means for collecting in-house text-based communication data, means for data cleansing the collected data and formatting it into a required format, means for supplying the formatted data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, and means for generating and displaying periodic reports based on the sentiment score and analysis results. This makes it possible to detect emotions from in-house communications in real time and take early countermeasures.

[0117] "Internal text-based communication data" refers to textual data generated from messaging tools and emails used within the company.

[0118] "Data cleansing" is the process of removing unnecessary information and noise from collected data and preparing it in a format suitable for analysis.

[0119] A "natural language processing engine" is a software system that includes algorithms and techniques for understanding, analyzing, and generating human language.

[0120] "Sentiment analysis" is the process of identifying positive, negative, and neutral sentiments from text data and calculating their scores.

[0121] An "emotion score" is data that quantifies the strength and type of emotion contained in specific text data.

[0122] A "threshold" is a reference value for determining whether a particular indicator or value exceeds a certain standard.

[0123] An "alert" is a notification of a warning or caution that is generated when a pre-defined condition is met.

[0124] A "report" is a document or digital information that compiles analytical results and statistical data and provides them to users.

[0125] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0126] System Configuration

[0127] 1. Server

[0128] The server has an API interface for collecting internal text-based communication data. The server has a built-in natural language processing engine and performs sentiment analysis on the collected data. The sentiment analysis results are stored in a database and a sentiment score is calculated. The server monitors the set threshold in real time and has the ability to generate alerts and send notifications if the sentiment score falls below a certain level. It also generates regular reports and provides them to users via an accessible dashboard or by email.

[0129] 2. Device (user's PC or smartphone)

[0130] The terminal has an interface that allows users to send text messages using an internal communication tool, and has the function of receiving emotion scores, alerts, and reports sent from the server and displaying them to the user.

[0131] 3. Users (managers, human resources department)

[0132] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0133] Program processing

[0134] The program of this system operates around three parties: the server, the terminal, and the user.

[0135] Server: AWS or a similar cloud infrastructure is used, and the program is implemented in Python or JavaScript. For natural language processing, the Google Cloud Natural Language API or engine is used as the NLP engine. MySQL or PostgreSQL is used as the database.

[0136] Device: Runs on the user's PC or smartphone. It uses a web browser to communicate with the server via API and displays sentiment scores, alerts, and reports in real time.

[0137] Users: Users monitor the emotional state of their organization through the provided dashboards and email reports and consider necessary countermeasures.

[0138] Specific examples

[0139] Data Collection and Cleansing

[0140] The device collects messages sent by users via internal communication tools and sends them to the server. For example, if a user sends a message on Slack saying, "The project is really tough right now," the message is sent to the server via Slack's API. The server cleanses the received message data, removing HTML tags and special characters, and formats it into the required format.

[0141] sentiment analysis

[0142] The server passes the cleansed text data to an NLP engine for sentiment analysis. For example, the text "The current project is really tough" is analyzed as a negative sentiment score of -0.5. The server stores these scores in a database and calculates average sentiment scores for each user or team.

[0143] Alerting and Notifications

[0144] The server generates an alert when the sentiment score falls below a set threshold and sends a notification to the relevant manager or HR department. For example, if a team's sentiment score falls below -0.3, an alert is sent to the HR department via email or push notification.

[0145] Report Generation and Viewing

[0146] The server generates regular reports based on the emotion scores and analysis results and provides them to users via dashboards or email. For example, at the end of each month, a report showing the emotion scores and trends for the entire company or department is created and provided to managers.

[0147] Feedback and Actions

[0148] Users can check reports and alerts provided by the server, consider and implement necessary measures, etc. For example, if a manager notices a decline in the team's emotional score, they can set up an online meeting to check in with team members on their mental health.

[0149] Prompt Sentence Examples

[0150] 1. "Write a program that collects Slack messages and sends them to a server."

[0151] 2. "Write a Python program to cleanse text data."

[0152] 3. "Implement an NLP engine for sentiment analysis of text data."

[0153] 4. "Write a SQL query to store the sentiment scores and perform statistical processing."

[0154] 5. "Write a program that generates an alert and sends a notification when a specific condition occurs."

[0155] 6. "Write a program that generates a report of emotion scores."

[0156] 7. "Create guidelines for feedback and action planning based on the sentiment score report."

[0157] In this way, the EmoSync system monitors the emotional state within an organization in real time and takes appropriate measures in a timely manner to improve the work environment.

[0158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0159] Step 1: Data collection

[0160] The device collects text messages sent by users using internal communication tools and sends them to a server via API. For example, if a user sends a message on Slack saying, "The project is really tough right now," that message is forwarded to the server via the Slack API. The input of this step is the user's message, and the output is the raw text data sent to the server.

[0161] Step 2: Data Preprocessing

[0162] The server cleanses the received message data, removes unnecessary HTML tags and special characters, and formats it into the required format. This project is really tough " into clean text "The current project is really tough." The input of this step is raw text data and the output is cleansed text data.

[0163] Step 3: Sentiment analysis

[0164] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis. Specifically, the NLP engine calculates a negative sentiment score (e.g., -0.5) for the text "This project is really tough." The input for this step is the cleansed text data, and the output is the sentiment score for the text.

[0165] Step 4: Storing and calculating sentiment scores

[0166] The server stores the analysis results in a database, records the sentiment score for each message, and calculates the average sentiment score for each user or team. Specifically, the server adds the score for each message to the database and calculates the average score for each user or team. The input for this step is the sentiment score, and the output is the score stored in the database and the calculated average score.

[0167] Step 5: Alerting and Notification

[0168] The server generates an alert when the emotion score falls below the set threshold and sends a notification to the relevant manager or HR department. For example, if the team's average emotion score falls below -0.3, an alert is sent to the HR department via email or push notification. The input of this step is the emotion score and the set threshold, and the output is the generated alert notification.

[0169] Step 6: Generate and view the report

[0170] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via dashboards or email. Specific operations include generating monthly reports, creating PDFs containing company-wide and department-specific sentiment scores and graphs showing trends, and distributing these to each manager via email. The input to this step is the sentiment scores and analysis results, and the output is the generated report.

[0171] Step 7: Feedback and action

[0172] The user checks the reports and alerts provided by the server, considers and implements the necessary response. For example, a manager may notice a decline in their team's emotional score and schedule an online meeting to conduct a mental health check-in. The input for this step is the report or alert, and the output is the user's response or feedback.

[0173] (Application example 1)

[0174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] In traditional brick-and-mortar stores, it was difficult to grasp customer satisfaction and emotional state in real time, making it difficult to respond quickly to directly improve the customer experience. Furthermore, there was a lack of means to effectively collect customer feedback and use it to improve store operations. This resulted in delays in taking concrete measures to prevent declines in customer satisfaction and customer defections.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0177] In this invention, the server includes a means for collecting video data for capturing customer behavior and facial expressions, a means for preprocessing the collected video data using an image processing library, and a means for supplying the preprocessed data to emotion recognition software to perform emotion analysis, thereby enabling real-time analysis of customer emotions and prompt response.

[0178] "Video data" is visual information that captures the customer's behavior and facial expressions.

[0179] An "image processing library" is a software tool for preprocessing collected video data.

[0180] "Emotion recognition software" is a program for analyzing customer emotions from pre-processed video data.

[0181] An "emotion score" is a numerical evaluation value of an emotion calculated by emotion recognition software.

[0182] An "alert" is a warning notification that is generated when an emotion score falls below a set threshold.

[0183] "Reports" are regular reports summarizing sentiment scores and analysis results.

[0184] "Data preprocessing" is the process of removing unnecessary information from collected video data and preparing it in a format suitable for analysis.

[0185] A "natural language processing engine" is a program that analyzes text feedback entered by customers and calculates an emotional score.

[0186] A "dashboard" is a web application interface for visually displaying sentiment scores and analysis results.

[0187] "Trend analysis" is the process of tracking fluctuations in sentiment scores and analyzing patterns over time.

[0188] As an embodiment of the present invention, the configuration of the CustomerEmoSync system and program processing will be described in detail below.

[0189] System Configuration

[0190] 1. Server

[0191] The server has a means for collecting video data to capture customer behavior and facial expressions.

[0192] The server uses an image processing library (e.g., OpenCV) to preprocess the collected video data.

[0193] The server uses emotion recognition software (e.g., Microsoft Azure Face API) to perform customer emotion analysis from the pre-processed video data.

[0194] The server stores the results of the sentiment analysis and has a means for calculating sentiment scores.

[0195] The server has a means to generate an alert and notify if the emotion score falls below a set threshold.

[0196] The server has the means to generate and display periodic reports based on the sentiment scores and analysis results.

[0197] 2. Device (user's smartphone)

[0198] The terminal has an interface for the customer to input a feedback message.

[0199] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0200] 3. Users (store managers and staff)

[0201] Users check the reports and alerts provided by the server and take the necessary measures.

[0202] Program processing

[0203] 1. Data Collection

[0204] The video data collected by the camera is transmitted to a server.

[0205] Text feedback entered by the customer using their smartphone is also sent to the server.

[0206] 2. Data Preprocessing

[0207] The server uses image processing libraries such as OpenCV to cleanse the collected video data and format it into the required format.

[0208] Customer text feedback is also passed to a natural language processing engine for cleansing.

[0209] 3. Sentiment analysis

[0210] The server passes the pre-processed video data to emotion recognition software such as Microsoft Azure Face API to perform sentiment analysis.

[0211] A natural language processing engine is also used to perform sentiment analysis of the text feedback.

[0212] Specific examples

[0213] For example, if a camera captures a customer's facial expression in a store and recognizes that the customer is happy, the emotion score will be calculated as a high positive value. Also, if a customer types feedback on their smartphone, such as "The service was great," the text will be analyzed by a natural language processing engine as a positive emotion.

[0214] An example of a prompt sentence might be, "Please perform sentiment analysis on the following text: 'I am very satisfied with your service today. Thank you.'"

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] The server collects video data from cameras in the store, which transmits customer behavior and facial expressions to the server in real time. The input is video data from the cameras, which the server receives and stores.

[0218] Step 2:

[0219] The server cleanses the collected video data using an image processing library (such as OpenCV). It removes unnecessary parts from the video data and formats it into the required format. The input is the video data collected in step 1, and the output is the video data after preprocessing.

[0220] Step 3:

[0221] The server then feeds the preprocessed video data to emotion recognition software (such as Microsoft Azure Face API) to perform emotion analysis. This software calculates an emotion score based on the customer's facial expressions. The input is the preprocessed video data from step 2, and the output is the analysis results, including the emotion score.

[0222] Step 4:

[0223] The user inputs a feedback message using a smartphone. The smartphone app collects the text data and sends it to the server. The input is the text feedback from the customer, and the output is the text data sent to the server.

[0224] Step 5:

[0225] The server uses a natural language processing engine to analyze the customer's text feedback and calculate a sentiment score from the text data. The input is the text data received in step 4, and the output is the sentiment score of the text.

[0226] Step 6:

[0227] The server integrates the emotion scores obtained from the video data and text data and stores the overall emotion score in a database. The inputs are the individual emotion scores, and the output is the integrated overall emotion score.

[0228] Step 7:

[0229] The server generates an alert when the integrated overall emotion score falls below a set threshold. The alert is sent to the administrator's terminal. The input is the integrated emotion score, and the output is an alert notification.

[0230] Step 8:

[0231] The server generates periodic reports based on the sentiment scores and analysis results. These reports are provided to users via a dashboard or email. The input is the sentiment scores and analysis results stored in the database, and the output is the periodic reports.

[0232] Step 9:

[0233] Users check the reports and alerts provided by the server and take necessary measures. For example, if customer satisfaction is low, they can implement specific improvement measures such as reviewing staff response. The input is the report or alert, and the output is the specific measures.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] As an embodiment of the present invention, a configuration in which the EmoSync system and an emotion engine are combined will be described in detail.

[0236] System Configuration

[0237] 1. Server

[0238] The server has an API interface for collecting text-based message data from internal communication tools.

[0239] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[0240] The server stores the analysis results in a database and calculates emotion scores in real time.

[0241] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[0242] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[0243] The server recognizes the user's emotions through an emotion engine and has the ability to analyze input other than text data (emoticons and voice data).

[0244] 2. Device (user's PC or smartphone)

[0245] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[0246] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0247] The terminal has the function of sending input such as voice data and emojis to a server.

[0248] 3. Users (managers, human resources department)

[0249] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0250] Program processing

[0251] The program of this system performs the following main processes.

[0252] 1. Data Collection

[0253] The device collects all messages, emoticons, and voice data sent by users using internal communication tools.

[0254] Example: An employee texts on a messaging app saying, "I'm stressed because the project isn't progressing," while also using a tired-face emoji.

[0255] 2. Data Transmission

[0256] The terminal transmits the acquired text messages, emojis, and voice data to the server.

[0257] Example: Sending text messages, emoji data, and voice data to a server via a messaging app's API.

[0258] 3. Data Preprocessing

[0259] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[0260] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing" and organize the emojis and audio files that were passed through it.

[0261] 4. Emotion recognition

[0262] The server passes the cleaned text data, emojis, and voice data to the emotion engine to perform emotion recognition.

[0263] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," along with an emoji of a tired expression, and also voice data, and calculates an overall emotion score (for example, -0.7) from all elements.

[0264] 5. Storing and calculating sentiment scores

[0265] The server stores the analysis results in a database and calculates an emotion score.

[0266] Example: Analysis results and sentiment scores are recorded in a database for each user and message, and then aggregated.

[0267] 6. Alert Generation and Notification

[0268] The server generates an alert if the emotion score falls below a set threshold and notifies the relevant manager or HR department.

[0269] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[0270] 7. Report Generation and Viewing

[0271] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[0272] Example: Create a monthly report for managers that includes company-wide and departmental sentiment scores and trend graphs.

[0273] 8. Feedback and Actions

[0274] Users check the reports and alerts provided by the server and take appropriate action.

[0275] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[0276] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[0277] The processing flow will be explained below.

[0278] Step 1:

[0279] The device captures text messages, emojis, and voice data sent by users using internal communication tools (e.g., messaging apps, email systems).

[0280] Example: An employee sends a text message on a messaging app saying, "I'm stressed because the project isn't progressing," along with a tired-face emoji and, if necessary, a voice message.

[0281] Step 2:

[0282] The device sends the acquired text messages, emojis, and voice data to the server all at once.

[0283] Example: Sending text messages, emojis, and voice data to a server through a messaging app's API.

[0284] Step 3:

[0285] The server receives text messages, emojis, and voice data sent from the device and temporarily stores them in a database.

[0286] Example: Record received messages, emoji information, and audio files in a database to prepare for the cleansing process.

[0287] Step 4:

[0288] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[0289] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing," convert emoji information into a format that's easier to analyze, and convert audio data into text.

[0290] Step 5:

[0291] The server passes the cleaned text data, emoji, and voice data to an emotion engine to perform emotion recognition.

[0292] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," an emoji showing a tired expression, and voice data, and calculates an overall emotion score (e.g., -0.7) from all of these elements.

[0293] Step 6:

[0294] The server stores the emotion scores obtained as a result of the emotion recognition in a database.

[0295] Example: Analysis results and sentiment scores are recorded in a database for each user and message, allowing for future reference.

[0296] Step 7:

[0297] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[0298] Example: Aggregate each user's emotion score from a week's worth of message data, emojis, and voice data, and calculate the average value.

[0299] Step 8:

[0300] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[0301] Example: Alert if a team's sentiment score falls below -0.3.

[0302] Step 9:

[0303] The server notifies the generated alerts to the HR department and relevant managers.

[0304] Example: Sending alert messages to HR and managers via email and push notifications.

[0305] Step 10:

[0306] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[0307] Example: Create a monthly report for managers that includes a trend graph of sentiment scores for the entire company and each department.

[0308] Step 11:

[0309] The terminal receives reports and alerts sent from the server and displays them to the user.

[0310] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[0311] Step 12:

[0312] Users check the alerts and reports provided by the server and take appropriate action.

[0313] Example: HR receives an alert and provides mental health counseling to a specific employee.

[0314] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[0315] Example 2

[0316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0317] Conventional systems only dealt with text-based communication data, which meant they had issues with insufficient understanding of emotions. Furthermore, they did not analyze emojis or voice data, making it difficult to accurately recognize users' emotions, leaving managers and HR departments with a lack of information to take appropriate action. Furthermore, there was a lack of a system that allowed managers to monitor emotion scores in real time and take prompt action.

[0318] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0319] In this invention, the server includes means for collecting text-based data, emoji, and voice data from in-house communication tools, means for supplying the collected data to a natural language processing engine and an emotion analysis engine to perform emotion recognition, means for storing the recognition results in a database and calculating individual data and overall emotion scores, means for generating an alert and notifying an administrator when the emotion score falls below a set threshold, and means for creating regular reports based on the emotion scores and data analysis results and displaying them on a dashboard or by email. This makes it possible to recognize user emotions in real time not only from text data but also from emoji and voice data, and to monitor the emotional state within an organization with greater accuracy.

[0320] "Internal communication tools" refers to digital platforms used to facilitate communication between employees.

[0321] "Text-based data" refers to communication content in the form of written information.

[0322] "Emoji" refers to small icons or pictograms used to visually express emotions or intentions.

[0323] "Audio data" refers to data that records words or sounds in digital format.

[0324] A "natural language processing engine" refers to software technology that analyzes text data and understands and processes it as natural language.

[0325] An "emotion analysis engine" refers to an analytical tool used to identify emotions and impressions from collected data.

[0326] "Emotion recognition" refers to the process of analyzing and extracting emotions from text, emoji, and voice data.

[0327] A "database" refers to a collection of information that allows data to be efficiently stored, managed, and retrieved.

[0328] "Emotion score" refers to a numerical representation of a user's emotional state based on analyzed data.

[0329] A "threshold" refers to a reference value that sets a certain condition or standard and changes operation depending on whether that standard is exceeded or fallen below.

[0330] An "alert" is a warning or notification that occurs based on set conditions.

[0331] "Administrator" means an individual or entity responsible for the oversight and management of a System.

[0332] A "dashboard" refers to a screen or interface designed to allow users to visually view a variety of data and information at a glance.

[0333] "Report" refers to a document that summarizes the results of data analysis and important information within a specific period of time.

[0334] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0335] System Configuration

[0336] The present invention is comprised of multiple elements: a server, a terminal, and a user.

[0337] server

[0338] The server has the following functions:

[0339] 1. Data Collection

[0340] The server has an API interface for collecting text-based messages, emojis, and voice data from internal communication tools.

[0341] 2. Data Preprocessing

[0342] The server cleanses the data it receives and formats it into an analyzable format, using a natural language processing engine to remove unnecessary tags and symbols from the text, and organizes and converts emojis and audio files.

[0343] 3. Emotion recognition

[0344] The server passes the cleansed data to a sentiment analysis engine for emotion recognition. The sentiment analysis engine analyzes the text "I'm stressed because the project isn't progressing," emojis showing tired expressions, and voice data to calculate an overall sentiment score.

[0345] 4. Database storage and emotion score calculation

[0346] The server stores the analysis results in a database and calculates individual data and overall emotion scores.

[0347] 5. Alerting and Notification

[0348] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold, allowing the administrator to grasp the user's emotional state in real time.

[0349] 6. Report Generation and Viewing

[0350] The server generates regular reports based on sentiment scores and data analysis results and provides them via dashboard or email.

[0351] Terminal

[0352] The terminal has the following features:

[0353] 1. Data entry and submission

[0354] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool and transmits them to the server.

[0355] 2. Results display

[0356] The terminal receives the emotion scores, alerts, and reports sent from the server and displays them to the user.

[0357] User

[0358] Users have the following roles:

[0359] 1. Check reports and alerts

[0360] Users (usually managers and HR departments) check the reports and alerts provided by the server and take necessary measures, such as scheduling team meetings to check in on mental health.

[0361] Specific examples

[0362] For example, if an employee uses a messaging app to send a text message saying, "I'm stressed because the project isn't progressing," and also uses a tired-face emoji, the following process will take place:

[0363] 1. The device receives this message and emoji and sends it to the server.

[0364] 2. The server preprocesses the received data and passes it to the sentiment analysis engine.

[0365] 3. The sentiment analysis engine analyzes the data and calculates a sentiment score (e.g., -0.7).

[0366] 4. The server stores the emotion score in a database and generates an alert to notify the administrator if the score falls below a threshold.

[0367] 5. The server generates periodic reports and provides them to the user via dashboard or email.

[0368] Prompt Sentence Examples

[0369] Use prompt statements of the following form to feed data into the generative AI model:

[0370] Prompt statement:

[0371] Please enter the text message "I'm stressed because the project isn't progressing," along with a tired-looking emoji, and any associated audio data.

[0372] As described above, the present invention makes it possible to recognize user emotions in real time not only from text data but also from pictograms and voice data, thereby monitoring the emotional state within an organization with higher accuracy.

[0373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0374] Step 1: Data collection

[0375] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool. This data is input.

[0376] Specifically, it uses the API of a messaging app to capture text messages such as "I'm stressed because the project isn't progressing," along with emojis of tired expressions, and related audio data. The output after data collection is the collected text data, emojis, and audio data.

[0377] Step 2: Send data

[0378] The device sends the captured text messages, emojis, and voice data to the server, where this data is entered.

[0379] Specifically, the device uses the messaging app's API to send this data to the server. The output after the data transmission is text data, emojis, and voice data sent to the server.

[0380] Step 3: Data Preprocessing

[0381] The server cleanses the received text messages, emojis, and voice data and formats them into an analyzable format. This received data is then input.

[0382] Specifically, it uses a natural language processing engine to remove unnecessary tags and symbols from text, and format and convert emoji and voice data. The output of the data preprocessing is cleansed text data, analyzable emoji, and voice data.

[0383] Step 4: Emotion Recognition

[0384] The server passes the formatted text data, emojis, and voice data to an emotion analysis engine to perform emotion recognition.

[0385] Specifically, the emotion analysis engine analyzes text such as "I'm stressed because the project isn't progressing," emojis showing tired expressions, and the tone of the voice, and calculates an overall emotion score (e.g., -0.7) from this data. The output after emotion recognition is the overall emotion score.

[0386] Step 5: Storing and calculating sentiment scores

[0387] The server stores the emotion recognition results in a database and calculates the emotion scores for each data item and the total emotion score.

[0388] Specifically, the system saves the acquired emotion scores and their analysis results as records in a database, and calculates the emotion trends for each user and each message. After the emotion scores are saved, the output is the emotion scores and analysis results saved in the database.

[0389] Step 6: Alerting and Notification

[0390] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold. This emotion score is input.

[0391] Specifically, whenever the server detects an emotion score below a certain threshold (e.g., -0.3), it automatically generates and sends an alert email or push notification to the administrator. The output after generating an alert is the sent alert notification.

[0392] Step 7: Generate and view the report

[0393] The server generates periodic reports based on the sentiment scores and analysis results and provides them to the user via a dashboard or email.

[0394] Specifically, the server automatically generates monthly or weekly reports, documents sentiment trends, summarized graphs, comments, etc., and sends them to the user (e.g., manager). The output of the report generation is the provided report.

[0395] Step 8: Feedback and action

[0396] The user checks the reports and alerts provided by the server and takes appropriate action. These reports and alerts are entered.

[0397] Specifically, the manager reviews the report, looks at the team's emotion scores and trends, and schedules team meetings to provide mental health support as needed. After the measures are implemented, the output is the measures taken.

[0398] The above is a description of the specific processing steps of this system.

[0399] (Application example 2)

[0400] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] Conventional communication systems are limited to text-based data, making it difficult to accurately grasp a user's emotional state and stress level. Furthermore, virtual stores are unable to analyze a user's emotions in real time and provide appropriate product recommendations and customer support. This makes it difficult to improve the user experience and effectively manage stress.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0403] In this invention, the server includes means for collecting in-house text-based communication data, means for supplying the collected data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, means for generating and displaying periodic reports based on the sentiment score and analysis results, and means for analyzing data entered by users and suggesting products tailored to their individual emotional states. This makes it possible to grasp the emotional state of users in real time and suggest products and manage stress according to that state.

[0404] "Internal text-based communication data" refers to text data sent and received via messaging applications and communication tools used by employees within the company.

[0405] A "natural language processing engine" is a software component that analyzes collected text data and understands its meaning and sentiment.

[0406] "Sentiment analysis" is the process of determining a user's emotional state from text data and quantifying that emotion.

[0407] An "emotion score" is a numerical representation of the emotional state analyzed from text data, and is often expressed in the range of -1 to 1.

[0408] An "alert" is a warning or notification that is generated when a value falls below a set threshold, and is intended to notify users and administrators of an abnormality.

[0409] "Means for generating and displaying reports" refers to a function that periodically creates reports based on sentiment scores and analysis results, and visually presents them to users and administrators.

[0410] "User input data" refers to text messages, voice data, emojis, etc. provided by users accessing the virtual store.

[0411] "Product suggestion" means automatically recommending appropriate products and services based on the user's emotional state.

[0412] As an embodiment of this invention, we will explain the configuration of a virtual store that combines the EmoSync system and an emotion engine. This system analyzes the user's emotional state in real time and provides product suggestions and customer support based on the results. The system configuration is explained in detail below.

[0413] System Configuration

[0414] 1. Server

[0415] The server has an API interface for collecting input data (text, emoji, and voice data) from users who access the virtual store. The collected data is then subjected to sentiment analysis using a natural language processing engine.

[0416] This server includes the following main functions:

[0417] A function that stores the results of sentiment analysis in a database and calculates sentiment scores in real time.

[0418] Alerts can be generated when sentiment scores fall below a set threshold.

[0419] Generate regular reports based on sentiment scores and analysis results.

[0420] Product suggestion function based on the user's emotional state.

[0421] 2. Device (user's smartphone or head-mounted display)

[0422] Users can access the virtual store and use the interface to send text messages or input voice data. The device then sends this data to the server, which then receives and displays emotion scores, alerts, product suggestions, and more.

[0423] 3. Users

[0424] Users browse products through a virtual store interface, input text messages or voice data, and receive product suggestions and customer support based on an emotion score provided by the server.

[0425] Program processing

[0426] Hardware and Software

[0427] Hardware

[0428] Smartphone (Android, iOS)

[0429] Head-mounted displays (Oculus Rift, HoloLens)

[0430] software

[0431] EmoSync Sentiment Analysis API

[0432] Python's requests library (sending HTTP requests)

[0433] Python json library (data cleansing and sending)

[0434] nltk and textblob (preprocessing text data)

[0435] Processing steps

[0436] 1. Data collection and transmission

[0437] Users can use their smartphones or head-mounted displays to input text messages or voice data when accessing a virtual store, and this data is sent to the server via an API.

[0438] 2. Data Preprocessing

[0439] The server cleanses the received text messages and voice data and formats them into the required format.

[0440] 3. Emotion recognition

[0441] The cleaned data is then passed to a natural language processing engine for emotion recognition. For example, if a user types, "I've been feeling stressed lately," the text data is analyzed and an "emotion score" of -0.7 is calculated.

[0442] 4. Saving sentiment scores and generating alerts

[0443] The server stores the analysis results and emotion scores in a database and generates an alert if the emotion score falls below a set threshold.

[0444] 5. Product proposal

[0445] The server recommends products and services based on the user's sentiment score and notifies the user.

[0446] Examples and prompts

[0447] As a concrete example, if a user visits a virtual store and enters the text "I've been feeling stressed lately," the following prompt will be sent to the server:

[0448] "I've been feeling stressed lately"

[0449] This prompt is sent to the server, which returns an "emotion score" as the result of emotion analysis. Product suggestions based on this score are then sent to the user. For example, if the stress score is high, relaxation items are recommended; if the score is medium, trendy products are recommended; and if the score is low, luxury products are recommended.

[0450] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0451] Step 1:

[0452] A user accesses a virtual store and inputs text or voice data. This input data is used as initial data for emotion analysis. Specifically, the user inputs "I've been feeling stressed lately" via a smartphone or head-mounted display.

[0453] Step 2:

[0454] The device sends user input data to the server via the API. The input data includes prompts, voice data, emojis, etc. For example, the text "I've been feeling stressed lately" is sent.

[0455] Step 3:

[0456] The server cleanses the received input data and performs preprocessing before passing it to the natural language processing engine. Specifically, it removes unnecessary tags and symbols from text data and formats voice data and emojis into appropriate formats.

[0457] Step 4:

[0458] The server then passes the cleansed data to a natural language processing engine for emotion recognition. For example, the text "I've been feeling stressed lately" is analyzed and an emotion score (e.g., -0.7) is calculated. This score is an important indicator of the user's emotional state.

[0459] Step 5:

[0460] The server stores the emotion analysis results in a database and calculates an emotion score in real time. For example, if an emotion score of -0.7 is recorded in the database and the user's emotional state is consistently low, the data will be saved as data for future measures.

[0461] Step 6:

[0462] The server generates an alert and notifies the administrator or customer support team when the emotion score falls below a set threshold. Specifically, when the emotion score falls below -0.3, an email or push notification is sent.

[0463] Step 7:

[0464] The server generates product recommendations suited to the user based on the emotion score and analysis results. For example, if the emotion score is low, it will suggest relaxation and healing items, if the score is medium, it will suggest trendy products, and if the score is high, it will suggest luxury items.

[0465] Step 8:

[0466] The server generates periodic reports and provides them to users and administrators via a dashboard or email, including sentiment score trends, trend analysis, and suggested product information.

[0467] Step 9:

[0468] The user checks the emotion score, product suggestions, and reports provided by the server and selects an appropriate product. For example, a user who inputs "I've been feeling stressed lately" may purchase a suggested relaxation item.

[0469] Step 10:

[0470] The terminal sends the product information selected by the user to the server to complete the purchase process, and the server records the purchase information in a database for future data analysis and recommendations.

[0471] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0472] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0473] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0474] [Second embodiment]

[0475] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0476] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0477] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0478] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0479] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0480] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0481] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0482] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0483] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0484] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0485] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0486] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0487] As an embodiment of the present invention, the configuration of the EmoSync system and program processing will be described in detail below.

[0488] System Configuration

[0489] 1. Server

[0490] The server has an API interface for collecting text-based message data from internal communication tools.

[0491] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[0492] The server stores the analysis results in a database and calculates emotion scores in real time.

[0493] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[0494] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[0495] 2. Device (user's PC or smartphone)

[0496] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[0497] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0498] 3. Users (managers, human resources department)

[0499] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0500] Program processing

[0501] The program of this system performs the following main processes.

[0502] 1. Data Collection

[0503] The device collects data on messages sent by users using internal communication tools and sends it to a server via an API.

[0504] Example: When a user sends a message on Slack saying, "The project is really tough right now," that message is collected on the server.

[0505] 2. Data Preprocessing

[0506] The server cleanses the received message data and formats it into the required format.

[0507] Example: Cleaning up text data by removing unnecessary HTML tags and special characters from messages.

[0508] 3. Sentiment analysis

[0509] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0510] Example: An NLP engine analyzes the text "The project is really tough right now" and calculates a negative sentiment score (e.g., -0.5).

[0511] 4. Storing and calculating sentiment scores

[0512] The server stores the analysis results in a database and calculates an emotion score.

[0513] Example: Record the sentiment score of each message in a database and calculate the average sentiment score for each user or team.

[0514] 5. Alerting and Notification

[0515] The server generates an alert if the emotion score falls below a set threshold and sends a notification to the relevant manager or HR department.

[0516] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[0517] 6. Report Generation and Viewing

[0518] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[0519] Example: Create a monthly report containing company-wide and departmental sentiment scores and trend graphs and provide it to managers.

[0520] 7. Feedback and Actions

[0521] Users check the reports and alerts provided by the server, consider and implement the necessary measures.

[0522] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[0523] In this way, the EmoSync system monitors the emotional climate within an organization in real time and takes appropriate measures at the right time to improve the work environment.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] The device receives text messages sent by the user using internal communication tools (e.g., messaging apps, email systems).

[0527] Example: An employee sends a message on a messaging app saying, "I'm stressed because the project isn't progressing."

[0528] Step 2:

[0529] The terminal transmits the acquired text message to the server.

[0530] Example: Sending text message data to a server through a messaging app's API.

[0531] Step 3:

[0532] The server receives the text message sent from the terminal and temporarily stores it in a database.

[0533] Example: Recording received messages in a database and preparing them for cleansing processing.

[0534] Step 4:

[0535] The server cleans the received message data, removing unnecessary HTML tags and special characters.

[0536] Example: Removing redundant tags and symbols from the text "I'm stressed because the project isn't progressing."

[0537] Step 5:

[0538] The server passes the cleaned text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0539] Example: An NLP engine analyzes the text "I'm stressed because the project isn't progressing" and calculates a sentiment score (e.g., -0.6).

[0540] Step 6:

[0541] The server stores the calculated emotion scores in a database.

[0542] Example: Analysis results and sentiment scores are recorded in a database for each user and message.

[0543] Step 7:

[0544] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[0545] Example: Aggregate each user's sentiment score from a week's worth of message data and calculate the average value.

[0546] Step 8:

[0547] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[0548] Example: Alert if a team's sentiment score falls below -0.3.

[0549] Step 9:

[0550] The server notifies the generated alerts to the HR department and relevant managers.

[0551] Example: Sending alert messages to HR and managers via email and push notifications.

[0552] Step 10:

[0553] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[0554] Example: Create and distribute a monthly report containing sentiment scores and trend graphs for each department.

[0555] Step 11:

[0556] The terminal receives reports and alerts sent from the server and displays them to the user.

[0557] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[0558] Step 12:

[0559] Users check the alerts and reports provided by the server and take appropriate action.

[0560] Example: HR receives an alert and provides mental health counseling to a specific employee.

[0561] In this way, the EmoSync system can monitor employees' emotional states in real time, detect problems early, and take countermeasures.

[0562] Example 1

[0563] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0564] Conventional internal communication tools do not analyze the emotional content of messages, making it difficult to provide the feedback and early action needed to improve the work environment. The accumulation of negative emotions, in particular, can lead to lower employee morale, poor performance, and even turnover. Furthermore, the lack of reports based on concrete data makes it difficult for managers to implement effective countermeasures.

[0565] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0566] In this invention, the server includes means for collecting in-house text-based communication data, means for data cleansing the collected data and formatting it into a required format, means for supplying the formatted data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, and means for generating and displaying periodic reports based on the sentiment score and analysis results. This makes it possible to detect emotions from in-house communications in real time and take early countermeasures.

[0567] "Internal text-based communication data" refers to textual data generated from messaging tools and emails used within the company.

[0568] "Data cleansing" is the process of removing unnecessary information and noise from collected data and preparing it in a format suitable for analysis.

[0569] A "natural language processing engine" is a software system that includes algorithms and techniques for understanding, analyzing, and generating human language.

[0570] "Sentiment analysis" is the process of identifying positive, negative, and neutral sentiments from text data and calculating their scores.

[0571] An "emotion score" is data that quantifies the strength and type of emotion contained in specific text data.

[0572] A "threshold" is a reference value for determining whether a particular indicator or value exceeds a certain standard.

[0573] An "alert" is a notification of a warning or caution that is generated when a pre-defined condition is met.

[0574] A "report" is a document or digital information that compiles analytical results and statistical data and provides them to users.

[0575] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0576] System Configuration

[0577] 1. Server

[0578] The server has an API interface for collecting internal text-based communication data. The server has a built-in natural language processing engine and performs sentiment analysis on the collected data. The sentiment analysis results are stored in a database and a sentiment score is calculated. The server monitors the set threshold in real time and has the ability to generate alerts and send notifications if the sentiment score falls below a certain level. It also generates regular reports and provides them to users via an accessible dashboard or by email.

[0579] 2. Device (user's PC or smartphone)

[0580] The terminal has an interface that allows users to send text messages using an internal communication tool, and has the function of receiving emotion scores, alerts, and reports sent from the server and displaying them to the user.

[0581] 3. Users (managers, human resources department)

[0582] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0583] Program processing

[0584] The program of this system operates around three parties: the server, the terminal, and the user.

[0585] Server: AWS or a similar cloud infrastructure is used, and the program is implemented in Python or JavaScript. For natural language processing, the Google Cloud Natural Language API or engine is used as the NLP engine. MySQL or PostgreSQL is used as the database.

[0586] Device: Runs on the user's PC or smartphone. It uses a web browser to communicate with the server via API and displays sentiment scores, alerts, and reports in real time.

[0587] Users: Users monitor the emotional state of their organization through the provided dashboards and email reports and consider necessary countermeasures.

[0588] Specific examples

[0589] Data Collection and Cleansing

[0590] The device collects messages sent by users via internal communication tools and sends them to the server. For example, if a user sends a message on Slack saying, "The project is really tough right now," the message is sent to the server via Slack's API. The server cleanses the received message data, removing HTML tags and special characters, and formats it into the required format.

[0591] sentiment analysis

[0592] The server passes the cleansed text data to an NLP engine for sentiment analysis. For example, the text "The current project is really tough" is analyzed as a negative sentiment score of -0.5. The server stores these scores in a database and calculates average sentiment scores for each user or team.

[0593] Alerting and Notifications

[0594] The server generates an alert when the sentiment score falls below a set threshold and sends a notification to the relevant manager or HR department. For example, if a team's sentiment score falls below -0.3, an alert is sent to the HR department via email or push notification.

[0595] Report Generation and Viewing

[0596] The server generates regular reports based on the emotion scores and analysis results and provides them to users via dashboards or email. For example, at the end of each month, a report showing the emotion scores and trends for the entire company or department is created and provided to managers.

[0597] Feedback and Actions

[0598] Users can check reports and alerts provided by the server, consider and implement necessary measures, etc. For example, if a manager notices a decline in the team's emotional score, they can set up an online meeting to check in with team members on their mental health.

[0599] Prompt Sentence Examples

[0600] 1. "Write a program that collects Slack messages and sends them to a server."

[0601] 2. "Write a Python program to cleanse text data."

[0602] 3. "Implement an NLP engine for sentiment analysis of text data."

[0603] 4. "Write a SQL query to store the sentiment scores and perform statistical processing."

[0604] 5. "Write a program that generates an alert and sends a notification when a specific condition occurs."

[0605] 6. "Write a program that generates a report of emotion scores."

[0606] 7. "Create guidelines for feedback and action planning based on the sentiment score report."

[0607] In this way, the EmoSync system monitors the emotional state within an organization in real time and takes appropriate measures in a timely manner to improve the work environment.

[0608] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0609] Step 1: Data collection

[0610] The device collects text messages sent by users using internal communication tools and sends them to a server via API. For example, if a user sends a message on Slack saying, "The project is really tough right now," that message is forwarded to the server via the Slack API. The input of this step is the user's message, and the output is the raw text data sent to the server.

[0611] Step 2: Data Preprocessing

[0612] The server cleanses the received message data, removes unnecessary HTML tags and special characters, and formats it into the required format. This project is really tough " into clean text "The current project is really tough." The input of this step is raw text data and the output is cleansed text data.

[0613] Step 3: Sentiment analysis

[0614] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis. Specifically, the NLP engine calculates a negative sentiment score (e.g., -0.5) for the text "This project is really tough." The input for this step is the cleansed text data, and the output is the sentiment score for the text.

[0615] Step 4: Storing and calculating sentiment scores

[0616] The server stores the analysis results in a database, records the sentiment score for each message, and calculates the average sentiment score for each user or team. Specifically, the server adds the score for each message to the database and calculates the average score for each user or team. The input for this step is the sentiment score, and the output is the score stored in the database and the calculated average score.

[0617] Step 5: Alerting and Notification

[0618] The server generates an alert when the emotion score falls below the set threshold and sends a notification to the relevant manager or HR department. For example, if the team's average emotion score falls below -0.3, an alert is sent to the HR department via email or push notification. The input of this step is the emotion score and the set threshold, and the output is the generated alert notification.

[0619] Step 6: Generate and view the report

[0620] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via dashboards or email. Specific operations include generating monthly reports, creating PDFs containing company-wide and department-specific sentiment scores and graphs showing trends, and distributing these to each manager via email. The input to this step is the sentiment scores and analysis results, and the output is the generated report.

[0621] Step 7: Feedback and action

[0622] The user checks the reports and alerts provided by the server, considers and implements the necessary response. For example, a manager may notice a decline in their team's emotional score and schedule an online meeting to conduct a mental health check-in. The input for this step is the report or alert, and the output is the user's response or feedback.

[0623] (Application example 1)

[0624] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0625] In traditional brick-and-mortar stores, it was difficult to grasp customer satisfaction and emotional state in real time, making it difficult to respond quickly to directly improve the customer experience. Furthermore, there was a lack of means to effectively collect customer feedback and use it to improve store operations. This resulted in delays in taking concrete measures to prevent declines in customer satisfaction and customer defections.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0627] In this invention, the server includes a means for collecting video data for capturing customer behavior and facial expressions, a means for preprocessing the collected video data using an image processing library, and a means for supplying the preprocessed data to emotion recognition software to perform emotion analysis, thereby enabling real-time analysis of customer emotions and prompt response.

[0628] "Video data" is visual information that captures the customer's behavior and facial expressions.

[0629] An "image processing library" is a software tool for preprocessing collected video data.

[0630] "Emotion recognition software" is a program for analyzing customer emotions from pre-processed video data.

[0631] An "emotion score" is a numerical evaluation value of an emotion calculated by emotion recognition software.

[0632] An "alert" is a warning notification that is generated when an emotion score falls below a set threshold.

[0633] "Reports" are regular reports summarizing sentiment scores and analysis results.

[0634] "Data preprocessing" is the process of removing unnecessary information from collected video data and preparing it in a format suitable for analysis.

[0635] A "natural language processing engine" is a program that analyzes text feedback entered by customers and calculates an emotional score.

[0636] A "dashboard" is a web application interface for visually displaying sentiment scores and analysis results.

[0637] "Trend analysis" is the process of tracking fluctuations in sentiment scores and analyzing patterns over time.

[0638] As an embodiment of the present invention, the configuration of the CustomerEmoSync system and program processing will be described in detail below.

[0639] System Configuration

[0640] 1. Server

[0641] The server has a means for collecting video data to capture customer behavior and facial expressions.

[0642] The server uses an image processing library (e.g., OpenCV) to preprocess the collected video data.

[0643] The server uses emotion recognition software (e.g., Microsoft Azure Face API) to perform customer emotion analysis from the pre-processed video data.

[0644] The server stores the results of the sentiment analysis and has a means for calculating sentiment scores.

[0645] The server has a means to generate an alert and notify if the emotion score falls below a set threshold.

[0646] The server has the means to generate and display periodic reports based on the sentiment scores and analysis results.

[0647] 2. Device (user's smartphone)

[0648] The terminal has an interface for the customer to input a feedback message.

[0649] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0650] 3. Users (store managers and staff)

[0651] Users check the reports and alerts provided by the server and take the necessary measures.

[0652] Program processing

[0653] 1. Data Collection

[0654] The video data collected by the camera is transmitted to a server.

[0655] Text feedback entered by the customer using their smartphone is also sent to the server.

[0656] 2. Data Preprocessing

[0657] The server uses image processing libraries such as OpenCV to cleanse the collected video data and format it into the required format.

[0658] Customer text feedback is also passed to a natural language processing engine for cleansing.

[0659] 3. Sentiment analysis

[0660] The server passes the pre-processed video data to emotion recognition software such as Microsoft Azure Face API to perform sentiment analysis.

[0661] A natural language processing engine is also used to perform sentiment analysis of the text feedback.

[0662] Specific examples

[0663] For example, if a camera captures a customer's facial expression in a store and recognizes that the customer is happy, the emotion score will be calculated as a high positive value. Also, if a customer types feedback on their smartphone, such as "The service was great," the text will be analyzed by a natural language processing engine as a positive emotion.

[0664] An example of a prompt sentence might be, "Please perform sentiment analysis on the following text: 'I am very satisfied with your service today. Thank you.'"

[0665] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0666] Step 1:

[0667] The server collects video data from cameras in the store, which transmits customer behavior and facial expressions to the server in real time. The input is video data from the cameras, which the server receives and stores.

[0668] Step 2:

[0669] The server cleanses the collected video data using an image processing library (such as OpenCV). It removes unnecessary parts from the video data and formats it into the required format. The input is the video data collected in step 1, and the output is the video data after preprocessing.

[0670] Step 3:

[0671] The server then feeds the preprocessed video data to emotion recognition software (such as Microsoft Azure Face API) to perform emotion analysis. This software calculates an emotion score based on the customer's facial expressions. The input is the preprocessed video data from step 2, and the output is the analysis results, including the emotion score.

[0672] Step 4:

[0673] The user inputs a feedback message using a smartphone. The smartphone app collects the text data and sends it to the server. The input is the text feedback from the customer, and the output is the text data sent to the server.

[0674] Step 5:

[0675] The server uses a natural language processing engine to analyze the customer's text feedback and calculate a sentiment score from the text data. The input is the text data received in step 4, and the output is the sentiment score of the text.

[0676] Step 6:

[0677] The server integrates the emotion scores obtained from the video data and text data and stores the overall emotion score in a database. The inputs are the individual emotion scores, and the output is the integrated overall emotion score.

[0678] Step 7:

[0679] The server generates an alert when the integrated overall emotion score falls below a set threshold. The alert is sent to the administrator's terminal. The input is the integrated emotion score, and the output is an alert notification.

[0680] Step 8:

[0681] The server generates periodic reports based on the sentiment scores and analysis results. These reports are provided to users via a dashboard or email. The input is the sentiment scores and analysis results stored in the database, and the output is the periodic reports.

[0682] Step 9:

[0683] Users check the reports and alerts provided by the server and take necessary measures. For example, if customer satisfaction is low, they can implement specific improvement measures such as reviewing staff response. The input is the report or alert, and the output is the specific measures.

[0684] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0685] As an embodiment of the present invention, a configuration in which the EmoSync system and an emotion engine are combined will be described in detail.

[0686] System Configuration

[0687] 1. Server

[0688] The server has an API interface for collecting text-based message data from internal communication tools.

[0689] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[0690] The server stores the analysis results in a database and calculates emotion scores in real time.

[0691] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[0692] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[0693] The server recognizes the user's emotions through an emotion engine and has the ability to analyze input other than text data (emoticons and voice data).

[0694] 2. Device (user's PC or smartphone)

[0695] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[0696] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0697] The terminal has the function of sending input such as voice data and emojis to a server.

[0698] 3. Users (managers, human resources department)

[0699] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0700] Program processing

[0701] The program of this system performs the following main processes.

[0702] 1. Data Collection

[0703] The device collects all messages, emoticons, and voice data sent by users using internal communication tools.

[0704] Example: An employee texts on a messaging app saying, "I'm stressed because the project isn't progressing," while also using a tired-face emoji.

[0705] 2. Data Transmission

[0706] The terminal transmits the acquired text messages, emojis, and voice data to the server.

[0707] Example: Sending text messages, emoji data, and voice data to a server via a messaging app's API.

[0708] 3. Data Preprocessing

[0709] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[0710] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing" and organize the emojis and audio files that were passed through it.

[0711] 4. Emotion recognition

[0712] The server passes the cleaned text data, emojis, and voice data to the emotion engine to perform emotion recognition.

[0713] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," along with an emoji of a tired expression, and also voice data, and calculates an overall emotion score (for example, -0.7) from all elements.

[0714] 5. Storing and calculating sentiment scores

[0715] The server stores the analysis results in a database and calculates an emotion score.

[0716] Example: Analysis results and sentiment scores are recorded in a database for each user and message, and then aggregated.

[0717] 6. Alert Generation and Notification

[0718] The server generates an alert if the emotion score falls below a set threshold and notifies the relevant manager or HR department.

[0719] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[0720] 7. Report Generation and Viewing

[0721] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[0722] Example: Create a monthly report for managers that includes company-wide and departmental sentiment scores and trend graphs.

[0723] 8. Feedback and Actions

[0724] Users check the reports and alerts provided by the server and take appropriate action.

[0725] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[0726] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[0727] The processing flow will be explained below.

[0728] Step 1:

[0729] The device captures text messages, emojis, and voice data sent by users using internal communication tools (e.g., messaging apps, email systems).

[0730] Example: An employee sends a text message on a messaging app saying, "I'm stressed because the project isn't progressing," along with a tired-face emoji and, if necessary, a voice message.

[0731] Step 2:

[0732] The device sends the acquired text messages, emojis, and voice data to the server all at once.

[0733] Example: Sending text messages, emojis, and voice data to a server through a messaging app's API.

[0734] Step 3:

[0735] The server receives text messages, emojis, and voice data sent from the device and temporarily stores them in a database.

[0736] Example: Record received messages, emoji information, and audio files in a database to prepare for the cleansing process.

[0737] Step 4:

[0738] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[0739] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing," convert emoji information into a format that's easier to analyze, and convert audio data into text.

[0740] Step 5:

[0741] The server passes the cleaned text data, emoji, and voice data to an emotion engine to perform emotion recognition.

[0742] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," an emoji showing a tired expression, and voice data, and calculates an overall emotion score (e.g., -0.7) from all of these elements.

[0743] Step 6:

[0744] The server stores the emotion scores obtained as a result of the emotion recognition in a database.

[0745] Example: Analysis results and sentiment scores are recorded in a database for each user and message, allowing for future reference.

[0746] Step 7:

[0747] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[0748] Example: Aggregate each user's emotion score from a week's worth of message data, emojis, and voice data, and calculate the average value.

[0749] Step 8:

[0750] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[0751] Example: Alert if a team's sentiment score falls below -0.3.

[0752] Step 9:

[0753] The server notifies the generated alerts to the HR department and relevant managers.

[0754] Example: Sending alert messages to HR and managers via email and push notifications.

[0755] Step 10:

[0756] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[0757] Example: Create a monthly report for managers that includes a trend graph of sentiment scores for the entire company and each department.

[0758] Step 11:

[0759] The terminal receives reports and alerts sent from the server and displays them to the user.

[0760] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[0761] Step 12:

[0762] Users check the alerts and reports provided by the server and take appropriate action.

[0763] Example: HR receives an alert and provides mental health counseling to a specific employee.

[0764] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[0765] Example 2

[0766] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0767] Conventional systems only dealt with text-based communication data, which meant they had issues with insufficient understanding of emotions. Furthermore, they did not analyze emojis or voice data, making it difficult to accurately recognize users' emotions, leaving managers and HR departments with a lack of information to take appropriate action. Furthermore, there was a lack of a system that allowed managers to monitor emotion scores in real time and take prompt action.

[0768] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0769] In this invention, the server includes means for collecting text-based data, emoji, and voice data from in-house communication tools, means for supplying the collected data to a natural language processing engine and an emotion analysis engine to perform emotion recognition, means for storing the recognition results in a database and calculating individual data and overall emotion scores, means for generating an alert and notifying an administrator when the emotion score falls below a set threshold, and means for creating regular reports based on the emotion scores and data analysis results and displaying them on a dashboard or by email. This makes it possible to recognize user emotions in real time not only from text data but also from emoji and voice data, and to monitor the emotional state within an organization with greater accuracy.

[0770] "Internal communication tools" refers to digital platforms used to facilitate communication between employees.

[0771] "Text-based data" refers to communication content in the form of written information.

[0772] "Emoji" refers to small icons or pictograms used to visually express emotions or intentions.

[0773] "Audio data" refers to data that records words or sounds in digital format.

[0774] A "natural language processing engine" refers to software technology that analyzes text data and understands and processes it as natural language.

[0775] An "emotion analysis engine" refers to an analytical tool used to identify emotions and impressions from collected data.

[0776] "Emotion recognition" refers to the process of analyzing and extracting emotions from text, emoji, and voice data.

[0777] A "database" refers to a collection of information that allows data to be efficiently stored, managed, and retrieved.

[0778] "Emotion score" refers to a numerical representation of a user's emotional state based on analyzed data.

[0779] A "threshold" refers to a reference value that sets a certain condition or standard and changes operation depending on whether that standard is exceeded or fallen below.

[0780] An "alert" is a warning or notification that occurs based on set conditions.

[0781] "Administrator" means an individual or entity responsible for the oversight and management of a System.

[0782] A "dashboard" refers to a screen or interface designed to allow users to visually view a variety of data and information at a glance.

[0783] "Report" refers to a document that summarizes the results of data analysis and important information within a specific period of time.

[0784] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[0785] System Configuration

[0786] The present invention is comprised of multiple elements: a server, a terminal, and a user.

[0787] server

[0788] The server has the following functions:

[0789] 1. Data Collection

[0790] The server has an API interface for collecting text-based messages, emojis, and voice data from internal communication tools.

[0791] 2. Data Preprocessing

[0792] The server cleanses the data it receives and formats it into an analyzable format, using a natural language processing engine to remove unnecessary tags and symbols from the text, and organizes and converts emojis and audio files.

[0793] 3. Emotion recognition

[0794] The server passes the cleansed data to a sentiment analysis engine for emotion recognition. The sentiment analysis engine analyzes the text "I'm stressed because the project isn't progressing," emojis showing tired expressions, and voice data to calculate an overall sentiment score.

[0795] 4. Database storage and emotion score calculation

[0796] The server stores the analysis results in a database and calculates individual data and overall emotion scores.

[0797] 5. Alerting and Notification

[0798] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold, allowing the administrator to grasp the user's emotional state in real time.

[0799] 6. Report Generation and Viewing

[0800] The server generates regular reports based on sentiment scores and data analysis results and provides them via dashboard or email.

[0801] Terminal

[0802] The terminal has the following features:

[0803] 1. Data entry and submission

[0804] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool and transmits them to the server.

[0805] 2. Results display

[0806] The terminal receives the emotion scores, alerts, and reports sent from the server and displays them to the user.

[0807] User

[0808] Users have the following roles:

[0809] 1. Check reports and alerts

[0810] Users (usually managers and HR departments) check the reports and alerts provided by the server and take necessary measures, such as scheduling team meetings to check in on mental health.

[0811] Specific examples

[0812] For example, if an employee uses a messaging app to send a text message saying, "I'm stressed because the project isn't progressing," and also uses a tired-face emoji, the following process will take place:

[0813] 1. The device receives this message and emoji and sends it to the server.

[0814] 2. The server preprocesses the received data and passes it to the sentiment analysis engine.

[0815] 3. The sentiment analysis engine analyzes the data and calculates a sentiment score (e.g., -0.7).

[0816] 4. The server stores the emotion score in a database and generates an alert to notify the administrator if the score falls below a threshold.

[0817] 5. The server generates periodic reports and provides them to the user via dashboard or email.

[0818] Prompt Sentence Examples

[0819] Use prompt statements of the following form to feed data into the generative AI model:

[0820] Prompt statement:

[0821] Please enter the text message "I'm stressed because the project isn't progressing," along with a tired-looking emoji, and any associated audio data.

[0822] As described above, the present invention makes it possible to recognize user emotions in real time not only from text data but also from pictograms and voice data, thereby monitoring the emotional state within an organization with higher accuracy.

[0823] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0824] Step 1: Data collection

[0825] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool. This data is input.

[0826] Specifically, it uses the API of a messaging app to capture text messages such as "I'm stressed because the project isn't progressing," along with emojis of tired expressions, and related audio data. The output after data collection is the collected text data, emojis, and audio data.

[0827] Step 2: Send data

[0828] The device sends the captured text messages, emojis, and voice data to the server, where this data is entered.

[0829] Specifically, the device uses the messaging app's API to send this data to the server. The output after the data transmission is text data, emojis, and voice data sent to the server.

[0830] Step 3: Data Preprocessing

[0831] The server cleanses the received text messages, emojis, and voice data and formats them into an analyzable format. This received data is then input.

[0832] Specifically, it uses a natural language processing engine to remove unnecessary tags and symbols from text, and format and convert emoji and voice data. The output of the data preprocessing is cleansed text data, analyzable emoji, and voice data.

[0833] Step 4: Emotion Recognition

[0834] The server passes the formatted text data, emojis, and voice data to an emotion analysis engine to perform emotion recognition.

[0835] Specifically, the emotion analysis engine analyzes text such as "I'm stressed because the project isn't progressing," emojis showing tired expressions, and the tone of the voice, and calculates an overall emotion score (e.g., -0.7) from this data. The output after emotion recognition is the overall emotion score.

[0836] Step 5: Storing and calculating sentiment scores

[0837] The server stores the emotion recognition results in a database and calculates the emotion scores for each data item and the total emotion score.

[0838] Specifically, the system saves the acquired emotion scores and their analysis results as records in a database, and calculates the emotion trends for each user and each message. After the emotion scores are saved, the output is the emotion scores and analysis results saved in the database.

[0839] Step 6: Alerting and Notification

[0840] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold. This emotion score is input.

[0841] Specifically, whenever the server detects an emotion score below a certain threshold (e.g., -0.3), it automatically generates and sends an alert email or push notification to the administrator. The output after generating an alert is the sent alert notification.

[0842] Step 7: Generate and view the report

[0843] The server generates periodic reports based on the sentiment scores and analysis results and provides them to the user via a dashboard or email.

[0844] Specifically, the server automatically generates monthly or weekly reports, documents sentiment trends, summarized graphs, comments, etc., and sends them to the user (e.g., manager). The output of the report generation is the provided report.

[0845] Step 8: Feedback and action

[0846] The user checks the reports and alerts provided by the server and takes appropriate action. These reports and alerts are entered.

[0847] Specifically, the manager reviews the report, looks at the team's emotion scores and trends, and schedules team meetings to provide mental health support as needed. After the measures are implemented, the output is the measures taken.

[0848] The above is a description of the specific processing steps of this system.

[0849] (Application example 2)

[0850] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] Conventional communication systems are limited to text-based data, making it difficult to accurately grasp a user's emotional state and stress level. Furthermore, virtual stores are unable to analyze a user's emotions in real time and provide appropriate product recommendations and customer support. This makes it difficult to improve the user experience and effectively manage stress.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0853] In this invention, the server includes means for collecting in-house text-based communication data, means for supplying the collected data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, means for generating and displaying periodic reports based on the sentiment score and analysis results, and means for analyzing data entered by users and suggesting products tailored to their individual emotional states. This makes it possible to grasp the emotional state of users in real time and suggest products and manage stress according to that state.

[0854] "Internal text-based communication data" refers to text data sent and received via messaging applications and communication tools used by employees within the company.

[0855] A "natural language processing engine" is a software component that analyzes collected text data and understands its meaning and sentiment.

[0856] "Sentiment analysis" is the process of determining a user's emotional state from text data and quantifying that emotion.

[0857] An "emotion score" is a numerical representation of the emotional state analyzed from text data, and is often expressed in the range of -1 to 1.

[0858] An "alert" is a warning or notification that is generated when a value falls below a set threshold, and is intended to notify users and administrators of an abnormality.

[0859] "Means for generating and displaying reports" refers to a function that periodically creates reports based on sentiment scores and analysis results, and visually presents them to users and administrators.

[0860] "User input data" refers to text messages, voice data, emojis, etc. provided by users accessing the virtual store.

[0861] "Product suggestion" means automatically recommending appropriate products and services based on the user's emotional state.

[0862] As an embodiment of this invention, we will explain the configuration of a virtual store that combines the EmoSync system and an emotion engine. This system analyzes the user's emotional state in real time and provides product suggestions and customer support based on the results. The system configuration is explained in detail below.

[0863] System Configuration

[0864] 1. Server

[0865] The server has an API interface for collecting input data (text, emoji, and voice data) from users who access the virtual store. The collected data is then subjected to sentiment analysis using a natural language processing engine.

[0866] This server includes the following main functions:

[0867] A function that stores the results of sentiment analysis in a database and calculates sentiment scores in real time.

[0868] Alerts can be generated when sentiment scores fall below a set threshold.

[0869] Generate regular reports based on sentiment scores and analysis results.

[0870] Product suggestion function based on the user's emotional state.

[0871] 2. Device (user's smartphone or head-mounted display)

[0872] Users can access the virtual store and use the interface to send text messages or input voice data. The device then sends this data to the server, which then receives and displays emotion scores, alerts, product suggestions, and more.

[0873] 3. Users

[0874] Users browse products through a virtual store interface, input text messages or voice data, and receive product suggestions and customer support based on an emotion score provided by the server.

[0875] Program processing

[0876] Hardware and Software

[0877] Hardware

[0878] Smartphone (Android, iOS)

[0879] Head-mounted displays (Oculus Rift, HoloLens)

[0880] software

[0881] EmoSync Sentiment Analysis API

[0882] Python's requests library (sending HTTP requests)

[0883] Python json library (data cleansing and sending)

[0884] nltk and textblob (preprocessing text data)

[0885] Processing steps

[0886] 1. Data collection and transmission

[0887] Users can use their smartphones or head-mounted displays to input text messages or voice data when accessing a virtual store, and this data is sent to the server via an API.

[0888] 2. Data Preprocessing

[0889] The server cleanses the received text messages and voice data and formats them into the required format.

[0890] 3. Emotion recognition

[0891] The cleaned data is then passed to a natural language processing engine for emotion recognition. For example, if a user types, "I've been feeling stressed lately," the text data is analyzed and an "emotion score" of -0.7 is calculated.

[0892] 4. Saving sentiment scores and generating alerts

[0893] The server stores the analysis results and emotion scores in a database and generates an alert if the emotion score falls below a set threshold.

[0894] 5. Product proposal

[0895] The server recommends products and services based on the user's sentiment score and notifies the user.

[0896] Examples and prompts

[0897] As a concrete example, if a user visits a virtual store and enters the text "I've been feeling stressed lately," the following prompt will be sent to the server:

[0898] "I've been feeling stressed lately"

[0899] This prompt is sent to the server, which returns an "emotion score" as the result of emotion analysis. Product suggestions based on this score are then sent to the user. For example, if the stress score is high, relaxation items are recommended; if the score is medium, trendy products are recommended; and if the score is low, luxury products are recommended.

[0900] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0901] Step 1:

[0902] A user accesses a virtual store and inputs text or voice data. This input data is used as initial data for emotion analysis. Specifically, the user inputs "I've been feeling stressed lately" via a smartphone or head-mounted display.

[0903] Step 2:

[0904] The device sends user input data to the server via the API. The input data includes prompts, voice data, emojis, etc. For example, the text "I've been feeling stressed lately" is sent.

[0905] Step 3:

[0906] The server cleanses the received input data and performs preprocessing before passing it to the natural language processing engine. Specifically, it removes unnecessary tags and symbols from text data and formats voice data and emojis into appropriate formats.

[0907] Step 4:

[0908] The server then passes the cleansed data to a natural language processing engine for emotion recognition. For example, the text "I've been feeling stressed lately" is analyzed and an emotion score (e.g., -0.7) is calculated. This score is an important indicator of the user's emotional state.

[0909] Step 5:

[0910] The server stores the emotion analysis results in a database and calculates an emotion score in real time. For example, if an emotion score of -0.7 is recorded in the database and the user's emotional state is consistently low, the data will be saved as data for future measures.

[0911] Step 6:

[0912] The server generates an alert and notifies the administrator or customer support team when the emotion score falls below a set threshold. Specifically, when the emotion score falls below -0.3, an email or push notification is sent.

[0913] Step 7:

[0914] The server generates product recommendations suited to the user based on the emotion score and analysis results. For example, if the emotion score is low, it will suggest relaxation and healing items, if the score is medium, it will suggest trendy products, and if the score is high, it will suggest luxury items.

[0915] Step 8:

[0916] The server generates periodic reports and provides them to users and administrators via a dashboard or email, including sentiment score trends, trend analysis, and suggested product information.

[0917] Step 9:

[0918] The user checks the emotion score, product suggestions, and reports provided by the server and selects an appropriate product. For example, a user who inputs "I've been feeling stressed lately" may purchase a suggested relaxation item.

[0919] Step 10:

[0920] The terminal sends the product information selected by the user to the server to complete the purchase process, and the server records the purchase information in a database for future data analysis and recommendations.

[0921] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0922] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0923] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0924] [Third embodiment]

[0925] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0926] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0927] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0928] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0929] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0930] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0931] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0932] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0933] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0934] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0935] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0936] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0937] As an embodiment of the present invention, the configuration of the EmoSync system and program processing will be described in detail below.

[0938] System Configuration

[0939] 1. Server

[0940] The server has an API interface for collecting text-based message data from internal communication tools.

[0941] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[0942] The server stores the analysis results in a database and calculates emotion scores in real time.

[0943] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[0944] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[0945] 2. Device (user's PC or smartphone)

[0946] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[0947] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[0948] 3. Users (managers, human resources department)

[0949] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[0950] Program processing

[0951] The program of this system performs the following main processes.

[0952] 1. Data Collection

[0953] The device collects data on messages sent by users using internal communication tools and sends it to a server via an API.

[0954] Example: When a user sends a message on Slack saying, "The project is really tough right now," that message is collected on the server.

[0955] 2. Data Preprocessing

[0956] The server cleanses the received message data and formats it into the required format.

[0957] Example: Cleaning up text data by removing unnecessary HTML tags and special characters from messages.

[0958] 3. Sentiment analysis

[0959] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0960] Example: An NLP engine analyzes the text "The project is really tough right now" and calculates a negative sentiment score (e.g., -0.5).

[0961] 4. Storing and calculating sentiment scores

[0962] The server stores the analysis results in a database and calculates an emotion score.

[0963] Example: Record the sentiment score of each message in a database and calculate the average sentiment score for each user or team.

[0964] 5. Alerting and Notification

[0965] The server generates an alert if the emotion score falls below a set threshold and sends a notification to the relevant manager or HR department.

[0966] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[0967] 6. Report Generation and Viewing

[0968] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[0969] Example: Create a monthly report containing company-wide and departmental sentiment scores and trend graphs and provide it to managers.

[0970] 7. Feedback and Actions

[0971] Users check the reports and alerts provided by the server, consider and implement the necessary measures.

[0972] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[0973] In this way, the EmoSync system monitors the emotional climate within an organization in real time and takes appropriate measures at the right time to improve the work environment.

[0974] The processing flow will be explained below.

[0975] Step 1:

[0976] The device receives text messages sent by the user using internal communication tools (e.g., messaging apps, email systems).

[0977] Example: An employee sends a message on a messaging app saying, "I'm stressed because the project isn't progressing."

[0978] Step 2:

[0979] The terminal transmits the acquired text message to the server.

[0980] Example: Sending text message data to a server through a messaging app's API.

[0981] Step 3:

[0982] The server receives the text message sent from the terminal and temporarily stores it in a database.

[0983] Example: Recording received messages in a database and preparing them for cleansing processing.

[0984] Step 4:

[0985] The server cleans the received message data, removing unnecessary HTML tags and special characters.

[0986] Example: Removing redundant tags and symbols from the text "I'm stressed because the project isn't progressing."

[0987] Step 5:

[0988] The server passes the cleaned text data to a natural language processing (NLP) engine to perform sentiment analysis.

[0989] Example: An NLP engine analyzes the text "I'm stressed because the project isn't progressing" and calculates a sentiment score (e.g., -0.6).

[0990] Step 6:

[0991] The server stores the calculated emotion scores in a database.

[0992] Example: Analysis results and sentiment scores are recorded in a database for each user and message.

[0993] Step 7:

[0994] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[0995] Example: Aggregate each user's sentiment score from a week's worth of message data and calculate the average value.

[0996] Step 8:

[0997] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[0998] Example: Alert if a team's sentiment score falls below -0.3.

[0999] Step 9:

[1000] The server notifies the generated alerts to the HR department and relevant managers.

[1001] Example: Sending alert messages to HR and managers via email and push notifications.

[1002] Step 10:

[1003] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[1004] Example: Create and distribute a monthly report containing sentiment scores and trend graphs for each department.

[1005] Step 11:

[1006] The terminal receives reports and alerts sent from the server and displays them to the user.

[1007] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[1008] Step 12:

[1009] Users check the alerts and reports provided by the server and take appropriate action.

[1010] Example: HR receives an alert and provides mental health counseling to a specific employee.

[1011] In this way, the EmoSync system can monitor employees' emotional states in real time, detect problems early, and take countermeasures.

[1012] Example 1

[1013] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1014] Conventional internal communication tools do not analyze the emotional content of messages, making it difficult to provide the feedback and early action needed to improve the work environment. The accumulation of negative emotions, in particular, can lead to lower employee morale, poor performance, and even turnover. Furthermore, the lack of reports based on concrete data makes it difficult for managers to implement effective countermeasures.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1016] In this invention, the server includes means for collecting in-house text-based communication data, means for data cleansing the collected data and formatting it into a required format, means for supplying the formatted data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, and means for generating and displaying periodic reports based on the sentiment score and analysis results. This makes it possible to detect emotions from in-house communications in real time and take early countermeasures.

[1017] "Internal text-based communication data" refers to textual data generated from messaging tools and emails used within the company.

[1018] "Data cleansing" is the process of removing unnecessary information and noise from collected data and preparing it in a format suitable for analysis.

[1019] A "natural language processing engine" is a software system that includes algorithms and techniques for understanding, analyzing, and generating human language.

[1020] "Sentiment analysis" is the process of identifying positive, negative, and neutral sentiments from text data and calculating their scores.

[1021] An "emotion score" is data that quantifies the strength and type of emotion contained in specific text data.

[1022] A "threshold" is a reference value for determining whether a particular indicator or value exceeds a certain standard.

[1023] An "alert" is a notification of a warning or caution that is generated when a pre-defined condition is met.

[1024] A "report" is a document or digital information that compiles analytical results and statistical data and provides them to users.

[1025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1026] System Configuration

[1027] 1. Server

[1028] The server has an API interface for collecting internal text-based communication data. The server has a built-in natural language processing engine and performs sentiment analysis on the collected data. The sentiment analysis results are stored in a database and a sentiment score is calculated. The server monitors the set threshold in real time and has the ability to generate alerts and send notifications if the sentiment score falls below a certain level. It also generates regular reports and provides them to users via an accessible dashboard or by email.

[1029] 2. Device (user's PC or smartphone)

[1030] The terminal has an interface that allows users to send text messages using an internal communication tool, and has the function of receiving emotion scores, alerts, and reports sent from the server and displaying them to the user.

[1031] 3. Users (managers, human resources department)

[1032] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[1033] Program processing

[1034] The program of this system operates around three parties: the server, the terminal, and the user.

[1035] Server: AWS or a similar cloud infrastructure is used, and the program is implemented in Python or JavaScript. For natural language processing, the Google Cloud Natural Language API or engine is used as the NLP engine. MySQL or PostgreSQL is used as the database.

[1036] Device: Runs on the user's PC or smartphone. It uses a web browser to communicate with the server via API and displays sentiment scores, alerts, and reports in real time.

[1037] Users: Users monitor the emotional state of their organization through the provided dashboards and email reports and consider necessary countermeasures.

[1038] Specific examples

[1039] Data Collection and Cleansing

[1040] The device collects messages sent by users via internal communication tools and sends them to the server. For example, if a user sends a message on Slack saying, "The project is really tough right now," the message is sent to the server via Slack's API. The server cleanses the received message data, removing HTML tags and special characters, and formats it into the required format.

[1041] sentiment analysis

[1042] The server passes the cleansed text data to an NLP engine for sentiment analysis. For example, the text "The current project is really tough" is analyzed as a negative sentiment score of -0.5. The server stores these scores in a database and calculates average sentiment scores for each user or team.

[1043] Alerting and Notifications

[1044] The server generates an alert when the sentiment score falls below a set threshold and sends a notification to the relevant manager or HR department. For example, if a team's sentiment score falls below -0.3, an alert is sent to the HR department via email or push notification.

[1045] Report Generation and Viewing

[1046] The server generates regular reports based on the emotion scores and analysis results and provides them to users via dashboards or email. For example, at the end of each month, a report showing the emotion scores and trends for the entire company or department is created and provided to managers.

[1047] Feedback and Actions

[1048] Users can check reports and alerts provided by the server, consider and implement necessary measures, etc. For example, if a manager notices a decline in the team's emotional score, they can set up an online meeting to check in with team members on their mental health.

[1049] Prompt Sentence Examples

[1050] 1. "Write a program that collects Slack messages and sends them to a server."

[1051] 2. "Write a Python program to cleanse text data."

[1052] 3. "Implement an NLP engine for sentiment analysis of text data."

[1053] 4. "Write a SQL query to store the sentiment scores and perform statistical processing."

[1054] 5. "Write a program that generates an alert and sends a notification when a specific condition occurs."

[1055] 6. "Write a program that generates a report of emotion scores."

[1056] 7. "Create guidelines for feedback and action planning based on the sentiment score report."

[1057] In this way, the EmoSync system monitors the emotional state within an organization in real time and takes appropriate measures in a timely manner to improve the work environment.

[1058] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1059] Step 1: Data collection

[1060] The device collects text messages sent by users using internal communication tools and sends them to a server via API. For example, if a user sends a message on Slack saying, "The project is really tough right now," that message is forwarded to the server via the Slack API. The input of this step is the user's message, and the output is the raw text data sent to the server.

[1061] Step 2: Data Preprocessing

[1062] The server cleanses the received message data, removes unnecessary HTML tags and special characters, and formats it into the required format. This project is really tough " into clean text "The current project is really tough." The input of this step is raw text data and the output is cleansed text data.

[1063] Step 3: Sentiment analysis

[1064] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis. Specifically, the NLP engine calculates a negative sentiment score (e.g., -0.5) for the text "This project is really tough." The input for this step is the cleansed text data, and the output is the sentiment score for the text.

[1065] Step 4: Storing and calculating sentiment scores

[1066] The server stores the analysis results in a database, records the sentiment score for each message, and calculates the average sentiment score for each user or team. Specifically, the server adds the score for each message to the database and calculates the average score for each user or team. The input for this step is the sentiment score, and the output is the score stored in the database and the calculated average score.

[1067] Step 5: Alerting and Notification

[1068] The server generates an alert when the emotion score falls below the set threshold and sends a notification to the relevant manager or HR department. For example, if the team's average emotion score falls below -0.3, an alert is sent to the HR department via email or push notification. The input of this step is the emotion score and the set threshold, and the output is the generated alert notification.

[1069] Step 6: Generate and view the report

[1070] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via dashboards or email. Specific operations include generating monthly reports, creating PDFs containing company-wide and department-specific sentiment scores and graphs showing trends, and distributing these to each manager via email. The input to this step is the sentiment scores and analysis results, and the output is the generated report.

[1071] Step 7: Feedback and action

[1072] The user checks the reports and alerts provided by the server, considers and implements the necessary response. For example, a manager may notice a decline in their team's emotional score and schedule an online meeting to conduct a mental health check-in. The input for this step is the report or alert, and the output is the user's response or feedback.

[1073] (Application example 1)

[1074] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] In traditional brick-and-mortar stores, it was difficult to grasp customer satisfaction and emotional state in real time, making it difficult to respond quickly to directly improve the customer experience. Furthermore, there was a lack of means to effectively collect customer feedback and use it to improve store operations. This resulted in delays in taking concrete measures to prevent declines in customer satisfaction and customer defections.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1077] In this invention, the server includes a means for collecting video data for capturing customer behavior and facial expressions, a means for preprocessing the collected video data using an image processing library, and a means for supplying the preprocessed data to emotion recognition software to perform emotion analysis, thereby enabling real-time analysis of customer emotions and prompt response.

[1078] "Video data" is visual information that captures the customer's behavior and facial expressions.

[1079] An "image processing library" is a software tool for preprocessing collected video data.

[1080] "Emotion recognition software" is a program for analyzing customer emotions from pre-processed video data.

[1081] An "emotion score" is a numerical evaluation value of an emotion calculated by emotion recognition software.

[1082] An "alert" is a warning notification that is generated when an emotion score falls below a set threshold.

[1083] "Reports" are regular reports summarizing sentiment scores and analysis results.

[1084] "Data preprocessing" is the process of removing unnecessary information from collected video data and preparing it in a format suitable for analysis.

[1085] A "natural language processing engine" is a program that analyzes text feedback entered by customers and calculates an emotional score.

[1086] A "dashboard" is a web application interface for visually displaying sentiment scores and analysis results.

[1087] "Trend analysis" is the process of tracking fluctuations in sentiment scores and analyzing patterns over time.

[1088] As an embodiment of the present invention, the configuration of the CustomerEmoSync system and program processing will be described in detail below.

[1089] System Configuration

[1090] 1. Server

[1091] The server has a means for collecting video data to capture customer behavior and facial expressions.

[1092] The server uses an image processing library (e.g., OpenCV) to preprocess the collected video data.

[1093] The server uses emotion recognition software (e.g., Microsoft Azure Face API) to perform customer emotion analysis from the pre-processed video data.

[1094] The server stores the results of the sentiment analysis and has a means for calculating sentiment scores.

[1095] The server has a means to generate an alert and notify if the emotion score falls below a set threshold.

[1096] The server has the means to generate and display periodic reports based on the sentiment scores and analysis results.

[1097] 2. Device (user's smartphone)

[1098] The terminal has an interface for the customer to input a feedback message.

[1099] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[1100] 3. Users (store managers and staff)

[1101] Users check the reports and alerts provided by the server and take the necessary measures.

[1102] Program processing

[1103] 1. Data Collection

[1104] The video data collected by the camera is transmitted to a server.

[1105] Text feedback entered by the customer using their smartphone is also sent to the server.

[1106] 2. Data Preprocessing

[1107] The server uses image processing libraries such as OpenCV to cleanse the collected video data and format it into the required format.

[1108] Customer text feedback is also passed to a natural language processing engine for cleansing.

[1109] 3. Sentiment analysis

[1110] The server passes the pre-processed video data to emotion recognition software such as Microsoft Azure Face API to perform sentiment analysis.

[1111] A natural language processing engine is also used to perform sentiment analysis of the text feedback.

[1112] Specific examples

[1113] For example, if a camera captures a customer's facial expression in a store and recognizes that the customer is happy, the emotion score will be calculated as a high positive value. Also, if a customer types feedback on their smartphone, such as "The service was great," the text will be analyzed by a natural language processing engine as a positive emotion.

[1114] An example of a prompt sentence might be, "Please perform sentiment analysis on the following text: 'I am very satisfied with your service today. Thank you.'"

[1115] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1116] Step 1:

[1117] The server collects video data from cameras in the store, which transmits customer behavior and facial expressions to the server in real time. The input is video data from the cameras, which the server receives and stores.

[1118] Step 2:

[1119] The server cleanses the collected video data using an image processing library (such as OpenCV). It removes unnecessary parts from the video data and formats it into the required format. The input is the video data collected in step 1, and the output is the video data after preprocessing.

[1120] Step 3:

[1121] The server then feeds the preprocessed video data to emotion recognition software (such as Microsoft Azure Face API) to perform emotion analysis. This software calculates an emotion score based on the customer's facial expressions. The input is the preprocessed video data from step 2, and the output is the analysis results, including the emotion score.

[1122] Step 4:

[1123] The user inputs a feedback message using a smartphone. The smartphone app collects the text data and sends it to the server. The input is the text feedback from the customer, and the output is the text data sent to the server.

[1124] Step 5:

[1125] The server uses a natural language processing engine to analyze the customer's text feedback and calculate a sentiment score from the text data. The input is the text data received in step 4, and the output is the sentiment score of the text.

[1126] Step 6:

[1127] The server integrates the emotion scores obtained from the video data and text data and stores the overall emotion score in a database. The inputs are the individual emotion scores, and the output is the integrated overall emotion score.

[1128] Step 7:

[1129] The server generates an alert when the integrated overall emotion score falls below a set threshold. The alert is sent to the administrator's terminal. The input is the integrated emotion score, and the output is an alert notification.

[1130] Step 8:

[1131] The server generates periodic reports based on the sentiment scores and analysis results. These reports are provided to users via a dashboard or email. The input is the sentiment scores and analysis results stored in the database, and the output is the periodic reports.

[1132] Step 9:

[1133] Users check the reports and alerts provided by the server and take necessary measures. For example, if customer satisfaction is low, they can implement specific improvement measures such as reviewing staff response. The input is the report or alert, and the output is the specific measures.

[1134] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1135] As an embodiment of the present invention, a configuration in which the EmoSync system and an emotion engine are combined will be described in detail.

[1136] System Configuration

[1137] 1. Server

[1138] The server has an API interface for collecting text-based message data from internal communication tools.

[1139] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[1140] The server stores the analysis results in a database and calculates emotion scores in real time.

[1141] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[1142] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[1143] The server recognizes the user's emotions through an emotion engine and has the ability to analyze input other than text data (emoticons and voice data).

[1144] 2. Device (user's PC or smartphone)

[1145] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[1146] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[1147] The terminal has the function of sending input such as voice data and emojis to a server.

[1148] 3. Users (managers, human resources department)

[1149] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[1150] Program processing

[1151] The program of this system performs the following main processes.

[1152] 1. Data Collection

[1153] The device collects all messages, emoticons, and voice data sent by users using internal communication tools.

[1154] Example: An employee texts on a messaging app saying, "I'm stressed because the project isn't progressing," while also using a tired-face emoji.

[1155] 2. Data Transmission

[1156] The terminal transmits the acquired text messages, emojis, and voice data to the server.

[1157] Example: Sending text messages, emoji data, and voice data to a server via a messaging app's API.

[1158] 3. Data Preprocessing

[1159] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[1160] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing" and organize the emojis and audio files that were passed through it.

[1161] 4. Emotion recognition

[1162] The server passes the cleaned text data, emojis, and voice data to the emotion engine to perform emotion recognition.

[1163] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," along with an emoji of a tired expression, and also voice data, and calculates an overall emotion score (for example, -0.7) from all elements.

[1164] 5. Storing and calculating sentiment scores

[1165] The server stores the analysis results in a database and calculates an emotion score.

[1166] Example: Analysis results and sentiment scores are recorded in a database for each user and message, and then aggregated.

[1167] 6. Alert Generation and Notification

[1168] The server generates an alert if the emotion score falls below a set threshold and notifies the relevant manager or HR department.

[1169] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[1170] 7. Report Generation and Viewing

[1171] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[1172] Example: Create a monthly report for managers that includes company-wide and departmental sentiment scores and trend graphs.

[1173] 8. Feedback and Actions

[1174] Users check the reports and alerts provided by the server and take appropriate action.

[1175] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[1176] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[1177] The processing flow will be explained below.

[1178] Step 1:

[1179] The device captures text messages, emojis, and voice data sent by users using internal communication tools (e.g., messaging apps, email systems).

[1180] Example: An employee sends a text message on a messaging app saying, "I'm stressed because the project isn't progressing," along with a tired-face emoji and, if necessary, a voice message.

[1181] Step 2:

[1182] The device sends the acquired text messages, emojis, and voice data to the server all at once.

[1183] Example: Sending text messages, emojis, and voice data to a server through a messaging app's API.

[1184] Step 3:

[1185] The server receives text messages, emojis, and voice data sent from the device and temporarily stores them in a database.

[1186] Example: Record received messages, emoji information, and audio files in a database to prepare for the cleansing process.

[1187] Step 4:

[1188] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[1189] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing," convert emoji information into a format that's easier to analyze, and convert audio data into text.

[1190] Step 5:

[1191] The server passes the cleaned text data, emoji, and voice data to an emotion engine to perform emotion recognition.

[1192] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," an emoji showing a tired expression, and voice data, and calculates an overall emotion score (e.g., -0.7) from all of these elements.

[1193] Step 6:

[1194] The server stores the emotion scores obtained as a result of the emotion recognition in a database.

[1195] Example: Analysis results and sentiment scores are recorded in a database for each user and message, allowing for future reference.

[1196] Step 7:

[1197] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[1198] Example: Aggregate each user's emotion score from a week's worth of message data, emojis, and voice data, and calculate the average value.

[1199] Step 8:

[1200] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[1201] Example: Alert if a team's sentiment score falls below -0.3.

[1202] Step 9:

[1203] The server notifies the generated alerts to the HR department and relevant managers.

[1204] Example: Sending alert messages to HR and managers via email and push notifications.

[1205] Step 10:

[1206] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[1207] Example: Create a monthly report for managers that includes a trend graph of sentiment scores for the entire company and each department.

[1208] Step 11:

[1209] The terminal receives reports and alerts sent from the server and displays them to the user.

[1210] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[1211] Step 12:

[1212] Users check the alerts and reports provided by the server and take appropriate action.

[1213] Example: HR receives an alert and provides mental health counseling to a specific employee.

[1214] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[1215] Example 2

[1216] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1217] Conventional systems only dealt with text-based communication data, which meant they had issues with insufficient understanding of emotions. Furthermore, they did not analyze emojis or voice data, making it difficult to accurately recognize users' emotions, leaving managers and HR departments with a lack of information to take appropriate action. Furthermore, there was a lack of a system that allowed managers to monitor emotion scores in real time and take prompt action.

[1218] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1219] In this invention, the server includes means for collecting text-based data, emoji, and voice data from in-house communication tools, means for supplying the collected data to a natural language processing engine and an emotion analysis engine to perform emotion recognition, means for storing the recognition results in a database and calculating individual data and overall emotion scores, means for generating an alert and notifying an administrator when the emotion score falls below a set threshold, and means for creating regular reports based on the emotion scores and data analysis results and displaying them on a dashboard or by email. This makes it possible to recognize user emotions in real time not only from text data but also from emoji and voice data, and to monitor the emotional state within an organization with greater accuracy.

[1220] "Internal communication tools" refers to digital platforms used to facilitate communication between employees.

[1221] "Text-based data" refers to communication content in the form of written information.

[1222] "Emoji" refers to small icons or pictograms used to visually express emotions or intentions.

[1223] "Audio data" refers to data that records words or sounds in digital format.

[1224] A "natural language processing engine" refers to software technology that analyzes text data and understands and processes it as natural language.

[1225] An "emotion analysis engine" refers to an analytical tool used to identify emotions and impressions from collected data.

[1226] "Emotion recognition" refers to the process of analyzing and extracting emotions from text, emoji, and voice data.

[1227] A "database" refers to a collection of information that allows data to be efficiently stored, managed, and retrieved.

[1228] "Emotion score" refers to a numerical representation of a user's emotional state based on analyzed data.

[1229] A "threshold" refers to a reference value that sets a certain condition or standard and changes operation depending on whether that standard is exceeded or fallen below.

[1230] An "alert" is a warning or notification that occurs based on set conditions.

[1231] "Administrator" means an individual or entity responsible for the oversight and management of a System.

[1232] A "dashboard" refers to a screen or interface designed to allow users to visually view a variety of data and information at a glance.

[1233] "Report" refers to a document that summarizes the results of data analysis and important information within a specific period of time.

[1234] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1235] System Configuration

[1236] The present invention is comprised of multiple elements: a server, a terminal, and a user.

[1237] server

[1238] The server has the following functions:

[1239] 1. Data Collection

[1240] The server has an API interface for collecting text-based messages, emojis, and voice data from internal communication tools.

[1241] 2. Data Preprocessing

[1242] The server cleanses the data it receives and formats it into an analyzable format, using a natural language processing engine to remove unnecessary tags and symbols from the text, and organizes and converts emojis and audio files.

[1243] 3. Emotion recognition

[1244] The server passes the cleansed data to a sentiment analysis engine for emotion recognition. The sentiment analysis engine analyzes the text "I'm stressed because the project isn't progressing," emojis showing tired expressions, and voice data to calculate an overall sentiment score.

[1245] 4. Database storage and emotion score calculation

[1246] The server stores the analysis results in a database and calculates individual data and overall emotion scores.

[1247] 5. Alerting and Notification

[1248] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold, allowing the administrator to grasp the user's emotional state in real time.

[1249] 6. Report Generation and Viewing

[1250] The server generates regular reports based on sentiment scores and data analysis results and provides them via dashboard or email.

[1251] Terminal

[1252] The terminal has the following features:

[1253] 1. Data entry and submission

[1254] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool and transmits them to the server.

[1255] 2. Results display

[1256] The terminal receives the emotion scores, alerts, and reports sent from the server and displays them to the user.

[1257] User

[1258] Users have the following roles:

[1259] 1. Check reports and alerts

[1260] Users (usually managers and HR departments) check the reports and alerts provided by the server and take necessary measures, such as scheduling team meetings to check in on mental health.

[1261] Specific examples

[1262] For example, if an employee uses a messaging app to send a text message saying, "I'm stressed because the project isn't progressing," and also uses a tired-face emoji, the following process will take place:

[1263] 1. The device receives this message and emoji and sends it to the server.

[1264] 2. The server preprocesses the received data and passes it to the sentiment analysis engine.

[1265] 3. The sentiment analysis engine analyzes the data and calculates a sentiment score (e.g., -0.7).

[1266] 4. The server stores the emotion score in a database and generates an alert to notify the administrator if the score falls below a threshold.

[1267] 5. The server generates periodic reports and provides them to the user via dashboard or email.

[1268] Prompt Sentence Examples

[1269] Use prompt statements of the following form to feed data into the generative AI model:

[1270] Prompt statement:

[1271] Please enter the text message "I'm stressed because the project isn't progressing," along with a tired-looking emoji, and any associated audio data.

[1272] As described above, the present invention makes it possible to recognize user emotions in real time not only from text data but also from pictograms and voice data, thereby monitoring the emotional state within an organization with higher accuracy.

[1273] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1274] Step 1: Data collection

[1275] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool. This data is input.

[1276] Specifically, it uses the API of a messaging app to capture text messages such as "I'm stressed because the project isn't progressing," along with emojis of tired expressions, and related audio data. The output after data collection is the collected text data, emojis, and audio data.

[1277] Step 2: Send data

[1278] The device sends the captured text messages, emojis, and voice data to the server, where this data is entered.

[1279] Specifically, the device uses the messaging app's API to send this data to the server. The output after the data transmission is text data, emojis, and voice data sent to the server.

[1280] Step 3: Data Preprocessing

[1281] The server cleanses the received text messages, emojis, and voice data and formats them into an analyzable format. This received data is then input.

[1282] Specifically, it uses a natural language processing engine to remove unnecessary tags and symbols from text, and format and convert emoji and voice data. The output of the data preprocessing is cleansed text data, analyzable emoji, and voice data.

[1283] Step 4: Emotion Recognition

[1284] The server passes the formatted text data, emojis, and voice data to an emotion analysis engine to perform emotion recognition.

[1285] Specifically, the emotion analysis engine analyzes text such as "I'm stressed because the project isn't progressing," emojis showing tired expressions, and the tone of the voice, and calculates an overall emotion score (e.g., -0.7) from this data. The output after emotion recognition is the overall emotion score.

[1286] Step 5: Storing and calculating sentiment scores

[1287] The server stores the emotion recognition results in a database and calculates the emotion scores for each data item and the total emotion score.

[1288] Specifically, the system saves the acquired emotion scores and their analysis results as records in a database, and calculates the emotion trends for each user and each message. After the emotion scores are saved, the output is the emotion scores and analysis results saved in the database.

[1289] Step 6: Alerting and Notification

[1290] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold. This emotion score is input.

[1291] Specifically, whenever the server detects an emotion score below a certain threshold (e.g., -0.3), it automatically generates and sends an alert email or push notification to the administrator. The output after generating an alert is the sent alert notification.

[1292] Step 7: Generate and view the report

[1293] The server generates periodic reports based on the sentiment scores and analysis results and provides them to the user via a dashboard or email.

[1294] Specifically, the server automatically generates monthly or weekly reports, documents sentiment trends, summarized graphs, comments, etc., and sends them to the user (e.g., manager). The output of the report generation is the provided report.

[1295] Step 8: Feedback and action

[1296] The user checks the reports and alerts provided by the server and takes appropriate action. These reports and alerts are entered.

[1297] Specifically, the manager reviews the report, looks at the team's emotion scores and trends, and schedules team meetings to provide mental health support as needed. After the measures are implemented, the output is the measures taken.

[1298] The above is a description of the specific processing steps of this system.

[1299] (Application example 2)

[1300] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1301] Conventional communication systems are limited to text-based data, making it difficult to accurately grasp a user's emotional state and stress level. Furthermore, virtual stores are unable to analyze a user's emotions in real time and provide appropriate product recommendations and customer support. This makes it difficult to improve the user experience and effectively manage stress.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1303] In this invention, the server includes means for collecting in-house text-based communication data, means for supplying the collected data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, means for generating and displaying periodic reports based on the sentiment score and analysis results, and means for analyzing data entered by users and suggesting products tailored to their individual emotional states. This makes it possible to grasp the emotional state of users in real time and suggest products and manage stress according to that state.

[1304] "Internal text-based communication data" refers to text data sent and received via messaging applications and communication tools used by employees within the company.

[1305] A "natural language processing engine" is a software component that analyzes collected text data and understands its meaning and sentiment.

[1306] "Sentiment analysis" is the process of determining a user's emotional state from text data and quantifying that emotion.

[1307] An "emotion score" is a numerical representation of the emotional state analyzed from text data, and is often expressed in the range of -1 to 1.

[1308] An "alert" is a warning or notification that is generated when a value falls below a set threshold, and is intended to notify users and administrators of an abnormality.

[1309] "Means for generating and displaying reports" refers to a function that periodically creates reports based on sentiment scores and analysis results, and visually presents them to users and administrators.

[1310] "User input data" refers to text messages, voice data, emojis, etc. provided by users accessing the virtual store.

[1311] "Product suggestion" means automatically recommending appropriate products and services based on the user's emotional state.

[1312] As an embodiment of this invention, we will explain the configuration of a virtual store that combines the EmoSync system and an emotion engine. This system analyzes the user's emotional state in real time and provides product suggestions and customer support based on the results. The system configuration is explained in detail below.

[1313] System Configuration

[1314] 1. Server

[1315] The server has an API interface for collecting input data (text, emoji, and voice data) from users who access the virtual store. The collected data is then subjected to sentiment analysis using a natural language processing engine.

[1316] This server includes the following main functions:

[1317] A function that stores the results of sentiment analysis in a database and calculates sentiment scores in real time.

[1318] Alerts can be generated when sentiment scores fall below a set threshold.

[1319] Generate regular reports based on sentiment scores and analysis results.

[1320] Product suggestion function based on the user's emotional state.

[1321] 2. Device (user's smartphone or head-mounted display)

[1322] Users can access the virtual store and use the interface to send text messages or input voice data. The device then sends this data to the server, which then receives and displays emotion scores, alerts, product suggestions, and more.

[1323] 3. Users

[1324] Users browse products through a virtual store interface, input text messages or voice data, and receive product suggestions and customer support based on an emotion score provided by the server.

[1325] Program processing

[1326] Hardware and Software

[1327] Hardware

[1328] Smartphone (Android, iOS)

[1329] Head-mounted displays (Oculus Rift, HoloLens)

[1330] software

[1331] EmoSync Sentiment Analysis API

[1332] Python's requests library (sending HTTP requests)

[1333] Python json library (data cleansing and sending)

[1334] nltk and textblob (preprocessing text data)

[1335] Processing steps

[1336] 1. Data collection and transmission

[1337] Users can use their smartphones or head-mounted displays to input text messages or voice data when accessing a virtual store, and this data is sent to the server via an API.

[1338] 2. Data Preprocessing

[1339] The server cleanses the received text messages and voice data and formats them into the required format.

[1340] 3. Emotion recognition

[1341] The cleaned data is then passed to a natural language processing engine for emotion recognition. For example, if a user types, "I've been feeling stressed lately," the text data is analyzed and an "emotion score" of -0.7 is calculated.

[1342] 4. Saving sentiment scores and generating alerts

[1343] The server stores the analysis results and emotion scores in a database and generates an alert if the emotion score falls below a set threshold.

[1344] 5. Product proposal

[1345] The server recommends products and services based on the user's sentiment score and notifies the user.

[1346] Examples and prompts

[1347] As a concrete example, if a user visits a virtual store and enters the text "I've been feeling stressed lately," the following prompt will be sent to the server:

[1348] "I've been feeling stressed lately"

[1349] This prompt is sent to the server, which returns an "emotion score" as the result of emotion analysis. Product suggestions based on this score are then sent to the user. For example, if the stress score is high, relaxation items are recommended; if the score is medium, trendy products are recommended; and if the score is low, luxury products are recommended.

[1350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1351] Step 1:

[1352] A user accesses a virtual store and inputs text or voice data. This input data is used as initial data for emotion analysis. Specifically, the user inputs "I've been feeling stressed lately" via a smartphone or head-mounted display.

[1353] Step 2:

[1354] The device sends user input data to the server via the API. The input data includes prompts, voice data, emojis, etc. For example, the text "I've been feeling stressed lately" is sent.

[1355] Step 3:

[1356] The server cleanses the received input data and performs preprocessing before passing it to the natural language processing engine. Specifically, it removes unnecessary tags and symbols from text data and formats voice data and emojis into appropriate formats.

[1357] Step 4:

[1358] The server then passes the cleansed data to a natural language processing engine for emotion recognition. For example, the text "I've been feeling stressed lately" is analyzed and an emotion score (e.g., -0.7) is calculated. This score is an important indicator of the user's emotional state.

[1359] Step 5:

[1360] The server stores the emotion analysis results in a database and calculates an emotion score in real time. For example, if an emotion score of -0.7 is recorded in the database and the user's emotional state is consistently low, the data will be saved as data for future measures.

[1361] Step 6:

[1362] The server generates an alert and notifies the administrator or customer support team when the emotion score falls below a set threshold. Specifically, when the emotion score falls below -0.3, an email or push notification is sent.

[1363] Step 7:

[1364] The server generates product recommendations suited to the user based on the emotion score and analysis results. For example, if the emotion score is low, it will suggest relaxation and healing items, if the score is medium, it will suggest trendy products, and if the score is high, it will suggest luxury items.

[1365] Step 8:

[1366] The server generates periodic reports and provides them to users and administrators via a dashboard or email, including sentiment score trends, trend analysis, and suggested product information.

[1367] Step 9:

[1368] The user checks the emotion score, product suggestions, and reports provided by the server and selects an appropriate product. For example, a user who inputs "I've been feeling stressed lately" may purchase a suggested relaxation item.

[1369] Step 10:

[1370] The terminal sends the product information selected by the user to the server to complete the purchase process, and the server records the purchase information in a database for future data analysis and recommendations.

[1371] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1372] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1373] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1374] [Fourth embodiment]

[1375] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1376] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1378] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1382] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1383] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1384] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1385] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1386] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1387] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1388] As an embodiment of the present invention, the configuration of the EmoSync system and program processing will be described in detail below.

[1389] System Configuration

[1390] 1. Server

[1391] The server has an API interface for collecting text-based message data from internal communication tools.

[1392] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[1393] The server stores the analysis results in a database and calculates emotion scores in real time.

[1394] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[1395] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[1396] 2. Device (user's PC or smartphone)

[1397] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[1398] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[1399] 3. Users (managers, human resources department)

[1400] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[1401] Program processing

[1402] The program of this system performs the following main processes.

[1403] 1. Data Collection

[1404] The device collects data on messages sent by users using internal communication tools and sends it to a server via an API.

[1405] Example: When a user sends a message on Slack saying, "The project is really tough right now," that message is collected on the server.

[1406] 2. Data Preprocessing

[1407] The server cleanses the received message data and formats it into the required format.

[1408] Example: Cleaning up text data by removing unnecessary HTML tags and special characters from messages.

[1409] 3. Sentiment analysis

[1410] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis.

[1411] Example: An NLP engine analyzes the text "The project is really tough right now" and calculates a negative sentiment score (e.g., -0.5).

[1412] 4. Storing and calculating sentiment scores

[1413] The server stores the analysis results in a database and calculates an emotion score.

[1414] Example: Record the sentiment score of each message in a database and calculate the average sentiment score for each user or team.

[1415] 5. Alerting and Notification

[1416] The server generates an alert if the emotion score falls below a set threshold and sends a notification to the relevant manager or HR department.

[1417] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[1418] 6. Report Generation and Viewing

[1419] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[1420] Example: Create a monthly report containing company-wide and departmental sentiment scores and trend graphs and provide it to managers.

[1421] 7. Feedback and Actions

[1422] Users check the reports and alerts provided by the server, consider and implement the necessary measures.

[1423] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[1424] In this way, the EmoSync system monitors the emotional climate within an organization in real time and takes appropriate measures at the right time to improve the work environment.

[1425] The processing flow will be explained below.

[1426] Step 1:

[1427] The device receives text messages sent by the user using internal communication tools (e.g., messaging apps, email systems).

[1428] Example: An employee sends a message on a messaging app saying, "I'm stressed because the project isn't progressing."

[1429] Step 2:

[1430] The terminal transmits the acquired text message to the server.

[1431] Example: Sending text message data to a server through a messaging app's API.

[1432] Step 3:

[1433] The server receives the text message sent from the terminal and temporarily stores it in a database.

[1434] Example: Recording received messages in a database and preparing them for cleansing processing.

[1435] Step 4:

[1436] The server cleans the received message data, removing unnecessary HTML tags and special characters.

[1437] Example: Removing redundant tags and symbols from the text "I'm stressed because the project isn't progressing."

[1438] Step 5:

[1439] The server passes the cleaned text data to a natural language processing (NLP) engine to perform sentiment analysis.

[1440] Example: An NLP engine analyzes the text "I'm stressed because the project isn't progressing" and calculates a sentiment score (e.g., -0.6).

[1441] Step 6:

[1442] The server stores the calculated emotion scores in a database.

[1443] Example: Analysis results and sentiment scores are recorded in a database for each user and message.

[1444] Step 7:

[1445] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[1446] Example: Aggregate each user's sentiment score from a week's worth of message data and calculate the average value.

[1447] Step 8:

[1448] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[1449] Example: Alert if a team's sentiment score falls below -0.3.

[1450] Step 9:

[1451] The server notifies the generated alerts to the HR department and relevant managers.

[1452] Example: Sending alert messages to HR and managers via email and push notifications.

[1453] Step 10:

[1454] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[1455] Example: Create and distribute a monthly report containing sentiment scores and trend graphs for each department.

[1456] Step 11:

[1457] The terminal receives reports and alerts sent from the server and displays them to the user.

[1458] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[1459] Step 12:

[1460] Users check the alerts and reports provided by the server and take appropriate action.

[1461] Example: HR receives an alert and provides mental health counseling to a specific employee.

[1462] In this way, the EmoSync system can monitor employees' emotional states in real time, detect problems early, and take countermeasures.

[1463] Example 1

[1464] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1465] Conventional internal communication tools do not analyze the emotional content of messages, making it difficult to provide the feedback and early action needed to improve the work environment. The accumulation of negative emotions, in particular, can lead to lower employee morale, poor performance, and even turnover. Furthermore, the lack of reports based on concrete data makes it difficult for managers to implement effective countermeasures.

[1466] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1467] In this invention, the server includes means for collecting in-house text-based communication data, means for data cleansing the collected data and formatting it into a required format, means for supplying the formatted data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, and means for generating and displaying periodic reports based on the sentiment score and analysis results. This makes it possible to detect emotions from in-house communications in real time and take early countermeasures.

[1468] "Internal text-based communication data" refers to textual data generated from messaging tools and emails used within the company.

[1469] "Data cleansing" is the process of removing unnecessary information and noise from collected data and preparing it in a format suitable for analysis.

[1470] A "natural language processing engine" is a software system that includes algorithms and techniques for understanding, analyzing, and generating human language.

[1471] "Sentiment analysis" is the process of identifying positive, negative, and neutral sentiments from text data and calculating their scores.

[1472] An "emotion score" is data that quantifies the strength and type of emotion contained in specific text data.

[1473] A "threshold" is a reference value for determining whether a particular indicator or value exceeds a certain standard.

[1474] An "alert" is a notification of a warning or caution that is generated when a pre-defined condition is met.

[1475] A "report" is a document or digital information that compiles analytical results and statistical data and provides them to users.

[1476] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1477] System Configuration

[1478] 1. Server

[1479] The server has an API interface for collecting internal text-based communication data. The server has a built-in natural language processing engine and performs sentiment analysis on the collected data. The sentiment analysis results are stored in a database and a sentiment score is calculated. The server monitors the set threshold in real time and has the ability to generate alerts and send notifications if the sentiment score falls below a certain level. It also generates regular reports and provides them to users via an accessible dashboard or by email.

[1480] 2. Device (user's PC or smartphone)

[1481] The terminal has an interface that allows users to send text messages using an internal communication tool, and has the function of receiving emotion scores, alerts, and reports sent from the server and displaying them to the user.

[1482] 3. Users (managers, human resources department)

[1483] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[1484] Program processing

[1485] The program of this system operates around three parties: the server, the terminal, and the user.

[1486] Server: AWS or a similar cloud infrastructure is used, and the program is implemented in Python or JavaScript. For natural language processing, the Google Cloud Natural Language API or engine is used as the NLP engine. MySQL or PostgreSQL is used as the database.

[1487] Device: Runs on the user's PC or smartphone. It uses a web browser to communicate with the server via API and displays sentiment scores, alerts, and reports in real time.

[1488] Users: Users monitor the emotional state of their organization through the provided dashboards and email reports and consider necessary countermeasures.

[1489] Specific examples

[1490] Data Collection and Cleansing

[1491] The device collects messages sent by users via internal communication tools and sends them to the server. For example, if a user sends a message on Slack saying, "The project is really tough right now," the message is sent to the server via Slack's API. The server cleanses the received message data, removing HTML tags and special characters, and formats it into the required format.

[1492] sentiment analysis

[1493] The server passes the cleansed text data to an NLP engine for sentiment analysis. For example, the text "The current project is really tough" is analyzed as a negative sentiment score of -0.5. The server stores these scores in a database and calculates average sentiment scores for each user or team.

[1494] Alerting and Notifications

[1495] The server generates an alert when the sentiment score falls below a set threshold and sends a notification to the relevant manager or HR department. For example, if a team's sentiment score falls below -0.3, an alert is sent to the HR department via email or push notification.

[1496] Report Generation and Viewing

[1497] The server generates regular reports based on the emotion scores and analysis results and provides them to users via dashboards or email. For example, at the end of each month, a report showing the emotion scores and trends for the entire company or department is created and provided to managers.

[1498] Feedback and Actions

[1499] Users can check reports and alerts provided by the server, consider and implement necessary measures, etc. For example, if a manager notices a decline in the team's emotional score, they can set up an online meeting to check in with team members on their mental health.

[1500] Prompt Sentence Examples

[1501] 1. "Write a program that collects Slack messages and sends them to a server."

[1502] 2. "Write a Python program to cleanse text data."

[1503] 3. "Implement an NLP engine for sentiment analysis of text data."

[1504] 4. "Write a SQL query to store the sentiment scores and perform statistical processing."

[1505] 5. "Write a program that generates an alert and sends a notification when a specific condition occurs."

[1506] 6. "Write a program that generates a report of emotion scores."

[1507] 7. "Create guidelines for feedback and action planning based on the sentiment score report."

[1508] In this way, the EmoSync system monitors the emotional state within an organization in real time and takes appropriate measures in a timely manner to improve the work environment.

[1509] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1510] Step 1: Data collection

[1511] The device collects text messages sent by users using internal communication tools and sends them to a server via API. For example, if a user sends a message on Slack saying, "The project is really tough right now," that message is forwarded to the server via the Slack API. The input of this step is the user's message, and the output is the raw text data sent to the server.

[1512] Step 2: Data Preprocessing

[1513] The server cleanses the received message data, removes unnecessary HTML tags and special characters, and formats it into the required format. This project is really tough " into clean text "The current project is really tough." The input of this step is raw text data and the output is cleansed text data.

[1514] Step 3: Sentiment analysis

[1515] The server passes the cleansed text data to a natural language processing (NLP) engine to perform sentiment analysis. Specifically, the NLP engine calculates a negative sentiment score (e.g., -0.5) for the text "This project is really tough." The input for this step is the cleansed text data, and the output is the sentiment score for the text.

[1516] Step 4: Storing and calculating sentiment scores

[1517] The server stores the analysis results in a database, records the sentiment score for each message, and calculates the average sentiment score for each user or team. Specifically, the server adds the score for each message to the database and calculates the average score for each user or team. The input for this step is the sentiment score, and the output is the score stored in the database and the calculated average score.

[1518] Step 5: Alerting and Notification

[1519] The server generates an alert when the emotion score falls below the set threshold and sends a notification to the relevant manager or HR department. For example, if the team's average emotion score falls below -0.3, an alert is sent to the HR department via email or push notification. The input of this step is the emotion score and the set threshold, and the output is the generated alert notification.

[1520] Step 6: Generate and view the report

[1521] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via dashboards or email. Specific operations include generating monthly reports, creating PDFs containing company-wide and department-specific sentiment scores and graphs showing trends, and distributing these to each manager via email. The input to this step is the sentiment scores and analysis results, and the output is the generated report.

[1522] Step 7: Feedback and action

[1523] The user checks the reports and alerts provided by the server, considers and implements the necessary response. For example, a manager may notice a decline in their team's emotional score and schedule an online meeting to conduct a mental health check-in. The input for this step is the report or alert, and the output is the user's response or feedback.

[1524] (Application example 1)

[1525] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1526] In traditional brick-and-mortar stores, it was difficult to grasp customer satisfaction and emotional state in real time, making it difficult to respond quickly to directly improve the customer experience. Furthermore, there was a lack of means to effectively collect customer feedback and use it to improve store operations. This resulted in delays in taking concrete measures to prevent declines in customer satisfaction and customer defections.

[1527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1528] In this invention, the server includes a means for collecting video data for capturing customer behavior and facial expressions, a means for preprocessing the collected video data using an image processing library, and a means for supplying the preprocessed data to emotion recognition software to perform emotion analysis, thereby enabling real-time analysis of customer emotions and prompt response.

[1529] "Video data" is visual information that captures the customer's behavior and facial expressions.

[1530] An "image processing library" is a software tool for preprocessing collected video data.

[1531] "Emotion recognition software" is a program for analyzing customer emotions from pre-processed video data.

[1532] An "emotion score" is a numerical evaluation value of an emotion calculated by emotion recognition software.

[1533] An "alert" is a warning notification that is generated when an emotion score falls below a set threshold.

[1534] "Reports" are regular reports summarizing sentiment scores and analysis results.

[1535] "Data preprocessing" is the process of removing unnecessary information from collected video data and preparing it in a format suitable for analysis.

[1536] A "natural language processing engine" is a program that analyzes text feedback entered by customers and calculates an emotional score.

[1537] A "dashboard" is a web application interface for visually displaying sentiment scores and analysis results.

[1538] "Trend analysis" is the process of tracking fluctuations in sentiment scores and analyzing patterns over time.

[1539] As an embodiment of the present invention, the configuration of the CustomerEmoSync system and program processing will be described in detail below.

[1540] System Configuration

[1541] 1. Server

[1542] The server has a means for collecting video data to capture customer behavior and facial expressions.

[1543] The server uses an image processing library (e.g., OpenCV) to preprocess the collected video data.

[1544] The server uses emotion recognition software (e.g., Microsoft Azure Face API) to perform customer emotion analysis from the pre-processed video data.

[1545] The server stores the results of the sentiment analysis and has a means for calculating sentiment scores.

[1546] The server has a means to generate an alert and notify if the emotion score falls below a set threshold.

[1547] The server has the means to generate and display periodic reports based on the sentiment scores and analysis results.

[1548] 2. Device (user's smartphone)

[1549] The terminal has an interface for the customer to input a feedback message.

[1550] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[1551] 3. Users (store managers and staff)

[1552] Users check the reports and alerts provided by the server and take the necessary measures.

[1553] Program processing

[1554] 1. Data Collection

[1555] The video data collected by the camera is transmitted to a server.

[1556] Text feedback entered by the customer using their smartphone is also sent to the server.

[1557] 2. Data Preprocessing

[1558] The server uses image processing libraries such as OpenCV to cleanse the collected video data and format it into the required format.

[1559] Customer text feedback is also passed to a natural language processing engine for cleansing.

[1560] 3. Sentiment analysis

[1561] The server passes the pre-processed video data to emotion recognition software such as Microsoft Azure Face API to perform sentiment analysis.

[1562] A natural language processing engine is also used to perform sentiment analysis of the text feedback.

[1563] Specific examples

[1564] For example, if a camera captures a customer's facial expression in a store and recognizes that the customer is happy, the emotion score will be calculated as a high positive value. Also, if a customer types feedback on their smartphone, such as "The service was great," the text will be analyzed by a natural language processing engine as a positive emotion.

[1565] An example of a prompt sentence might be, "Please perform sentiment analysis on the following text: 'I am very satisfied with your service today. Thank you.'"

[1566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1567] Step 1:

[1568] The server collects video data from cameras in the store, which transmits customer behavior and facial expressions to the server in real time. The input is video data from the cameras, which the server receives and stores.

[1569] Step 2:

[1570] The server cleanses the collected video data using an image processing library (such as OpenCV). It removes unnecessary parts from the video data and formats it into the required format. The input is the video data collected in step 1, and the output is the video data after preprocessing.

[1571] Step 3:

[1572] The server then feeds the preprocessed video data to emotion recognition software (such as Microsoft Azure Face API) to perform emotion analysis. This software calculates an emotion score based on the customer's facial expressions. The input is the preprocessed video data from step 2, and the output is the analysis results, including the emotion score.

[1573] Step 4:

[1574] The user inputs a feedback message using a smartphone. The smartphone app collects the text data and sends it to the server. The input is the text feedback from the customer, and the output is the text data sent to the server.

[1575] Step 5:

[1576] The server uses a natural language processing engine to analyze the customer's text feedback and calculate a sentiment score from the text data. The input is the text data received in step 4, and the output is the sentiment score of the text.

[1577] Step 6:

[1578] The server integrates the emotion scores obtained from the video data and text data and stores the overall emotion score in a database. The inputs are the individual emotion scores, and the output is the integrated overall emotion score.

[1579] Step 7:

[1580] The server generates an alert when the integrated overall emotion score falls below a set threshold. The alert is sent to the administrator's terminal. The input is the integrated emotion score, and the output is an alert notification.

[1581] Step 8:

[1582] The server generates periodic reports based on the sentiment scores and analysis results. These reports are provided to users via a dashboard or email. The input is the sentiment scores and analysis results stored in the database, and the output is the periodic reports.

[1583] Step 9:

[1584] Users check the reports and alerts provided by the server and take necessary measures. For example, if customer satisfaction is low, they can implement specific improvement measures such as reviewing staff response. The input is the report or alert, and the output is the specific measures.

[1585] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1586] As an embodiment of the present invention, a configuration in which the EmoSync system and an emotion engine are combined will be described in detail.

[1587] System Configuration

[1588] 1. Server

[1589] The server has an API interface for collecting text-based message data from internal communication tools.

[1590] The server has a built-in natural language processing engine and performs sentiment analysis on the collected data.

[1591] The server stores the analysis results in a database and calculates emotion scores in real time.

[1592] The server monitors the set thresholds and generates alerts and sends notifications when the emotion score falls below them.

[1593] The server generates periodic reports and provides them via a user-accessible dashboard or via email.

[1594] The server recognizes the user's emotions through an emotion engine and has the ability to analyze input other than text data (emoticons and voice data).

[1595] 2. Device (user's PC or smartphone)

[1596] The terminal has an interface that allows the user to send text messages using an in-house communication tool.

[1597] The device has the ability to receive emotion scores, alerts, and reports sent from the server and display them to the user.

[1598] The terminal has the function of sending input such as voice data and emojis to a server.

[1599] 3. Users (managers, human resources department)

[1600] The user is responsible for checking the reports and alerts provided by the server and taking any necessary measures.

[1601] Program processing

[1602] The program of this system performs the following main processes.

[1603] 1. Data Collection

[1604] The device collects all messages, emoticons, and voice data sent by users using internal communication tools.

[1605] Example: An employee texts on a messaging app saying, "I'm stressed because the project isn't progressing," while also using a tired-face emoji.

[1606] 2. Data Transmission

[1607] The terminal transmits the acquired text messages, emojis, and voice data to the server.

[1608] Example: Sending text messages, emoji data, and voice data to a server via a messaging app's API.

[1609] 3. Data Preprocessing

[1610] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[1611] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing" and organize the emojis and audio files that were passed through it.

[1612] 4. Emotion recognition

[1613] The server passes the cleaned text data, emojis, and voice data to the emotion engine to perform emotion recognition.

[1614] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," along with an emoji of a tired expression, and also voice data, and calculates an overall emotion score (for example, -0.7) from all elements.

[1615] 5. Storing and calculating sentiment scores

[1616] The server stores the analysis results in a database and calculates an emotion score.

[1617] Example: Analysis results and sentiment scores are recorded in a database for each user and message, and then aggregated.

[1618] 6. Alert Generation and Notification

[1619] The server generates an alert if the emotion score falls below a set threshold and notifies the relevant manager or HR department.

[1620] Example: If a team's sentiment score falls below -0.3, send an email or push notification to the HR department.

[1621] 7. Report Generation and Viewing

[1622] The server generates regular reports based on the sentiment scores and analysis results and provides them to users via a dashboard or email.

[1623] Example: Create a monthly report for managers that includes company-wide and departmental sentiment scores and trend graphs.

[1624] 8. Feedback and Actions

[1625] Users check the reports and alerts provided by the server and take appropriate action.

[1626] Example: A manager sees a drop in their team's sentiment score and schedules a team meeting for a mental health check-in.

[1627] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[1628] The processing flow will be explained below.

[1629] Step 1:

[1630] The device captures text messages, emojis, and voice data sent by users using internal communication tools (e.g., messaging apps, email systems).

[1631] Example: An employee sends a text message on a messaging app saying, "I'm stressed because the project isn't progressing," along with a tired-face emoji and, if necessary, a voice message.

[1632] Step 2:

[1633] The device sends the acquired text messages, emojis, and voice data to the server all at once.

[1634] Example: Sending text messages, emojis, and voice data to a server through a messaging app's API.

[1635] Step 3:

[1636] The server receives text messages, emojis, and voice data sent from the device and temporarily stores them in a database.

[1637] Example: Record received messages, emoji information, and audio files in a database to prepare for the cleansing process.

[1638] Step 4:

[1639] The server cleanses the incoming text messages, emojis, and voice data and formats them into the required format.

[1640] Example: Remove unnecessary tags and symbols from the text "I'm stressed because the project isn't progressing," convert emoji information into a format that's easier to analyze, and convert audio data into text.

[1641] Step 5:

[1642] The server passes the cleaned text data, emoji, and voice data to an emotion engine to perform emotion recognition.

[1643] Example: An emotion engine analyzes the text "I'm stressed because the project isn't progressing," an emoji showing a tired expression, and voice data, and calculates an overall emotion score (e.g., -0.7) from all of these elements.

[1644] Step 6:

[1645] The server stores the emotion scores obtained as a result of the emotion recognition in a database.

[1646] Example: Analysis results and sentiment scores are recorded in a database for each user and message, allowing for future reference.

[1647] Step 7:

[1648] The server calculates the average emotion score for each user and the entire team for a certain period of time (for example, weekly or monthly).

[1649] Example: Aggregate each user's emotion score from a week's worth of message data, emojis, and voice data, and calculate the average value.

[1650] Step 8:

[1651] The server generates an alert if the calculated emotion score falls below a pre-set threshold.

[1652] Example: Alert if a team's sentiment score falls below -0.3.

[1653] Step 9:

[1654] The server notifies the generated alerts to the HR department and relevant managers.

[1655] Example: Sending alert messages to HR and managers via email and push notifications.

[1656] Step 10:

[1657] The server generates periodic reports and outputs sentiment score trends and detailed analysis results.

[1658] Example: Create a monthly report for managers that includes a trend graph of sentiment scores for the entire company and each department.

[1659] Step 11:

[1660] The terminal receives reports and alerts sent from the server and displays them to the user.

[1661] Example: A notification appears on the manager's PC screen saying, "This month's team sentiment score is -0.4."

[1662] Step 12:

[1663] Users check the alerts and reports provided by the server and take appropriate action.

[1664] Example: HR receives an alert and provides mental health counseling to a specific employee.

[1665] In this way, by combining the EmoSync system with an emotion engine, it is possible to recognize user emotions in real time not only from text data but also from emojis and voice data, making it possible to monitor the emotional state within an organization with greater accuracy.

[1666] Example 2

[1667] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1668] Conventional systems only dealt with text-based communication data, which meant they had issues with insufficient understanding of emotions. Furthermore, they did not analyze emojis or voice data, making it difficult to accurately recognize users' emotions, leaving managers and HR departments with a lack of information to take appropriate action. Furthermore, there was a lack of a system that allowed managers to monitor emotion scores in real time and take prompt action.

[1669] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1670] In this invention, the server includes means for collecting text-based data, emoji, and voice data from in-house communication tools, means for supplying the collected data to a natural language processing engine and an emotion analysis engine to perform emotion recognition, means for storing the recognition results in a database and calculating individual data and overall emotion scores, means for generating an alert and notifying an administrator when the emotion score falls below a set threshold, and means for creating regular reports based on the emotion scores and data analysis results and displaying them on a dashboard or by email. This makes it possible to recognize user emotions in real time not only from text data but also from emoji and voice data, and to monitor the emotional state within an organization with greater accuracy.

[1671] "Internal communication tools" refers to digital platforms used to facilitate communication between employees.

[1672] "Text-based data" refers to communication content in the form of written information.

[1673] "Emoji" refers to small icons or pictograms used to visually express emotions or intentions.

[1674] "Audio data" refers to data that records words or sounds in digital format.

[1675] A "natural language processing engine" refers to software technology that analyzes text data and understands and processes it as natural language.

[1676] An "emotion analysis engine" refers to an analytical tool used to identify emotions and impressions from collected data.

[1677] "Emotion recognition" refers to the process of analyzing and extracting emotions from text, emoji, and voice data.

[1678] A "database" refers to a collection of information that allows data to be efficiently stored, managed, and retrieved.

[1679] "Emotion score" refers to a numerical representation of a user's emotional state based on analyzed data.

[1680] A "threshold" refers to a reference value that sets a certain condition or standard and changes operation depending on whether that standard is exceeded or fallen below.

[1681] An "alert" is a warning or notification that occurs based on set conditions.

[1682] "Administrator" means an individual or entity responsible for the oversight and management of a System.

[1683] A "dashboard" refers to a screen or interface designed to allow users to visually view a variety of data and information at a glance.

[1684] "Report" refers to a document that summarizes the results of data analysis and important information within a specific period of time.

[1685] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings.

[1686] System Configuration

[1687] The present invention is comprised of multiple elements: a server, a terminal, and a user.

[1688] server

[1689] The server has the following functions:

[1690] 1. Data Collection

[1691] The server has an API interface for collecting text-based messages, emojis, and voice data from internal communication tools.

[1692] 2. Data Preprocessing

[1693] The server cleanses the data it receives and formats it into an analyzable format, using a natural language processing engine to remove unnecessary tags and symbols from the text, and organizes and converts emojis and audio files.

[1694] 3. Emotion recognition

[1695] The server passes the cleansed data to a sentiment analysis engine for emotion recognition. The sentiment analysis engine analyzes the text "I'm stressed because the project isn't progressing," emojis showing tired expressions, and voice data to calculate an overall sentiment score.

[1696] 4. Database storage and emotion score calculation

[1697] The server stores the analysis results in a database and calculates individual data and overall emotion scores.

[1698] 5. Alerting and Notification

[1699] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold, allowing the administrator to grasp the user's emotional state in real time.

[1700] 6. Report Generation and Viewing

[1701] The server generates regular reports based on sentiment scores and data analysis results and provides them via dashboard or email.

[1702] Terminal

[1703] The terminal has the following features:

[1704] 1. Data entry and submission

[1705] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool and transmits them to the server.

[1706] 2. Results display

[1707] The terminal receives the emotion scores, alerts, and reports sent from the server and displays them to the user.

[1708] User

[1709] Users have the following roles:

[1710] 1. Check reports and alerts

[1711] Users (usually managers and HR departments) check the reports and alerts provided by the server and take necessary measures, such as scheduling team meetings to check in on mental health.

[1712] Specific examples

[1713] For example, if an employee uses a messaging app to send a text message saying, "I'm stressed because the project isn't progressing," and also uses a tired-face emoji, the following process will take place:

[1714] 1. The device receives this message and emoji and sends it to the server.

[1715] 2. The server preprocesses the received data and passes it to the sentiment analysis engine.

[1716] 3. The sentiment analysis engine analyzes the data and calculates a sentiment score (e.g., -0.7).

[1717] 4. The server stores the emotion score in a database and generates an alert to notify the administrator if the score falls below a threshold.

[1718] 5. The server generates periodic reports and provides them to the user via dashboard or email.

[1719] Prompt Sentence Examples

[1720] Use prompt statements of the following form to feed data into the generative AI model:

[1721] Prompt statement:

[1722] Please enter the text message "I'm stressed because the project isn't progressing," along with a tired-looking emoji, and any associated audio data.

[1723] As described above, the present invention makes it possible to recognize user emotions in real time not only from text data but also from pictograms and voice data, thereby monitoring the emotional state within an organization with higher accuracy.

[1724] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1725] Step 1: Data collection

[1726] The terminal acquires messages, pictograms, and voice data sent by the user using the in-house communication tool. This data is input.

[1727] Specifically, it uses the API of a messaging app to capture text messages such as "I'm stressed because the project isn't progressing," along with emojis of tired expressions, and related audio data. The output after data collection is the collected text data, emojis, and audio data.

[1728] Step 2: Send data

[1729] The device sends the captured text messages, emojis, and voice data to the server, where this data is entered.

[1730] Specifically, the device uses the messaging app's API to send this data to the server. The output after the data transmission is text data, emojis, and voice data sent to the server.

[1731] Step 3: Data Preprocessing

[1732] The server cleanses the received text messages, emojis, and voice data and formats them into an analyzable format. This received data is then input.

[1733] Specifically, it uses a natural language processing engine to remove unnecessary tags and symbols from text, and format and convert emoji and voice data. The output of the data preprocessing is cleansed text data, analyzable emoji, and voice data.

[1734] Step 4: Emotion Recognition

[1735] The server passes the formatted text data, emojis, and voice data to an emotion analysis engine to perform emotion recognition.

[1736] Specifically, the emotion analysis engine analyzes text such as "I'm stressed because the project isn't progressing," emojis showing tired expressions, and the tone of the voice, and calculates an overall emotion score (e.g., -0.7) from this data. The output after emotion recognition is the overall emotion score.

[1737] Step 5: Storing and calculating sentiment scores

[1738] The server stores the emotion recognition results in a database and calculates the emotion scores for each data item and the total emotion score.

[1739] Specifically, the system saves the acquired emotion scores and their analysis results as records in a database, and calculates the emotion trends for each user and each message. After the emotion scores are saved, the output is the emotion scores and analysis results saved in the database.

[1740] Step 6: Alerting and Notification

[1741] The server generates an alert and notifies the administrator when the emotion score falls below a set threshold. This emotion score is input.

[1742] Specifically, whenever the server detects an emotion score below a certain threshold (e.g., -0.3), it automatically generates and sends an alert email or push notification to the administrator. The output after generating an alert is the sent alert notification.

[1743] Step 7: Generate and view the report

[1744] The server generates periodic reports based on the sentiment scores and analysis results and provides them to the user via a dashboard or email.

[1745] Specifically, the server automatically generates monthly or weekly reports, documents sentiment trends, summarized graphs, comments, etc., and sends them to the user (e.g., manager). The output of the report generation is the provided report.

[1746] Step 8: Feedback and action

[1747] The user checks the reports and alerts provided by the server and takes appropriate action. These reports and alerts are entered.

[1748] Specifically, the manager reviews the report, looks at the team's emotion scores and trends, and schedules team meetings to provide mental health support as needed. After the measures are implemented, the output is the measures taken.

[1749] The above is a description of the specific processing steps of this system.

[1750] (Application example 2)

[1751] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1752] Conventional communication systems are limited to text-based data, making it difficult to accurately grasp a user's emotional state and stress level. Furthermore, virtual stores are unable to analyze a user's emotions in real time and provide appropriate product recommendations and customer support. This makes it difficult to improve the user experience and effectively manage stress.

[1753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1754] In this invention, the server includes means for collecting in-house text-based communication data, means for supplying the collected data to a natural language processing engine and performing sentiment analysis, means for saving the analysis results and calculating a sentiment score, means for generating and notifying an alert when the sentiment score falls below a set threshold, means for generating and displaying periodic reports based on the sentiment score and analysis results, and means for analyzing data entered by users and suggesting products tailored to their individual emotional states. This makes it possible to grasp the emotional state of users in real time and suggest products and manage stress according to that state.

[1755] "Internal text-based communication data" refers to text data sent and received via messaging applications and communication tools used by employees within the company.

[1756] A "natural language processing engine" is a software component that analyzes collected text data and understands its meaning and sentiment.

[1757] "Sentiment analysis" is the process of determining a user's emotional state from text data and quantifying that emotion.

[1758] An "emotion score" is a numerical representation of the emotional state analyzed from text data, and is often expressed in the range of -1 to 1.

[1759] An "alert" is a warning or notification that is generated when a value falls below a set threshold, and is intended to notify users and administrators of an abnormality.

[1760] "Means for generating and displaying reports" refers to a function that periodically creates reports based on sentiment scores and analysis results, and visually presents them to users and administrators.

[1761] "User input data" refers to text messages, voice data, emojis, etc. provided by users accessing the virtual store.

[1762] "Product suggestion" means automatically recommending appropriate products and services based on the user's emotional state.

[1763] As an embodiment of this invention, we will explain the configuration of a virtual store that combines the EmoSync system and an emotion engine. This system analyzes the user's emotional state in real time and provides product suggestions and customer support based on the results. The system configuration is explained in detail below.

[1764] System Configuration

[1765] 1. Server

[1766] The server has an API interface for collecting input data (text, emoji, and voice data) from users who access the virtual store. The collected data is then subjected to sentiment analysis using a natural language processing engine.

[1767] This server includes the following main functions:

[1768] A function that stores the results of sentiment analysis in a database and calculates sentiment scores in real time.

[1769] Alerts can be generated when sentiment scores fall below a set threshold.

[1770] Generate regular reports based on sentiment scores and analysis results.

[1771] Product suggestion function based on the user's emotional state.

[1772] 2. Device (user's smartphone or head-mounted display)

[1773] Users can access the virtual store and use the interface to send text messages or input voice data. The device then sends this data to the server, which then receives and displays emotion scores, alerts, product suggestions, and more.

[1774] 3. Users

[1775] Users browse products through a virtual store interface, input text messages or voice data, and receive product suggestions and customer support based on an emotion score provided by the server.

[1776] Program processing

[1777] Hardware and Software

[1778] Hardware

[1779] Smartphone (Android, iOS)

[1780] Head-mounted displays (Oculus Rift, HoloLens)

[1781] software

[1782] EmoSync Sentiment Analysis API

[1783] Python's requests library (sending HTTP requests)

[1784] Python json library (data cleansing and sending)

[1785] nltk and textblob (preprocessing text data)

[1786] Processing steps

[1787] 1. Data collection and transmission

[1788] Users can use their smartphones or head-mounted displays to input text messages or voice data when accessing a virtual store, and this data is sent to the server via an API.

[1789] 2. Data Preprocessing

[1790] The server cleanses the received text messages and voice data and formats them into the required format.

[1791] 3. Emotion recognition

[1792] The cleaned data is then passed to a natural language processing engine for emotion recognition. For example, if a user types, "I've been feeling stressed lately," the text data is analyzed and an "emotion score" of -0.7 is calculated.

[1793] 4. Saving sentiment scores and generating alerts

[1794] The server stores the analysis results and emotion scores in a database and generates an alert if the emotion score falls below a set threshold.

[1795] 5. Product proposal

[1796] The server recommends products and services based on the user's sentiment score and notifies the user.

[1797] Examples and prompts

[1798] As a concrete example, if a user visits a virtual store and enters the text "I've been feeling stressed lately," the following prompt will be sent to the server:

[1799] "I've been feeling stressed lately"

[1800] This prompt is sent to the server, which returns an "emotion score" as the result of emotion analysis. Product suggestions based on this score are then sent to the user. For example, if the stress score is high, relaxation items are recommended; if the score is medium, trendy products are recommended; and if the score is low, luxury products are recommended.

[1801] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1802] Step 1:

[1803] A user accesses a virtual store and inputs text or voice data. This input data is used as initial data for emotion analysis. Specifically, the user inputs "I've been feeling stressed lately" via a smartphone or head-mounted display.

[1804] Step 2:

[1805] The device sends user input data to the server via the API. The input data includes prompts, voice data, emojis, etc. For example, the text "I've been feeling stressed lately" is sent.

[1806] Step 3:

[1807] The server cleanses the received input data and performs preprocessing before passing it to the natural language processing engine. Specifically, it removes unnecessary tags and symbols from text data and formats voice data and emojis into appropriate formats.

[1808] Step 4:

[1809] The server then passes the cleansed data to a natural language processing engine for emotion recognition. For example, the text "I've been feeling stressed lately" is analyzed and an emotion score (e.g., -0.7) is calculated. This score is an important indicator of the user's emotional state.

[1810] Step 5:

[1811] The server stores the emotion analysis results in a database and calculates an emotion score in real time. For example, if an emotion score of -0.7 is recorded in the database and the user's emotional state is consistently low, the data will be saved as data for future measures.

[1812] Step 6:

[1813] The server generates an alert and notifies the administrator or customer support team when the emotion score falls below a set threshold. Specifically, when the emotion score falls below -0.3, an email or push notification is sent.

[1814] Step 7:

[1815] The server generates product recommendations suited to the user based on the emotion score and analysis results. For example, if the emotion score is low, it will suggest relaxation and healing items, if the score is medium, it will suggest trendy products, and if the score is high, it will suggest luxury items.

[1816] Step 8:

[1817] The server generates periodic reports and provides them to users and administrators via a dashboard or email, including sentiment score trends, trend analysis, and suggested product information.

[1818] Step 9:

[1819] The user checks the emotion score, product suggestions, and reports provided by the server and selects an appropriate product. For example, a user who inputs "I've been feeling stressed lately" may purchase a suggested relaxation item.

[1820] Step 10:

[1821] The terminal sends the product information selected by the user to the server to complete the purchase process, and the server records the purchase information in a database for future data analysis and recommendations.

[1822] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1823] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1824] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1825] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1826] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1827] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1828] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1829] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1830] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1831] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1832] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1833] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1834] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1835] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1836] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1837] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1838] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1839] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1840] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1841] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1842] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1843] The following is further disclosed regarding the above embodiment.

[1844] (Claim 1)

[1845] A means of collecting internal text-based communication data;

[1846] means for feeding the collected data to a natural language processing engine to perform sentiment analysis;

[1847] means for storing the analysis results and calculating a sentiment score;

[1848] a means for generating an alert and notifying the user when the emotion score falls below a set threshold;

[1849] The system includes a means for generating and displaying periodic reports based on the sentiment scores and analysis results.

[1850] (Claim 2)

[1851] 10. The system of claim 1, further comprising: means for collecting data from a plurality of internal company communication tools.

[1852] (Claim 3)

[1853] 10. The system of claim 1, further comprising means for tracking and trending fluctuations in the emotion scores.

[1854] "Example 1"

[1855] (Claim 1)

[1856] A means of collecting internal text-based communication data;

[1857] A means of data cleansing and formatting the collected data into the required format;

[1858] means for feeding the formatted data to a natural language processing engine to perform sentiment analysis;

[1859] means for storing the analysis results and calculating a sentiment score;

[1860] a means for generating an alert and notifying the user when the emotion score falls below a set threshold;

[1861] The system includes a means for generating and displaying periodic reports based on the sentiment scores and analysis results.

[1862] (Claim 2)

[1863] 10. The system of claim 1, further comprising: means for collecting data from a plurality of internal company communication tools.

[1864] (Claim 3)

[1865] 10. The system of claim 1, further comprising means for tracking and trending fluctuations in the emotion scores.

[1866] "Application Example 1"

[1867] (Claim 1)

[1868] A means of collecting video data to capture customer behavior and facial expressions;

[1869] A means of preprocessing the collected video data using an image processing library;

[1870] means for providing the pre-processed data to emotion recognition software to perform emotion analysis;

[1871] means for storing the results of the sentiment analysis and calculating a sentiment score;

[1872] a means for generating an alert and notifying the user when the emotion score falls below a set threshold;

[1873] The system includes a means for generating and displaying periodic reports based on the sentiment scores and analysis results.

[1874] (Claim 2)

[1875] 10. The system of claim 1, further comprising means for collecting a plurality of video data and text data.

[1876] (Claim 3)

[1877] 10. The system of claim 1, further comprising means for tracking and trending fluctuations in the emotion scores.

[1878] "Example 2: Combining Emotion Engines"

[1879] (Claim 1)

[1880] A means of collecting text-based data, emojis, and voice data from internal company communication tools;

[1881] means for feeding the collected data to a natural language processing engine and a sentiment analysis engine to perform sentiment recognition;

[1882] A means for storing the recognition results in a database and calculating individual data and overall emotion scores;

[1883] A means for generating an alert and notifying an administrator when the sentiment score falls below a set threshold;

[1884] A system that includes the means to generate regular reports based on sentiment scores and data analysis results and display them on a dashboard or via email.

[1885] (Claim 2)

[1886] 10. The system of claim 1, further comprising: means for collecting data from a plurality of internal company communication tools.

[1887] (Claim 3)

[1888] 10. The system of claim 1, further comprising means for monitoring fluctuations in the emotion scores and analyzing trends.

[1889] "Application example 2 when combining emotion engines"

[1890] (Claim 1)

[1891] A means of collecting internal text-based communication data;

[1892] means for feeding the collected data to a natural language processing engine to perform sentiment analysis;

[1893] means for storing the analysis results and calculating a sentiment score;

[1894] a means for generating an alert and notifying the user when the emotion score falls below a set threshold;

[1895] A means to generate and display periodic reports based on sentiment scores and analysis results;

[1896] A means of analyzing user input data and making product suggestions tailored to each individual's emotional state;

[1897] A system including:

[1898] (Claim 2)

[1899] 10. The system of claim 1, further comprising: means for collecting data from a plurality of internal communication tools; and means for transmitting input data of a user accessing the virtual store to the emotion engine.

[1900] (Claim 3)

[1901] 10. The system of claim 1, further comprising means for tracking fluctuations in the sentiment scores and analyzing trends, and means for determining recommendations. [Explanation of symbols]

[1902] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting internal text-based communication data; means for feeding the collected data to a natural language processing engine to perform sentiment analysis; means for storing the analysis results and calculating a sentiment score; a means for generating an alert and notifying the user when the emotion score falls below a set threshold; The system includes a means for generating and displaying periodic reports based on the sentiment scores and analysis results.

2. The system of claim 1 , further comprising means for collecting data from a plurality of internal company communication tools.

3. The system of claim 1 , further comprising means for tracking fluctuations in the emotion scores and analyzing trends.

Images