System

Abstract

A system is provided.SOLUTION: A headgear configured to collect brain wave data; a terminal configured to receive brain wave data transmitted from the headgear; a processing unit configured to receive the brain wave data transmitted from the terminal as input and perform preprocessing; an analysis unit configured to perform frequency analysis on the preprocessed brain wave data; an estimation unit configured to estimate a mental state based on the data subjected to the frequency analysis by the analysis unit; A system comprising: a proposal unit configured to propose an appropriate action to an individual user; and a notification unit configured to notify a terminal of the user of a proposal result by the proposal unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Stress and mental health problems are on the rise in modern society, creating a need for appropriate care and early detection. However, conventional methods have struggled to accurately grasp each individual's mental state and propose appropriate actions in real time. As a result, many people do not receive appropriate care, and in many cases, their problems worsen. Furthermore, with limited resources at medical institutions and experts, there is a lack of easy ways for people to understand their condition and obtain countermeasures. This is the problem that needs to be solved. [Means for solving the problem]

[0005] The present invention uses headgear for collecting EEG data and a terminal that receives the EEG data transmitted from the headgear. It also includes a server processing means for receiving the EEG data transmitted from the terminal and performing preprocessing such as noise removal and filtering. It includes an analysis means for frequency-analyzing the preprocessed EEG data using a Fourier transform, and an estimation means for estimating the mental state from the analysis results. By using a suggestion means that suggests appropriate actions to each user based on the estimation results and a notification means that notifies the user terminal of the suggestion results in real time, the user can understand their own mental state in real time and take appropriate action. This enables early detection of mental health problems and appropriate care.

[0006] "Electroencephalogram data" is digital data that records the electrical activity of the brain.

[0007] "Headgear" is a device worn on the user's head to collect brainwave data.

[0008] A "terminal" is a device used by a user to receive brain wave data transmitted from the headgear.

[0009] The "processing means" is a system element that receives the electroencephalogram data transmitted from the terminal and performs pre-processing such as noise removal and filtering.

[0010] The "analysis means" is a system element for performing frequency analysis on preprocessed electroencephalogram data.

[0011] The "estimation means" is a system element that estimates the state of mind from the frequency data obtained by the analysis means.

[0012] The "suggestion means" is a system element that suggests appropriate actions to the user based on the mental state obtained by the estimation means.

[0013] The "notification means" is a system element that notifies the user terminal of the proposal results in real time. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[0036] Basic configuration

[0037] 1. Headgear

[0038] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0039] 2. Terminal

[0040] The user's smartphone, PC, or other device receives the brainwave data sent from the headgear. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[0041] 3. Server

[0042] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0043] Program processing overview

[0044] Server Processing

[0045] 1. Data Reception

[0046] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0047] 2. Data Preprocessing

[0048] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[0049] 3. Frequency analysis

[0050] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0051] 4. Mental State Estimation

[0052] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0053] 5. Proposal Generation

[0054] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[0055] 6. Notification

[0056] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[0057] Terminal handling

[0058] 1. Data Reception

[0059] The device receives real-time EEG data from the headgear, which is temporarily stored in a buffer within the device.

[0060] 2. Data Transmission

[0061] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0062] 3. Receiving and displaying notifications

[0063] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[0064] User operations

[0065] 1. Put on the headgear

[0066] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0067] 2. Feedback Check

[0068] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0069] 3. Action Practice

[0070] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0071] Specific examples

[0072] Example 1: Mental health care

[0073] To relieve work-related stress, the user wears headgear to collect brainwave data. The server analyzes the data and determines that "alpha waves are high, indicating a state of relaxation." The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[0074] Example 2: Improved concentration

[0075] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that their beta waves are high and they are in a state of concentration. The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[0076] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by using electroencephalogram data to analyze the user's mental state and suggest appropriate actions.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[0080] Step 2:

[0081] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[0082] Step 3:

[0083] The device removes noise from the EEG data stored in the buffer by applying low-pass and high-pass filters.

[0084] Step 4:

[0085] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[0086] Step 5:

[0087] The server receives the EEG data sent from the device, and the received data undergoes further preprocessing such as noise removal and filtering.

[0088] Step 6:

[0089] The server applies a Fourier transform (FFT) to the preprocessed data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0090] Step 7:

[0091] The server evaluates the intensity of each frequency band and uses this to estimate the user's mental state. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration.

[0092] Step 8:

[0093] The server generates appropriate action suggestions for the user based on the estimated mental state, for example, "You are currently in a relaxed state. Please continue to take deep breaths."

[0094] Step 9:

[0095] The server notifies the user of the generated suggestions in real time, supporting the user's actions.

[0096] Step 10:

[0097] The device receives the suggestions and displays them in an easy-to-understand format to the user. Notifications are sent via push notifications or special screens within the app.

[0098] Step 11:

[0099] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[0100] Step 12:

[0101] The user puts on the headgear again, and EEG data is collected after the behavior is performed. This data is again sent to the terminal, and the process repeats from step 1.

[0102] In this way, the system repeats the processing steps of collecting the user's brainwave data, analyzing it, notifying them of suggestions, and then providing feedback, continuously supporting the user's mental state.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Stress and decreased concentration have become problems in modern society. Conventional methods have made it difficult to quickly provide effective solutions to these problems. Therefore, the challenge is to provide a system that can analyze EEG data to grasp the user's mental state in real time and suggest appropriate actions.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a device for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the device, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, and notification means for notifying the user's terminal of the suggestion result, thereby making it possible to grasp the user's mental state in real time and suggest appropriate actions.

[0108] "EEG data" refers to digital signals obtained by detecting electrical activity in the brain.

[0109] The "device" refers to a device worn on the head to collect EEG data, and is equipped with multiple electrodes.

[0110] "Terminal" refers to an electronic device that receives the brainwave data transmitted from the device, including smartphones and PCs.

[0111] The "processing means" is a means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing such as noise removal.

[0112] The "analysis means" refers to a means for frequency-analyzing the preprocessed electroencephalogram data, and uses an analysis method such as Fourier transform.

[0113] The "estimation means" is a means for estimating the state of mind from the data frequency-analyzed by the analysis means.

[0114] The "suggestion means" is a means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means.

[0115] The "notification means" is a means for notifying the user terminal of the result of the proposal made by the proposal means.

[0116] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[0117] Basic configuration

[0118] 1. Device for collecting EEG data

[0119] The user wears a device on their head to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0120] 2. Terminal

[0121] The EEG data sent from the device is received by the user's smartphone, PC, or other device, which communicates with the device using a connection method such as Bluetooth or USB.

[0122] 3. Server

[0123] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0124] Program processing overview

[0125] Server Processing

[0126] 1. Data Reception

[0127] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0128] 2. Data Preprocessing

[0129] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it in a form suitable for analysis.

[0130] 3. Frequency analysis

[0131] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT), which categorizes it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0132] 4. Mental State Estimation

[0133] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0134] 5. Proposal Generation

[0135] The server then suggests appropriate actions to the user based on the estimated mental state. These suggestions are based on past data and general health information. For example, a generative AI model could be used to generate specific suggestions such as "Try taking deep breaths."

[0136] 6. Notification

[0137] The server notifies the user of the results in real time, either as a pop-up or in-app message.

[0138] Terminal handling

[0139] 1. Data Reception

[0140] The terminal receives EEG data from the device in real time, and the received data is temporarily stored in a buffer within the terminal.

[0141] 2. Data Transmission

[0142] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0143] 3. Receiving and displaying notifications

[0144] The device receives the proposed results sent from the server and displays them in a user-friendly format via push notifications or a special screen within the app.

[0145] User operations

[0146] 1. Wearing the device

[0147] The user first puts on the device and connects it to a smartphone or PC to begin collecting brainwave data.

[0148] 2. Feedback Check

[0149] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0150] 3. Action Practice

[0151] The user can then put the device back on to perform the suggested action and receive feedback.

[0152] Specific examples

[0153] Example 1: Mental health care

[0154] To relieve work-related stress, the user wears the device and brainwave data is collected. The server analyzes the data and determines that the user's alpha waves are high, indicating a state of relaxation. The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[0155] Example 2: Improved concentration

[0156] Students wear a device to measure their concentration levels during breaks from studying. The server analyzes the data and determines that "beta waves are high and you are in a state of concentration." The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[0157] Example input to a generative AI model

[0158] "Please suggest specific actions to relieve stress. Based on the user's brain wave data, we have determined that their alpha waves are currently high."

[0159] In this way, an embodiment of the invention provides a system that supports mental health care and improved concentration by analyzing the user's mental state in real time using electroencephalogram data and suggesting appropriate actions.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] EEG data collection

[0163] The user wears a device on their head to collect brainwave data. The device uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The input is the user's brain's electrical activity, and the output is digitized brainwave data. Specifically, the electrodes capture the brainwaves, and the built-in A / D converter converts the analog signal into digital data.

[0164] Step 2:

[0165] Transmission of brainwave data to a terminal

[0166] The device transmits the collected EEG data to the user's smartphone or PC via Bluetooth or USB connection. The input is digitized EEG data, and the output is EEG data transmitted to the terminal. Specifically, the communication module inside the device encrypts the data and transmits it to the terminal wirelessly or via a wired connection.

[0167] Step 3:

[0168] Data buffering on the device

[0169] The device temporarily stores the received EEG data in a buffer. The input is the EEG data sent from the device, and the output is the data stored in the buffer. Specifically, a dedicated app on the device runs in the background, receiving and saving data in real time.

[0170] Step 4:

[0171] Sending data to the server

[0172] The device sends the EEG data stored in the buffer to the server using a secure communication protocol. The input is the EEG data stored in the buffer, and the output is the data sent to the server. Specifically, a dedicated app on the device compiles the data and sends it to the cloud server using a protocol such as HTTPS.

[0173] Step 5:

[0174] Data reception

[0175] The server receives the EEG data sent from the device in real time. The input is the EEG data sent from the device, and the output is the data received within the server. Specifically, the server's receiving module decodes the data and stores it in storage.

[0176] Step 6:

[0177] Data Preprocessing

[0178] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it for analysis. The input is the received raw data, and the output is the noise-removed data. Specifically, a pre-processing algorithm in the server filters the data and removes unnecessary frequency components.

[0179] Step 7:

[0180] Frequency Analysis

[0181] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT). The input is noise-removed data, and the output is data classified into each frequency band. Specifically, the FFT algorithm is applied to classify the data into alpha waves, beta waves, theta waves, delta waves, gamma waves, etc.

[0182] Step 8:

[0183] Mental state estimation

[0184] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns. The input is data classified into each frequency band, and the output is an estimated mental state. Specifically, it uses a machine learning model to identify states such as relaxation, concentration, and drowsiness.

[0185] Step 9:

[0186] Proposal Generation

[0187] The server then suggests appropriate actions to the user based on the estimated mental state. The input is the estimated mental state, and the output is the suggested action. Specific operations include using a generative AI model to generate specific suggestions such as "Try taking deep breaths."

[0188] Step 10:

[0189] Proposal Notification

[0190] The server notifies the user's device of the proposed action in real time. The input is the proposed action, and the output is a notification displayed on the user's device. Specifically, the server's notification module converts the proposed action into an appropriate format and sends it to the device.

[0191] Step 11:

[0192] Check and implement user feedback

[0193] The user checks the suggested results displayed on the device and performs specific actions. The input is the notification received from the server, and the output is the change in the user's mental state as a result of the user performing the action. Specific actions include the user taking a deep breath or taking a break.

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] In modern society, it is extremely important to accurately grasp a user's psychological state in real time and provide products and services that correspond to that state. However, existing virtual store systems have difficulty accurately analyzing a user's psychological state and recommending products based on that state. In particular, they lack the ability to propose appropriate products and services that respond to subtle changes in the user's psychological state, such as whether the user is relaxed or focused. This reduces the quality of the user experience and hinders improvement in satisfaction. The present invention aims to solve these problems and provide detailed product recommendations in real time based on the user's psychological state.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes headgear for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, notification means for notifying the user's terminal of the suggestion result, and recommendation means for recommending products in a virtual store based on the mental state estimated in real time. This enables optimal product recommendations based on the user's psychological state, increasing user satisfaction and providing a personalized experience tailored to each user.

[0199] "Electroencephalogram data" is digital data that indicates the electrical activity of the brain and is collected by headgear worn on the head.

[0200] "Headgear" is a device that is worn on the user's head and collects brain wave data using multiple electrodes.

[0201] A "terminal" is a device such as a smartphone or PC that receives brainwave data sent from the headgear and sends it to a server.

[0202] The "processing means" refers to a device that receives EEG data transmitted from a terminal as input and has the function of preprocessing the data using noise removal, low-pass filters, high-pass filters, etc.

[0203] The "analysis means" is a mechanism used to perform frequency analysis on preprocessed EEG data, specifically, a mechanism that has the function of classifying the data into frequency bands using a Fourier transform (FFT).

[0204] The "estimation means" has a function of estimating the user's mental state from the data frequency-analyzed by the analysis means.

[0205] The "suggestion means" has a function of suggesting appropriate actions to each user based on the mental state estimated by the estimation means.

[0206] The "notification means" has a function of notifying the user terminal of the result of the proposal made by the proposal means.

[0207] The "recommendation means" has the function of recommending products within a virtual store based on the results of real-time estimation of the user's mental state.

[0208] This invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. This system is mainly composed of headgear, a terminal, and a server.

[0209] Basic configuration

[0210] 1. Headgear

[0211] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0212] 2. Terminal

[0213] The brainwave data sent from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB. The device also has the role of sending the received data to a server.

[0214] 3. Server

[0215] The server receives the EEG data sent from the device and performs various processing. It is installed on the cloud and uses a high-performance processor to preprocess and analyze the data. The server-side processing mainly consists of the following steps.

[0216] Server Processing

[0217] 1. Data Reception

[0218] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0219] 2. Data Preprocessing

[0220] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it for analysis. Signal processing libraries such as SciPy are used here.

[0221] 3. Frequency analysis

[0222] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0223] 4. Mental State Estimation

[0224] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0225] 5. Proposal Generation

[0226] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[0227] 6. Notification

[0228] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[0229] User operations

[0230] 1. Put on the headgear

[0231] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0232] 2. Feedback Check

[0233] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0234] 3. Action Practice

[0235] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0236] Specific examples

[0237] As a specific example, by introducing this system into a virtual store, if a user is relaxed, it can recommend products that have a relaxing effect, and if a user is concentrating, it can suggest products that will further enhance concentration. This allows users to select products that best suit their own psychological state, providing a more satisfying shopping experience.

[0238] Prompt Sentence Examples

[0239] While a user is moving around in a virtual store, analyze their brainwave data in real time. If it is determined that the user is in a relaxed state, recommend products that have a relaxing effect to the user.

[0240] In this way, the present invention can realize detailed product recommendations based on the user's psychological state, improving the quality of the user's experience in a virtual store.

[0241] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0242] Step 1:

[0243] Headgear installation and data collection

[0244] The user wears a headgear to collect EEG data. The headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The output data from the headgear is then sent to a terminal.

[0245] Input: User's electrical brain activity

[0246] Output: Digital data sent from the headgear to the device

[0247] Step 2:

[0248] Receiving data

[0249] The device receives the EEG data transmitted from the headgear in real time, and the received data is temporarily stored in a buffer within the device.

[0250] Input: Digital data transmitted from the headgear

[0251] Output: EEG data stored in a temporary buffer on the device

[0252] Step 3:

[0253] Sending data

[0254] The device transmits the buffered EEG data to a server at regular intervals using a secure communication protocol (e.g., HTTPS).

[0255] Input: EEG data stored in a temporary buffer on the device

[0256] Output: EEG data sent to the server

[0257] Step 4:

[0258] Data reception and noise reduction

[0259] The server receives the EEG data sent from the device in real time. Because the received data may contain noise, it is first subjected to noise removal. Specifically, low-pass and high-pass filters are used to remove unnecessary frequency components.

[0260] Input: EEG data sent from the device

[0261] Output: Denoised data

[0262] Step 5:

[0263] Frequency Analysis

[0264] The pre-processed (noise-removed) EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies the data into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0265] Input: Denoised data

[0266] Output: Data classified into each frequency band

[0267] Step 6:

[0268] Mental state estimation

[0269] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's state of mind based on specific patterns. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration. AI models may be used for this estimation.

[0270] Input: Data classified into each frequency band

[0271] Output: Estimated state of mind

[0272] Step 7:

[0273] Proposal Generation

[0274] Based on the estimated mental state, the system suggests appropriate actions to the user. For example, if the user is in a relaxed state, it will recommend relaxation products, and if the user is in a concentrated state, it will recommend products that will improve concentration. These suggestions are generated on the server side.

[0275] Input: Inferred state of mind

[0276] Output: Suggested actions or products

[0277] Step 8:

[0278] Notifications and Recommendations

[0279] The results of the recommendations are sent to the user's device in real time as pop-ups or in-app messages, and product recommendations are also made in the virtual store.

[0280] Input: Suggested action or product

[0281] Output: Notification to user device and product recommendations

[0282] Step 9:

[0283] Reviewing feedback

[0284] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. We recommend the relaxation goods section" is displayed.

[0285] Input: Notification to user terminal

[0286] Output: User confirms the proposal

[0287] Step 10:

[0288] Action implementation and additional data collection

[0289] The user performs the suggested action and receives feedback by putting the headgear back on to collect additional EEG data, a process that allows the system to make even more personalized suggestions.

[0290] Input: Additional EEG data after user action

[0291] Output: Collected data on new headgear worn

[0292] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0293] The present invention is a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and suggests appropriate actions, and specific embodiments thereof will be described below.

[0294] Basic configuration

[0295] 1. Headgear

[0296] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0297] 2. Terminal

[0298] The brainwave data transmitted from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[0299] 3. Server

[0300] A server that receives and processes EEG data sent from the device. This server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0301] Program processing overview

[0302] Server Processing

[0303] 1. Data Reception

[0304] The server receives the EEG data sent from the device in real time, and the received data is preprocessed to remove noise.

[0305] 2. Data Preprocessing

[0306] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[0307] 3. Frequency analysis

[0308] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0309] 4. Emotional state estimation using emotion engine

[0310] Based on the frequency analysis results, an emotion engine is used to estimate the user's emotional state from the EEG data. This emotion engine performs emotion estimation based on a trained model using a machine learning algorithm.

[0311] 5. Mental State Estimation

[0312] The frequency-analyzed data is used to estimate the user's mental state (relaxed, focused, excited, etc.) along with their emotional state.

[0313] 6. Proposal Generation

[0314] Based on the estimated mental and emotional state, the system generates appropriate action suggestions for the user, such as "You are currently in a relaxed state, so please continue to take deep breaths" or "You are feeling stressed, so please take a short walk."

[0315] 7. Notification

[0316] The results of the suggestions are sent to the user's device in real time via push notifications or in-app messages.

[0317] Terminal handling

[0318] 1. Data Reception

[0319] The device receives real-time brainwave data from the headgear and temporarily stores it in a buffer.

[0320] 2. Data Transmission

[0321] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0322] 3. Receiving and displaying notifications

[0323] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[0324] User operations

[0325] 1. Put on the headgear

[0326] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0327] 2. Feedback Check

[0328] The user can check the analysis results and suggestions sent from the server on their device. For example, advice such as "You are currently in a relaxed state" or "You are feeling stressed. Relax" will be displayed.

[0329] 3. Action Practice

[0330] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0331] Specific examples

[0332] Example 1: Mental health care

[0333] To relieve work-related stress, the user wears headgear to collect brain wave data. The server analyzes the data and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[0334] Example 2: Improved concentration

[0335] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[0336] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by analyzing the user's mental state using brain wave data and emotional state and suggesting appropriate actions.

[0337] The processing flow will be explained below.

[0338] Step 1:

[0339] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[0340] Step 2:

[0341] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[0342] Step 3:

[0343] The device preprocesses the EEG data stored in the buffer, specifically by applying low-pass and high-pass filters to remove noise.

[0344] Step 4:

[0345] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[0346] Step 5:

[0347] The server receives the EEG data sent from the device, and then performs pre-processing such as filtering and noise removal on the received data.

[0348] Step 6:

[0349] The server applies a Fourier transform (FFT) to the preprocessed EEG data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0350] Step 7:

[0351] The server evaluates the data from each frequency band and uses this to estimate the user's mental state (relaxed, focused, excited, etc.).

[0352] Step 8:

[0353] The server then uses an emotion engine to estimate the user's emotional state (happiness, anxiety, stress, etc.) from the EEG data. The emotion engine uses machine learning algorithms to make predictions based on a trained model.

[0354] Step 9:

[0355] The server generates appropriate action suggestions for the user based on the estimated mental and emotional state, such as "You are currently in a relaxed state. Please continue to take deep breaths" or "You are feeling stressed. It would be good to take a short walk."

[0356] Step 10:

[0357] The server generates suggestions and sends them to the user's device in real time, either as push notifications or in-app messages.

[0358] Step 11:

[0359] The device receives the suggestions and displays them in a format that is easy for the user to understand. Notifications are sent via push notifications or special screens within the app.

[0360] Step 12:

[0361] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[0362] Step 13:

[0363] The user puts on the headgear again and collects EEG data after performing the action. This allows the server to analyze the effect of the user's action again and provide feedback. This data is again sent to the device, where it is analyzed by the server, and the process from step 1 is repeated.

[0364] In this way, the system repeats processing steps from collecting the user's brainwave data to estimating their emotional state, notifying them of suggested actions, and providing feedback on the results of their actions, thereby continuously supporting the user's mental and emotional state.

[0365] Example 2

[0366] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0367] The present invention aims to provide a system that accurately grasps a user's mental and emotional state based on electroencephalogram (EEG) data and provides appropriate action suggestions in real time based on the results. Existing technologies have problems such as inaccurate analysis of EEG data and action suggestions, and difficulty in real-time notification. In particular, the objective is to solve the problem of low accuracy in noise removal and emotion estimation, making it difficult to provide accurate feedback to the user.

[0368] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0369] In this invention, the server includes a device for collecting electroencephalogram data, a communication device for receiving the electroencephalogram data transmitted from the device, a calculation device for receiving the electroencephalogram data transmitted from the communication device as input and performing preprocessing, a computation device for frequency analyzing the preprocessed electroencephalogram data, an estimation device for estimating a mental state from the data frequency analyzed by the computation device, a suggestion device for making appropriate action suggestions to individual users based on the mental state estimated by the estimation device, and a notification device for notifying the user's communication device of the suggestion results. This makes it possible to analyze noise-removed electroencephalogram data in real time, accurately estimate the mental state and emotional state, and provide appropriate action suggestions to users in real time.

[0370] "Electroencephalogram data" is information that represents in digital form signals that measure the electrical activity of the brain.

[0371] "Device" refers to equipment including headgear and sensors worn on the user's head to collect brainwave data.

[0372] A "communication device" is a device such as a smartphone or PC that receives the brainwave data sent from the device and sends it to a server.

[0373] A "computing device" is a device that includes a computer or processor for pre-processing electroencephalogram data received from a communication device.

[0374] The "arithmetic unit" is a device for frequency analysis of preprocessed electroencephalogram data, and executes algorithms such as Fourier transform.

[0375] The "estimation device" is a device for estimating the mental state of a user from frequency analysis data obtained by a computing device, and uses a machine learning algorithm.

[0376] The "suggestion device" is a device that suggests appropriate actions to a user based on the mental state estimated by the estimation device.

[0377] The "notification device" is a device that notifies the user's communication device of the proposal results from the proposal device in real time.

[0378] The present invention provides a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and proposes appropriate actions. This system is implemented using the following main hardware and software components.

[0379] 1. Headgear

[0380] The user wears a headgear to collect EEG data. This headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs the signals as digital data, allowing for accurate collection of EEG data.

[0381] 2. Terminal

[0382] The brainwave data sent from the headgear is received by a device such as a smartphone or PC via a connection method such as Bluetooth or USB. The device temporarily stores the received data in a buffer and then transmits it to a server using a secure communication protocol.

[0383] 3. Server

[0384] The server receives the EEG data sent from the device in real time and performs the following processing. First, it applies a low-pass filter and a high-pass filter to remove noise. Next, it uses a Fourier transform (FFT) on the preprocessed EEG data to perform frequency analysis. This allows it to be classified into various frequency bands, such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0385] Furthermore, an emotion engine is used to estimate the user's emotional state based on the analysis results. The emotion engine uses a machine learning algorithm to estimate emotions based on a model. Then, based on the estimated emotional state and frequency analysis data, the user's mental state (relaxed, focused, excited, etc.) is evaluated.

[0386] Based on these evaluation results, the server generates appropriate action suggestions for the user, such as "Keep taking deep breaths" or "Take a short walk." The generated suggestions are sent to the user's device in real time as push notifications or in-app messages.

[0387] Specific examples

[0388] Example 1: Mental health care

[0389] When a user puts on the headgear to relieve work stress, the server analyzes the situation and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[0390] Example 2: Improved concentration

[0391] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[0392] Prompt Sentence Examples

[0393] Sample prompt 1: "After a user wears the headgear and collects EEG data, explain how that data can be analyzed to support mental health care."

[0394] Sample prompt 2: "Please explain with a concrete example how EEG data can be used to generate appropriate action suggestions to improve a user's focus."

[0395] As described above, the present invention provides a specific embodiment that supports mental health care and improved concentration by analyzing the user's mental state using electroencephalogram data and emotional state and suggesting appropriate actions.

[0396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0397] Step 1: Receiving data

[0398] The server receives the EEG data transmitted from the device in real time. As input, the EEG data acquired by the device from the headgear is used. The received data is passed to a pre-processing step for noise reduction. Specifically, the data is temporarily stored in the server's receiving buffer.

[0399] Step 2: Preprocessing the data

[0400] The server preprocesses the received EEG data. In this step, noise is removed by applying low-pass and high-pass filters. The EEG data received in step 1 is used as input, and noise-removed EEG data is obtained as output. Specifically, a fast moving average algorithm is used to reduce noise in the signal.

[0401] Step 3: Frequency analysis

[0402] The server performs a Fourier transform (FFT) on the preprocessed EEG data. The input is the data from which noise was removed in step 2, and the output is data classified into frequency bands such as alpha, beta, theta, delta, and gamma waves. Specifically, the FFT algorithm is run to quantify the intensity of each frequency band.

[0403] Step 4: Estimating emotional state

[0404] The server uses an emotion engine to estimate the user's emotional state based on the results of the frequency analysis. The frequency data classified in step 3 is used as input, and the emotional state, such as relaxation, excitement, or stress, is obtained as output. Specifically, the emotion engine, which uses a machine learning algorithm, estimates the emotional state based on the model.

[0405] Step 5: Mental state estimation

[0406] The server estimates the user's mental state based on the emotional state and the frequency analysis results. The emotional state data from step 4 and the frequency data from step 3 are used as input, and the specific mental state (relaxed, focused, excited, etc.) is obtained as output. Specifically, the mental state is evaluated multidimensionally by combining multiple data indicators.

[0407] Step 6: Generate proposals

[0408] The server generates action suggestions for the user based on the estimated mental state. The mental state data obtained in step 5 is used as input, and specific action suggestions (e.g., "Continue to take deep breaths" or "Take a short break") are obtained as output. Specifically, suggestions are automatically generated according to predefined suggestion generation rules.

[0409] Step 7: Notification

[0410] The server notifies the generated action suggestions to the user's device in real time. The suggestion data generated in step 6 is used as input, and a notification message is sent to the user's device as output. Specifically, the server uses push notification and in-app messaging functions to provide instant feedback to the user.

[0411] Step 8: Put on the headgear

[0412] The user puts on the headgear and connects it to the terminal. The input is to confirm that the headgear is attached according to a specific procedure, and the output is to start collecting brainwave data. Specifically, the headgear's sensors measure the electrical activity of the brainwaves in real time and transmit the data to the terminal.

[0413] Step 9: Feedback confirmation

[0414] The user checks the suggestion displayed on the device and decides on the next action. The suggestion message notified in step 7 is used as input, and the action to be taken by the user is determined as output. Specifically, the device screen displays "Please continue to take deep breaths," and the user follows the advice.

[0415] Step 10: Take Action

[0416] The user puts on the headgear again to practice the suggested behavior and get feedback. The behavior determined in step 9 is used as input, and results such as improved performance or mood are obtained as output. Specifically, the user actually takes a deep breath, and EEG data is collected again to confirm the effect.

[0417] (Application example 2)

[0418] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0419] Systems already exist that use EEG data to understand a user's mental and emotional state and suggest appropriate actions. However, these systems are primarily limited to personal use, mental health care, and improving concentration, and have not been applied to improving customer service in brick-and-mortar stores. Another issue is that if suggestions are not made in real time, it is difficult to improve customer satisfaction.

[0420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0421] In this invention, the server includes headgear for collecting electroencephalogram data, an information terminal device for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the information terminal device as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means, notification means for notifying the user's information terminal device of the suggestion results, and suggestion means for suggesting appropriate services and products in real time based on the mental state and emotional state of the customer using an electroencephalogram data collection device temporarily worn by the customer in a physical store. This enables personalized service offerings based on the customer's emotional and mental state in a physical store.

[0422] "Electroencephalogram data" is information that measures the electrical activity of the brain and expresses it in digital form.

[0423] "Headgear" is a device worn on the head to collect the wearer's brain wave data.

[0424] An "information terminal device" is a device for receiving and processing brain wave data transmitted from the headgear, and includes smartphones, personal computers, etc.

[0425] The "processing means" is a means having a function for receiving and preprocessing electroencephalogram data transmitted from an information terminal device.

[0426] The "analysis means" is a mechanism for performing frequency analysis on preprocessed electroencephalogram data.

[0427] "Frequency analysis" is an analytical method that divides EEG data into multiple frequency bands, and uses techniques such as Fourier transform.

[0428] The "estimation means" is a means for estimating a mental state from data frequency-analyzed by the analysis means.

[0429] The "suggestion means" is a means for suggesting an appropriate action to the user based on the mental state estimated by the estimation means.

[0430] The "notification means" is a mechanism for notifying the user's information terminal device of the result of the proposal made by the proposal means.

[0431] "Brick and mortar store" refers to a physical store where customers visit to purchase products.

[0432] An "electroencephalogram data collection device" is a temporarily wearable device for collecting electroencephalogram data.

[0433] A "generative AI model" is an artificial intelligence model that has been trained to perform a specific task using machine learning algorithms.

[0434] A "prompt sentence" is a sentence of guidance or advice that is generated based on the user's mental and emotional state.

[0435] MODE FOR CARRYING OUT THE INVENTION

[0436] The present invention provides a system that analyzes electroencephalogram data collected from a user wearing a headgear, and aims to improve the customer experience, particularly in physical stores. Specific embodiments are described below.

[0437] System configuration

[0438] The system consists of the following main components:

[0439] 1. Headgear

[0440] This is a device that is temporarily worn by the user (customer) to collect brain wave data. It acquires brain wave data in real time and transmits it to an information terminal device.

[0441] 2. Information terminal device

[0442] Mobile devices such as smartphones and tablets are used to receive EEG data sent from the headgear and send it to a server.

[0443] 3. Server

[0444] The cloud-based system uses a high-performance processor to preprocess and analyze EEG data, specifically removing noise from the data, analyzing frequencies, estimating mental and emotional states, and generating appropriate action suggestions, which are then sent to the information terminal device.

[0445] Program processing overview

[0446] The server is equipped with the following processing means:

[0447] 1. Data Reception

[0448] The server receives the EEG data sent from the information terminal device in real time, and the data is first temporarily stored in storage.

[0449] 2. Noise Reduction and Preprocessing

[0450] Low-pass and high-pass filters are used to remove noise from the received EEG data, resulting in clean data suitable for analysis.

[0451] 3. Frequency analysis

[0452] The preprocessed EEG data is then classified by frequency using a Fourier transform (FFT), and waves that reflect psychological states, such as alpha waves, beta waves, and theta waves, are identified.

[0453] 4. Estimating Mental and Emotional States

[0454] Mental and emotional states are estimated from the analysis results using an estimation method based on a generative AI model using machine learning algorithms.

[0455] 5. Generating action suggestions

[0456] Based on the estimated mental and emotional state, appropriate action suggestions (e.g., information about products and services that will help customers relax) are generated.

[0457] 6. Notification

[0458] The generated suggestions are sent to the information terminal device in real time via push notifications or in-app messages.

[0459] Example

[0460] A specific example is given below.

[0461] Example 1: Proposing a relaxing environment

[0462] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[0463] The server analyzes the data and estimates the state of relaxation. Based on the generative AI model, it generates a prompt in real time, such as "You are in a relaxed state. You can use the relaxation area."

[0464] The user's information terminal device is notified.

[0465] Example 2: Proposing a stress-relieving product

[0466] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[0467] The server analyzes the data and estimates the state of stress. Based on the generative AI model, it generates a prompt in real time, such as "You are in a stressful state. Please try some stress relief products."

[0468] The user's information terminal device is notified.

[0469] Prompt Sentence Examples

[0470] An example of a prompt sentence to input to the generative AI model is as follows:

[0471] plaintext

[0472] A customer puts on smart glasses and enters a store. Analyze their brainwave data to estimate their current emotional and mental state. Based on the estimation results, generate a message suggesting appropriate services and products. For example, if the customer is in a relaxed state, introduce them to a relaxation area, or if they are in a stressed state, suggest stress-relieving products.

[0473] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0474] Step 1:

[0475] The user puts on the headgear and establishes a connection with the information terminal device.

[0476] Input: EEG data from the headgear

[0477] Output: Raw EEG data sent to an information terminal device

[0478] Specific operation: Electrodes in the headgear detect brain waves and transmit the data to an information terminal device via Bluetooth or Wi-Fi.

[0479] Step 2:

[0480] The terminal receives the brainwave data transmitted from the headgear and temporarily stores it in a buffer.

[0481] Input: Raw EEG data transmitted from the headgear

[0482] Output: Buffered EEG data for sending to the server

[0483] Specific operation: The information terminal device receives data packets from the headgear and buffers a certain amount of brain wave data.

[0484] Step 3:

[0485] The device sends the buffered EEG data to a server, where the data is transferred using a secure communication protocol.

[0486] Input: EEG data stored in a buffer

[0487] Output: EEG data sent to the server

[0488] Specific operation: The information terminal device uploads the buffered data to the server using the HTTPS or SSH protocol.

[0489] Step 4:

[0490] The server receives the brainwave data in real time and temporarily stores it in storage.

[0491] Input: EEG data sent from the device

[0492] Output: Saved EEG data

[0493] Specific operation: The server uses the data receiving module to continuously receive streams of data from the terminal and store them in storage.

[0494] Step 5:

[0495] The server denoises and preprocesses the EEG data, using low-pass and high-pass filters to remove noise and generate clean data.

[0496] Input: Stored EEG data

[0497] Output: Noise-removed EEG data

[0498] What it does: Preprocessing algorithms remove high- and low-frequency noise to prepare data suitable for analysis.

[0499] Step 6:

[0500] The server performs frequency analysis on the preprocessed EEG data, classifying it into frequency bands using a Fourier transform (FFT).

[0501] Input: EEG data after noise removal

[0502] Output: EEG data categorized by frequency

[0503] Specific operation: Runs the FFT algorithm to decompose brainwave data into alpha waves, beta waves, theta waves, etc.

[0504] Step 7:

[0505] The server uses the frequency analysis results to estimate the mental and emotional state, and utilizes a generative AI model.

[0506] Input: Frequency analysis results

[0507] Output: Inferred mental and emotional state

[0508] Specific operation: The analysis results are input into a generative AI model, which estimates mental state (relaxed, stressed, etc.) and emotional state (happiness, anxiety, etc.) based on a trained algorithm.

[0509] Step 8:

[0510] Based on the estimation results, the server generates appropriate action suggestions and creates prompt sentences.

[0511] Input: Inferred mental and emotional states

[0512] Output: Action suggestion prompt

[0513] Specific operation: A rule-based engine is activated to provide action suggestions based on the inference results, generating prompt statements such as "You are in a relaxed state. You can use the relaxation area."

[0514] Step 9:

[0515] The server notifies the device of the proposed results in the form of a push notification or an in-app message.

[0516] Input: Prompt for suggested action

[0517] Output: Proposal message displayed on the terminal

[0518] Specific operation: The server uses the notification system to send the generated prompt text to the terminal and notify the user.

[0519] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0520] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0521] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0522] [Second embodiment]

[0523] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0524] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0525] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0526] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0527] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0528] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0529] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0530] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0531] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0532] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0533] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0534] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0535] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[0536] Basic configuration

[0537] 1. Headgear

[0538] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0539] 2. Terminal

[0540] The user's smartphone, PC, or other device receives the brainwave data sent from the headgear. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[0541] 3. Server

[0542] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0543] Program processing overview

[0544] Server Processing

[0545] 1. Data Reception

[0546] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0547] 2. Data Preprocessing

[0548] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[0549] 3. Frequency analysis

[0550] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0551] 4. Mental State Estimation

[0552] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0553] 5. Proposal Generation

[0554] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[0555] 6. Notification

[0556] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[0557] Terminal handling

[0558] 1. Data Reception

[0559] The device receives real-time EEG data from the headgear, which is temporarily stored in a buffer within the device.

[0560] 2. Data Transmission

[0561] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0562] 3. Receiving and displaying notifications

[0563] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[0564] User operations

[0565] 1. Put on the headgear

[0566] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0567] 2. Feedback Check

[0568] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0569] 3. Action Practice

[0570] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0571] Specific examples

[0572] Example 1: Mental health care

[0573] To relieve work-related stress, the user wears headgear to collect brainwave data. The server analyzes the data and determines that "alpha waves are high, indicating a state of relaxation." The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[0574] Example 2: Improved concentration

[0575] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that their beta waves are high and they are in a state of concentration. The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[0576] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by using electroencephalogram data to analyze the user's mental state and suggest appropriate actions.

[0577] The processing flow will be explained below.

[0578] Step 1:

[0579] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[0580] Step 2:

[0581] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[0582] Step 3:

[0583] The device removes noise from the EEG data stored in the buffer by applying low-pass and high-pass filters.

[0584] Step 4:

[0585] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[0586] Step 5:

[0587] The server receives the EEG data sent from the device, and the received data undergoes further preprocessing such as noise removal and filtering.

[0588] Step 6:

[0589] The server applies a Fourier transform (FFT) to the preprocessed data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0590] Step 7:

[0591] The server evaluates the intensity of each frequency band and uses this to estimate the user's mental state. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration.

[0592] Step 8:

[0593] The server generates appropriate action suggestions for the user based on the estimated mental state, for example, "You are currently in a relaxed state. Please continue to take deep breaths."

[0594] Step 9:

[0595] The server notifies the user of the generated suggestions in real time, supporting the user's actions.

[0596] Step 10:

[0597] The device receives the suggestions and displays them in an easy-to-understand format to the user. Notifications are sent via push notifications or special screens within the app.

[0598] Step 11:

[0599] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[0600] Step 12:

[0601] The user puts on the headgear again, and EEG data is collected after the behavior is performed. This data is again sent to the terminal, and the process repeats from step 1.

[0602] In this way, the system repeats the processing steps of collecting the user's brainwave data, analyzing it, notifying them of suggestions, and then providing feedback, continuously supporting the user's mental state.

[0603] Example 1

[0604] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Stress and decreased concentration have become problems in modern society. Conventional methods have made it difficult to quickly provide effective solutions to these problems. Therefore, the challenge is to provide a system that can analyze EEG data to grasp the user's mental state in real time and suggest appropriate actions.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0607] In this invention, the server includes a device for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the device, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, and notification means for notifying the user's terminal of the suggestion result, thereby making it possible to grasp the user's mental state in real time and suggest appropriate actions.

[0608] "EEG data" refers to digital signals obtained by detecting electrical activity in the brain.

[0609] The "device" refers to a device worn on the head to collect EEG data, and is equipped with multiple electrodes.

[0610] "Terminal" refers to an electronic device that receives the brainwave data transmitted from the device, including smartphones and PCs.

[0611] The "processing means" is a means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing such as noise removal.

[0612] The "analysis means" refers to a means for frequency-analyzing the preprocessed electroencephalogram data, and uses an analysis method such as Fourier transform.

[0613] The "estimation means" is a means for estimating the state of mind from the data frequency-analyzed by the analysis means.

[0614] The "suggestion means" is a means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means.

[0615] The "notification means" is a means for notifying the user terminal of the result of the proposal made by the proposal means.

[0616] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[0617] Basic configuration

[0618] 1. Device for collecting EEG data

[0619] The user wears a device on their head to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0620] 2. Terminal

[0621] The EEG data sent from the device is received by the user's smartphone, PC, or other device, which communicates with the device using a connection method such as Bluetooth or USB.

[0622] 3. Server

[0623] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0624] Program processing overview

[0625] Server Processing

[0626] 1. Data Reception

[0627] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0628] 2. Data Preprocessing

[0629] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it in a form suitable for analysis.

[0630] 3. Frequency analysis

[0631] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT), which categorizes it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0632] 4. Mental State Estimation

[0633] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0634] 5. Proposal Generation

[0635] The server then suggests appropriate actions to the user based on the estimated mental state. These suggestions are based on past data and general health information. For example, a generative AI model could be used to generate specific suggestions such as "Try taking deep breaths."

[0636] 6. Notification

[0637] The server notifies the user of the results in real time, either as a pop-up or in-app message.

[0638] Terminal handling

[0639] 1. Data Reception

[0640] The terminal receives EEG data from the device in real time, and the received data is temporarily stored in a buffer within the terminal.

[0641] 2. Data Transmission

[0642] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0643] 3. Receiving and displaying notifications

[0644] The device receives the proposed results sent from the server and displays them in a user-friendly format via push notifications or a special screen within the app.

[0645] User operations

[0646] 1. Wearing the device

[0647] The user first puts on the device and connects it to a smartphone or PC to begin collecting brainwave data.

[0648] 2. Feedback Check

[0649] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0650] 3. Action Practice

[0651] The user can then put the device back on to perform the suggested action and receive feedback.

[0652] Specific examples

[0653] Example 1: Mental health care

[0654] To relieve work-related stress, the user wears the device and brainwave data is collected. The server analyzes the data and determines that the user's alpha waves are high, indicating a state of relaxation. The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[0655] Example 2: Improved concentration

[0656] Students wear a device to measure their concentration levels during breaks from studying. The server analyzes the data and determines that "beta waves are high and you are in a state of concentration." The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[0657] Example input to a generative AI model

[0658] "Please suggest specific actions to relieve stress. Based on the user's brain wave data, we have determined that their alpha waves are currently high."

[0659] In this way, an embodiment of the invention provides a system that supports mental health care and improved concentration by analyzing the user's mental state in real time using electroencephalogram data and suggesting appropriate actions.

[0660] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0661] Step 1:

[0662] EEG data collection

[0663] The user wears a device on their head to collect brainwave data. The device uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The input is the user's brain's electrical activity, and the output is digitized brainwave data. Specifically, the electrodes capture the brainwaves, and the built-in A / D converter converts the analog signal into digital data.

[0664] Step 2:

[0665] Transmission of brainwave data to a terminal

[0666] The device transmits the collected EEG data to the user's smartphone or PC via Bluetooth or USB connection. The input is digitized EEG data, and the output is EEG data transmitted to the terminal. Specifically, the communication module inside the device encrypts the data and transmits it to the terminal wirelessly or via a wired connection.

[0667] Step 3:

[0668] Data buffering on the device

[0669] The device temporarily stores the received EEG data in a buffer. The input is the EEG data sent from the device, and the output is the data stored in the buffer. Specifically, a dedicated app on the device runs in the background, receiving and saving data in real time.

[0670] Step 4:

[0671] Sending data to the server

[0672] The device sends the EEG data stored in the buffer to the server using a secure communication protocol. The input is the EEG data stored in the buffer, and the output is the data sent to the server. Specifically, a dedicated app on the device compiles the data and sends it to the cloud server using a protocol such as HTTPS.

[0673] Step 5:

[0674] Data reception

[0675] The server receives the EEG data sent from the device in real time. The input is the EEG data sent from the device, and the output is the data received within the server. Specifically, the server's receiving module decodes the data and stores it in storage.

[0676] Step 6:

[0677] Data Preprocessing

[0678] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it for analysis. The input is the received raw data, and the output is the noise-removed data. Specifically, a pre-processing algorithm in the server filters the data and removes unnecessary frequency components.

[0679] Step 7:

[0680] Frequency Analysis

[0681] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT). The input is noise-removed data, and the output is data classified into each frequency band. Specifically, the FFT algorithm is applied to classify the data into alpha waves, beta waves, theta waves, delta waves, gamma waves, etc.

[0682] Step 8:

[0683] Mental state estimation

[0684] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns. The input is data classified into each frequency band, and the output is an estimated mental state. Specifically, it uses a machine learning model to identify states such as relaxation, concentration, and drowsiness.

[0685] Step 9:

[0686] Proposal Generation

[0687] The server then suggests appropriate actions to the user based on the estimated mental state. The input is the estimated mental state, and the output is the suggested action. Specific operations include using a generative AI model to generate specific suggestions such as "Try taking deep breaths."

[0688] Step 10:

[0689] Proposal Notification

[0690] The server notifies the user's device of the proposed action in real time. The input is the proposed action, and the output is a notification displayed on the user's device. Specifically, the server's notification module converts the proposed action into an appropriate format and sends it to the device.

[0691] Step 11:

[0692] Check and implement user feedback

[0693] The user checks the suggested results displayed on the device and performs specific actions. The input is the notification received from the server, and the output is the change in the user's mental state as a result of the user performing the action. Specific actions include the user taking a deep breath or taking a break.

[0694] (Application example 1)

[0695] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] In modern society, it is extremely important to accurately grasp a user's psychological state in real time and provide products and services that correspond to that state. However, existing virtual store systems have difficulty accurately analyzing a user's psychological state and recommending products based on that state. In particular, they lack the ability to propose appropriate products and services that respond to subtle changes in the user's psychological state, such as whether the user is relaxed or focused. This reduces the quality of the user experience and hinders improvement in satisfaction. The present invention aims to solve these problems and provide detailed product recommendations in real time based on the user's psychological state.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0698] In this invention, the server includes headgear for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, notification means for notifying the user's terminal of the suggestion result, and recommendation means for recommending products in a virtual store based on the mental state estimated in real time. This enables optimal product recommendations based on the user's psychological state, increasing user satisfaction and providing a personalized experience tailored to each user.

[0699] "Electroencephalogram data" is digital data that indicates the electrical activity of the brain and is collected by headgear worn on the head.

[0700] "Headgear" is a device that is worn on the user's head and collects brain wave data using multiple electrodes.

[0701] A "terminal" is a device such as a smartphone or PC that receives brainwave data sent from the headgear and sends it to a server.

[0702] The "processing means" refers to a device that receives EEG data transmitted from a terminal as input and has the function of preprocessing the data using noise removal, low-pass filters, high-pass filters, etc.

[0703] The "analysis means" is a mechanism used to perform frequency analysis on preprocessed EEG data, specifically, a mechanism that has the function of classifying the data into frequency bands using a Fourier transform (FFT).

[0704] The "estimation means" has a function of estimating the user's mental state from the data frequency-analyzed by the analysis means.

[0705] The "suggestion means" has a function of suggesting appropriate actions to each user based on the mental state estimated by the estimation means.

[0706] The "notification means" has a function of notifying the user terminal of the result of the proposal made by the proposal means.

[0707] The "recommendation means" has the function of recommending products within a virtual store based on the results of real-time estimation of the user's mental state.

[0708] This invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. This system is mainly composed of headgear, a terminal, and a server.

[0709] Basic configuration

[0710] 1. Headgear

[0711] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0712] 2. Terminal

[0713] The brainwave data sent from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB. The device also has the role of sending the received data to a server.

[0714] 3. Server

[0715] The server receives the EEG data sent from the device and performs various processing. It is installed on the cloud and uses a high-performance processor to preprocess and analyze the data. The server-side processing mainly consists of the following steps.

[0716] Server Processing

[0717] 1. Data Reception

[0718] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[0719] 2. Data Preprocessing

[0720] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it for analysis. Signal processing libraries such as SciPy are used here.

[0721] 3. Frequency analysis

[0722] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0723] 4. Mental State Estimation

[0724] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[0725] 5. Proposal Generation

[0726] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[0727] 6. Notification

[0728] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[0729] User operations

[0730] 1. Put on the headgear

[0731] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0732] 2. Feedback Check

[0733] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[0734] 3. Action Practice

[0735] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0736] Specific examples

[0737] As a specific example, by introducing this system into a virtual store, if a user is relaxed, it can recommend products that have a relaxing effect, and if a user is concentrating, it can suggest products that will further enhance concentration. This allows users to select products that best suit their own psychological state, providing a more satisfying shopping experience.

[0738] Prompt Sentence Examples

[0739] While a user is moving around in a virtual store, analyze their brainwave data in real time. If it is determined that the user is in a relaxed state, recommend products that have a relaxing effect to the user.

[0740] In this way, the present invention can realize detailed product recommendations based on the user's psychological state, improving the quality of the user's experience in a virtual store.

[0741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] Headgear installation and data collection

[0744] The user wears a headgear to collect EEG data. The headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The output data from the headgear is then sent to a terminal.

[0745] Input: User's electrical brain activity

[0746] Output: Digital data sent from the headgear to the device

[0747] Step 2:

[0748] Receiving data

[0749] The device receives the EEG data transmitted from the headgear in real time, and the received data is temporarily stored in a buffer within the device.

[0750] Input: Digital data transmitted from the headgear

[0751] Output: EEG data stored in a temporary buffer on the device

[0752] Step 3:

[0753] Sending data

[0754] The device transmits the buffered EEG data to a server at regular intervals using a secure communication protocol (e.g., HTTPS).

[0755] Input: EEG data stored in a temporary buffer on the device

[0756] Output: EEG data sent to the server

[0757] Step 4:

[0758] Data reception and noise reduction

[0759] The server receives the EEG data sent from the device in real time. Because the received data may contain noise, it is first subjected to noise removal. Specifically, low-pass and high-pass filters are used to remove unnecessary frequency components.

[0760] Input: EEG data sent from the device

[0761] Output: Denoised data

[0762] Step 5:

[0763] Frequency Analysis

[0764] The pre-processed (noise-removed) EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies the data into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0765] Input: Denoised data

[0766] Output: Data classified into each frequency band

[0767] Step 6:

[0768] Mental state estimation

[0769] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's state of mind based on specific patterns. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration. AI models may be used for this estimation.

[0770] Input: Data classified into each frequency band

[0771] Output: Estimated state of mind

[0772] Step 7:

[0773] Proposal Generation

[0774] Based on the estimated mental state, the system suggests appropriate actions to the user. For example, if the user is in a relaxed state, it will recommend relaxation products, and if the user is in a concentrated state, it will recommend products that will improve concentration. These suggestions are generated on the server side.

[0775] Input: Inferred state of mind

[0776] Output: Suggested actions or products

[0777] Step 8:

[0778] Notifications and Recommendations

[0779] The results of the recommendations are sent to the user's device in real time as pop-ups or in-app messages, and product recommendations are also made in the virtual store.

[0780] Input: Suggested action or product

[0781] Output: Notification to user device and product recommendations

[0782] Step 9:

[0783] Reviewing feedback

[0784] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. We recommend the relaxation goods section" is displayed.

[0785] Input: Notification to user terminal

[0786] Output: User confirms the proposal

[0787] Step 10:

[0788] Action implementation and additional data collection

[0789] The user performs the suggested action and receives feedback by putting the headgear back on to collect additional EEG data, a process that allows the system to make even more personalized suggestions.

[0790] Input: Additional EEG data after user action

[0791] Output: Collected data on new headgear worn

[0792] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0793] The present invention is a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and suggests appropriate actions, and specific embodiments thereof will be described below.

[0794] Basic configuration

[0795] 1. Headgear

[0796] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[0797] 2. Terminal

[0798] The brainwave data transmitted from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[0799] 3. Server

[0800] A server that receives and processes EEG data sent from the device. This server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[0801] Program processing overview

[0802] Server Processing

[0803] 1. Data Reception

[0804] The server receives the EEG data sent from the device in real time, and the received data is preprocessed to remove noise.

[0805] 2. Data Preprocessing

[0806] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[0807] 3. Frequency analysis

[0808] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0809] 4. Emotional state estimation using emotion engine

[0810] Based on the frequency analysis results, an emotion engine is used to estimate the user's emotional state from the EEG data. This emotion engine performs emotion estimation based on a trained model using a machine learning algorithm.

[0811] 5. Mental State Estimation

[0812] The frequency-analyzed data is used to estimate the user's mental state (relaxed, focused, excited, etc.) along with their emotional state.

[0813] 6. Proposal Generation

[0814] Based on the estimated mental and emotional state, the system generates appropriate action suggestions for the user, such as "You are currently in a relaxed state, so please continue to take deep breaths" or "You are feeling stressed, so please take a short walk."

[0815] 7. Notification

[0816] The results of the suggestions are sent to the user's device in real time via push notifications or in-app messages.

[0817] Terminal handling

[0818] 1. Data Reception

[0819] The device receives real-time brainwave data from the headgear and temporarily stores it in a buffer.

[0820] 2. Data Transmission

[0821] The device sends the data stored in the buffer to the server using a secure communication protocol.

[0822] 3. Receiving and displaying notifications

[0823] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[0824] User operations

[0825] 1. Put on the headgear

[0826] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[0827] 2. Feedback Check

[0828] The user can check the analysis results and suggestions sent from the server on their device. For example, advice such as "You are currently in a relaxed state" or "You are feeling stressed. Relax" will be displayed.

[0829] 3. Action Practice

[0830] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[0831] Specific examples

[0832] Example 1: Mental health care

[0833] To relieve work-related stress, the user wears headgear to collect brain wave data. The server analyzes the data and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[0834] Example 2: Improved concentration

[0835] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[0836] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by analyzing the user's mental state using brain wave data and emotional state and suggesting appropriate actions.

[0837] The processing flow will be explained below.

[0838] Step 1:

[0839] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[0840] Step 2:

[0841] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[0842] Step 3:

[0843] The device preprocesses the EEG data stored in the buffer, specifically by applying low-pass and high-pass filters to remove noise.

[0844] Step 4:

[0845] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[0846] Step 5:

[0847] The server receives the EEG data sent from the device, and then performs pre-processing such as filtering and noise removal on the received data.

[0848] Step 6:

[0849] The server applies a Fourier transform (FFT) to the preprocessed EEG data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0850] Step 7:

[0851] The server evaluates the data from each frequency band and uses this to estimate the user's mental state (relaxed, focused, excited, etc.).

[0852] Step 8:

[0853] The server then uses an emotion engine to estimate the user's emotional state (happiness, anxiety, stress, etc.) from the EEG data. The emotion engine uses machine learning algorithms to make predictions based on a trained model.

[0854] Step 9:

[0855] The server generates appropriate action suggestions for the user based on the estimated mental and emotional state, such as "You are currently in a relaxed state. Please continue to take deep breaths" or "You are feeling stressed. It would be good to take a short walk."

[0856] Step 10:

[0857] The server generates suggestions and sends them to the user's device in real time, either as push notifications or in-app messages.

[0858] Step 11:

[0859] The device receives the suggestions and displays them in a format that is easy for the user to understand. Notifications are sent via push notifications or special screens within the app.

[0860] Step 12:

[0861] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[0862] Step 13:

[0863] The user puts on the headgear again and collects EEG data after performing the action. This allows the server to analyze the effect of the user's action again and provide feedback. This data is again sent to the device, where it is analyzed by the server, and the process from step 1 is repeated.

[0864] In this way, the system repeats processing steps from collecting the user's brainwave data to estimating their emotional state, notifying them of suggested actions, and providing feedback on the results of their actions, thereby continuously supporting the user's mental and emotional state.

[0865] Example 2

[0866] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0867] The present invention aims to provide a system that accurately grasps a user's mental and emotional state based on electroencephalogram (EEG) data and provides appropriate action suggestions in real time based on the results. Existing technologies have problems such as inaccurate analysis of EEG data and action suggestions, and difficulty in real-time notification. In particular, the objective is to solve the problem of low accuracy in noise removal and emotion estimation, making it difficult to provide accurate feedback to the user.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0869] In this invention, the server includes a device for collecting electroencephalogram data, a communication device for receiving the electroencephalogram data transmitted from the device, a calculation device for receiving the electroencephalogram data transmitted from the communication device as input and performing preprocessing, a computation device for frequency analyzing the preprocessed electroencephalogram data, an estimation device for estimating a mental state from the data frequency analyzed by the computation device, a suggestion device for making appropriate action suggestions to individual users based on the mental state estimated by the estimation device, and a notification device for notifying the user's communication device of the suggestion results. This makes it possible to analyze noise-removed electroencephalogram data in real time, accurately estimate the mental state and emotional state, and provide appropriate action suggestions to users in real time.

[0870] "Electroencephalogram data" is information that represents in digital form signals that measure the electrical activity of the brain.

[0871] "Device" refers to equipment including headgear and sensors worn on the user's head to collect brainwave data.

[0872] A "communication device" is a device such as a smartphone or PC that receives the brainwave data sent from the device and sends it to a server.

[0873] A "computing device" is a device that includes a computer or processor for pre-processing electroencephalogram data received from a communication device.

[0874] The "arithmetic unit" is a device for frequency analysis of preprocessed electroencephalogram data, and executes algorithms such as Fourier transform.

[0875] The "estimation device" is a device for estimating the mental state of a user from frequency analysis data obtained by a computing device, and uses a machine learning algorithm.

[0876] The "suggestion device" is a device that suggests appropriate actions to a user based on the mental state estimated by the estimation device.

[0877] The "notification device" is a device that notifies the user's communication device of the proposal results from the proposal device in real time.

[0878] The present invention provides a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and proposes appropriate actions. This system is implemented using the following main hardware and software components.

[0879] 1. Headgear

[0880] The user wears a headgear to collect EEG data. This headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs the signals as digital data, allowing for accurate collection of EEG data.

[0881] 2. Terminal

[0882] The brainwave data sent from the headgear is received by a device such as a smartphone or PC via a connection method such as Bluetooth or USB. The device temporarily stores the received data in a buffer and then transmits it to a server using a secure communication protocol.

[0883] 3. Server

[0884] The server receives the EEG data sent from the device in real time and performs the following processing. First, it applies a low-pass filter and a high-pass filter to remove noise. Next, it uses a Fourier transform (FFT) on the preprocessed EEG data to perform frequency analysis. This allows it to be classified into various frequency bands, such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[0885] Furthermore, an emotion engine is used to estimate the user's emotional state based on the analysis results. The emotion engine uses a machine learning algorithm to estimate emotions based on a model. Then, based on the estimated emotional state and frequency analysis data, the user's mental state (relaxed, focused, excited, etc.) is evaluated.

[0886] Based on these evaluation results, the server generates appropriate action suggestions for the user, such as "Keep taking deep breaths" or "Take a short walk." The generated suggestions are sent to the user's device in real time as push notifications or in-app messages.

[0887] Specific examples

[0888] Example 1: Mental health care

[0889] When a user puts on the headgear to relieve work stress, the server analyzes the situation and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[0890] Example 2: Improved concentration

[0891] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[0892] Prompt Sentence Examples

[0893] Sample prompt 1: "After a user wears the headgear and collects EEG data, explain how that data can be analyzed to support mental health care."

[0894] Sample prompt 2: "Please explain with a concrete example how EEG data can be used to generate appropriate action suggestions to improve a user's focus."

[0895] As described above, the present invention provides a specific embodiment that supports mental health care and improved concentration by analyzing the user's mental state using electroencephalogram data and emotional state and suggesting appropriate actions.

[0896] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0897] Step 1: Receiving data

[0898] The server receives the EEG data transmitted from the device in real time. As input, the EEG data acquired by the device from the headgear is used. The received data is passed to a pre-processing step for noise reduction. Specifically, the data is temporarily stored in the server's receiving buffer.

[0899] Step 2: Preprocessing the data

[0900] The server preprocesses the received EEG data. In this step, noise is removed by applying low-pass and high-pass filters. The EEG data received in step 1 is used as input, and noise-removed EEG data is obtained as output. Specifically, a fast moving average algorithm is used to reduce noise in the signal.

[0901] Step 3: Frequency analysis

[0902] The server performs a Fourier transform (FFT) on the preprocessed EEG data. The input is the data from which noise was removed in step 2, and the output is data classified into frequency bands such as alpha, beta, theta, delta, and gamma waves. Specifically, the FFT algorithm is run to quantify the intensity of each frequency band.

[0903] Step 4: Estimating emotional state

[0904] The server uses an emotion engine to estimate the user's emotional state based on the results of the frequency analysis. The frequency data classified in step 3 is used as input, and the emotional state, such as relaxation, excitement, or stress, is obtained as output. Specifically, the emotion engine, which uses a machine learning algorithm, estimates the emotional state based on the model.

[0905] Step 5: Mental state estimation

[0906] The server estimates the user's mental state based on the emotional state and the frequency analysis results. The emotional state data from step 4 and the frequency data from step 3 are used as input, and the specific mental state (relaxed, focused, excited, etc.) is obtained as output. Specifically, the mental state is evaluated multidimensionally by combining multiple data indicators.

[0907] Step 6: Generate proposals

[0908] The server generates action suggestions for the user based on the estimated mental state. The mental state data obtained in step 5 is used as input, and specific action suggestions (e.g., "Continue to take deep breaths" or "Take a short break") are obtained as output. Specifically, suggestions are automatically generated according to predefined suggestion generation rules.

[0909] Step 7: Notification

[0910] The server notifies the generated action suggestions to the user's device in real time. The suggestion data generated in step 6 is used as input, and a notification message is sent to the user's device as output. Specifically, the server uses push notification and in-app messaging functions to provide instant feedback to the user.

[0911] Step 8: Put on the headgear

[0912] The user puts on the headgear and connects it to the terminal. The input is to confirm that the headgear is attached according to a specific procedure, and the output is to start collecting brainwave data. Specifically, the headgear's sensors measure the electrical activity of the brainwaves in real time and transmit the data to the terminal.

[0913] Step 9: Feedback confirmation

[0914] The user checks the suggestion displayed on the device and decides on the next action. The suggestion message notified in step 7 is used as input, and the action to be taken by the user is determined as output. Specifically, the device screen displays "Please continue to take deep breaths," and the user follows the advice.

[0915] Step 10: Take Action

[0916] The user puts on the headgear again to practice the suggested behavior and get feedback. The behavior determined in step 9 is used as input, and results such as improved performance or mood are obtained as output. Specifically, the user actually takes a deep breath, and EEG data is collected again to confirm the effect.

[0917] (Application example 2)

[0918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0919] Systems already exist that use EEG data to understand a user's mental and emotional state and suggest appropriate actions. However, these systems are primarily limited to personal use, mental health care, and improving concentration, and have not been applied to improving customer service in brick-and-mortar stores. Another issue is that if suggestions are not made in real time, it is difficult to improve customer satisfaction.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0921] In this invention, the server includes headgear for collecting electroencephalogram data, an information terminal device for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the information terminal device as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means, notification means for notifying the user's information terminal device of the suggestion results, and suggestion means for suggesting appropriate services and products in real time based on the mental state and emotional state of the customer using an electroencephalogram data collection device temporarily worn by the customer in a physical store. This enables personalized service offerings based on the customer's emotional and mental state in a physical store.

[0922] "Electroencephalogram data" is information that measures the electrical activity of the brain and expresses it in digital form.

[0923] "Headgear" is a device worn on the head to collect the wearer's brain wave data.

[0924] An "information terminal device" is a device for receiving and processing brain wave data transmitted from the headgear, and includes smartphones, personal computers, etc.

[0925] The "processing means" is a means having a function for receiving and preprocessing electroencephalogram data transmitted from an information terminal device.

[0926] The "analysis means" is a mechanism for performing frequency analysis on preprocessed electroencephalogram data.

[0927] "Frequency analysis" is an analytical method that divides EEG data into multiple frequency bands, and uses techniques such as Fourier transform.

[0928] The "estimation means" is a means for estimating a mental state from data frequency-analyzed by the analysis means.

[0929] The "suggestion means" is a means for suggesting an appropriate action to the user based on the mental state estimated by the estimation means.

[0930] The "notification means" is a mechanism for notifying the user's information terminal device of the result of the proposal made by the proposal means.

[0931] "Brick and mortar store" refers to a physical store where customers visit to purchase products.

[0932] An "electroencephalogram data collection device" is a temporarily wearable device for collecting electroencephalogram data.

[0933] A "generative AI model" is an artificial intelligence model that has been trained to perform a specific task using machine learning algorithms.

[0934] A "prompt sentence" is a sentence of guidance or advice that is generated based on the user's mental and emotional state.

[0935] MODE FOR CARRYING OUT THE INVENTION

[0936] The present invention provides a system that analyzes electroencephalogram data collected from a user wearing a headgear, and aims to improve the customer experience, particularly in physical stores. Specific embodiments are described below.

[0937] System configuration

[0938] The system consists of the following main components:

[0939] 1. Headgear

[0940] This is a device that is temporarily worn by the user (customer) to collect brain wave data. It acquires brain wave data in real time and transmits it to an information terminal device.

[0941] 2. Information terminal device

[0942] Mobile devices such as smartphones and tablets are used to receive EEG data sent from the headgear and send it to a server.

[0943] 3. Server

[0944] The cloud-based system uses a high-performance processor to preprocess and analyze EEG data, specifically removing noise from the data, analyzing frequencies, estimating mental and emotional states, and generating appropriate action suggestions, which are then sent to the information terminal device.

[0945] Program processing overview

[0946] The server is equipped with the following processing means:

[0947] 1. Data Reception

[0948] The server receives the EEG data sent from the information terminal device in real time, and the data is first temporarily stored in storage.

[0949] 2. Noise Reduction and Preprocessing

[0950] Low-pass and high-pass filters are used to remove noise from the received EEG data, resulting in clean data suitable for analysis.

[0951] 3. Frequency analysis

[0952] The preprocessed EEG data is then classified by frequency using a Fourier transform (FFT), and waves that reflect psychological states, such as alpha waves, beta waves, and theta waves, are identified.

[0953] 4. Estimating Mental and Emotional States

[0954] Mental and emotional states are estimated from the analysis results using an estimation method based on a generative AI model using machine learning algorithms.

[0955] 5. Generating action suggestions

[0956] Based on the estimated mental and emotional state, appropriate action suggestions (e.g., information about products and services that will help customers relax) are generated.

[0957] 6. Notification

[0958] The generated suggestions are sent to the information terminal device in real time via push notifications or in-app messages.

[0959] Example

[0960] A specific example is given below.

[0961] Example 1: Proposing a relaxing environment

[0962] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[0963] The server analyzes the data and estimates the state of relaxation. Based on the generative AI model, it generates a prompt in real time, such as "You are in a relaxed state. You can use the relaxation area."

[0964] The user's information terminal device is notified.

[0965] Example 2: Proposing a stress-relieving product

[0966] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[0967] The server analyzes the data and estimates the state of stress. Based on the generative AI model, it generates a prompt in real time, such as "You are in a stressful state. Please try some stress relief products."

[0968] The user's information terminal device is notified.

[0969] Prompt Sentence Examples

[0970] An example of a prompt sentence to input to the generative AI model is as follows:

[0971] plaintext

[0972] A customer puts on smart glasses and enters a store. Analyze their brainwave data to estimate their current emotional and mental state. Based on the estimation results, generate a message suggesting appropriate services and products. For example, if the customer is in a relaxed state, introduce them to a relaxation area, or if they are in a stressed state, suggest stress-relieving products.

[0973] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0974] Step 1:

[0975] The user puts on the headgear and establishes a connection with the information terminal device.

[0976] Input: EEG data from the headgear

[0977] Output: Raw EEG data sent to an information terminal device

[0978] Specific operation: Electrodes in the headgear detect brain waves and transmit the data to an information terminal device via Bluetooth or Wi-Fi.

[0979] Step 2:

[0980] The terminal receives the brainwave data transmitted from the headgear and temporarily stores it in a buffer.

[0981] Input: Raw EEG data transmitted from the headgear

[0982] Output: Buffered EEG data for sending to the server

[0983] Specific operation: The information terminal device receives data packets from the headgear and buffers a certain amount of brain wave data.

[0984] Step 3:

[0985] The device sends the buffered EEG data to a server, where the data is transferred using a secure communication protocol.

[0986] Input: EEG data stored in a buffer

[0987] Output: EEG data sent to the server

[0988] Specific operation: The information terminal device uploads the buffered data to the server using the HTTPS or SSH protocol.

[0989] Step 4:

[0990] The server receives the brainwave data in real time and temporarily stores it in storage.

[0991] Input: EEG data sent from the device

[0992] Output: Saved EEG data

[0993] Specific operation: The server uses the data receiving module to continuously receive streams of data from the terminal and store them in storage.

[0994] Step 5:

[0995] The server denoises and preprocesses the EEG data, using low-pass and high-pass filters to remove noise and generate clean data.

[0996] Input: Stored EEG data

[0997] Output: Noise-removed EEG data

[0998] What it does: Preprocessing algorithms remove high- and low-frequency noise to prepare data suitable for analysis.

[0999] Step 6:

[1000] The server performs frequency analysis on the preprocessed EEG data, classifying it into frequency bands using a Fourier transform (FFT).

[1001] Input: EEG data after noise removal

[1002] Output: EEG data categorized by frequency

[1003] Specific operation: Runs the FFT algorithm to decompose brainwave data into alpha waves, beta waves, theta waves, etc.

[1004] Step 7:

[1005] The server uses the frequency analysis results to estimate the mental and emotional state, and utilizes a generative AI model.

[1006] Input: Frequency analysis results

[1007] Output: Inferred mental and emotional state

[1008] Specific operation: The analysis results are input into a generative AI model, which estimates mental state (relaxed, stressed, etc.) and emotional state (happiness, anxiety, etc.) based on a trained algorithm.

[1009] Step 8:

[1010] Based on the estimation results, the server generates appropriate action suggestions and creates prompt sentences.

[1011] Input: Inferred mental and emotional states

[1012] Output: Action suggestion prompt

[1013] Specific operation: A rule-based engine is activated to provide action suggestions based on the inference results, generating prompt statements such as "You are in a relaxed state. You can use the relaxation area."

[1014] Step 9:

[1015] The server notifies the device of the proposed results in the form of a push notification or an in-app message.

[1016] Input: Prompt for suggested action

[1017] Output: Proposal message displayed on the terminal

[1018] Specific operation: The server uses the notification system to send the generated prompt text to the terminal and notify the user.

[1019] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1020] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1021] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1022] [Third embodiment]

[1023] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1024] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1026] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1027] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1028] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1030] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1031] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1033] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1034] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1035] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[1036] Basic configuration

[1037] 1. Headgear

[1038] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1039] 2. Terminal

[1040] The user's smartphone, PC, or other device receives the brainwave data sent from the headgear. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[1041] 3. Server

[1042] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1043] Program processing overview

[1044] Server Processing

[1045] 1. Data Reception

[1046] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1047] 2. Data Preprocessing

[1048] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[1049] 3. Frequency analysis

[1050] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1051] 4. Mental State Estimation

[1052] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1053] 5. Proposal Generation

[1054] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[1055] 6. Notification

[1056] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[1057] Terminal handling

[1058] 1. Data Reception

[1059] The device receives real-time EEG data from the headgear, which is temporarily stored in a buffer within the device.

[1060] 2. Data Transmission

[1061] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1062] 3. Receiving and displaying notifications

[1063] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[1064] User operations

[1065] 1. Put on the headgear

[1066] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1067] 2. Feedback Check

[1068] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1069] 3. Action Practice

[1070] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1071] Specific examples

[1072] Example 1: Mental health care

[1073] To relieve work-related stress, the user wears headgear to collect brainwave data. The server analyzes the data and determines that "alpha waves are high, indicating a state of relaxation." The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[1074] Example 2: Improved concentration

[1075] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that their beta waves are high and they are in a state of concentration. The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[1076] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by using electroencephalogram data to analyze the user's mental state and suggest appropriate actions.

[1077] The processing flow will be explained below.

[1078] Step 1:

[1079] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[1080] Step 2:

[1081] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[1082] Step 3:

[1083] The device removes noise from the EEG data stored in the buffer by applying low-pass and high-pass filters.

[1084] Step 4:

[1085] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[1086] Step 5:

[1087] The server receives the EEG data sent from the device, and the received data undergoes further preprocessing such as noise removal and filtering.

[1088] Step 6:

[1089] The server applies a Fourier transform (FFT) to the preprocessed data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1090] Step 7:

[1091] The server evaluates the intensity of each frequency band and uses this to estimate the user's mental state. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration.

[1092] Step 8:

[1093] The server generates appropriate action suggestions for the user based on the estimated mental state, for example, "You are currently in a relaxed state. Please continue to take deep breaths."

[1094] Step 9:

[1095] The server notifies the user of the generated suggestions in real time, supporting the user's actions.

[1096] Step 10:

[1097] The device receives the suggestions and displays them in an easy-to-understand format to the user. Notifications are sent via push notifications or special screens within the app.

[1098] Step 11:

[1099] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[1100] Step 12:

[1101] The user puts on the headgear again, and EEG data is collected after the behavior is performed. This data is again sent to the terminal, and the process repeats from step 1.

[1102] In this way, the system repeats the processing steps of collecting the user's brainwave data, analyzing it, notifying them of suggestions, and then providing feedback, continuously supporting the user's mental state.

[1103] Example 1

[1104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1105] Stress and decreased concentration have become problems in modern society. Conventional methods have made it difficult to quickly provide effective solutions to these problems. Therefore, the challenge is to provide a system that can analyze EEG data to grasp the user's mental state in real time and suggest appropriate actions.

[1106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1107] In this invention, the server includes a device for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the device, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, and notification means for notifying the user's terminal of the suggestion result, thereby making it possible to grasp the user's mental state in real time and suggest appropriate actions.

[1108] "EEG data" refers to digital signals obtained by detecting electrical activity in the brain.

[1109] The "device" refers to a device worn on the head to collect EEG data, and is equipped with multiple electrodes.

[1110] "Terminal" refers to an electronic device that receives the brainwave data transmitted from the device, including smartphones and PCs.

[1111] The "processing means" is a means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing such as noise removal.

[1112] The "analysis means" refers to a means for frequency-analyzing the preprocessed electroencephalogram data, and uses an analysis method such as Fourier transform.

[1113] The "estimation means" is a means for estimating the state of mind from the data frequency-analyzed by the analysis means.

[1114] The "suggestion means" is a means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means.

[1115] The "notification means" is a means for notifying the user terminal of the result of the proposal made by the proposal means.

[1116] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[1117] Basic configuration

[1118] 1. Device for collecting EEG data

[1119] The user wears a device on their head to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1120] 2. Terminal

[1121] The EEG data sent from the device is received by the user's smartphone, PC, or other device, which communicates with the device using a connection method such as Bluetooth or USB.

[1122] 3. Server

[1123] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1124] Program processing overview

[1125] Server Processing

[1126] 1. Data Reception

[1127] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1128] 2. Data Preprocessing

[1129] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it in a form suitable for analysis.

[1130] 3. Frequency analysis

[1131] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT), which categorizes it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1132] 4. Mental State Estimation

[1133] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1134] 5. Proposal Generation

[1135] The server then suggests appropriate actions to the user based on the estimated mental state. These suggestions are based on past data and general health information. For example, a generative AI model could be used to generate specific suggestions such as "Try taking deep breaths."

[1136] 6. Notification

[1137] The server notifies the user of the results in real time, either as a pop-up or in-app message.

[1138] Terminal handling

[1139] 1. Data Reception

[1140] The terminal receives EEG data from the device in real time, and the received data is temporarily stored in a buffer within the terminal.

[1141] 2. Data Transmission

[1142] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1143] 3. Receiving and displaying notifications

[1144] The device receives the proposed results sent from the server and displays them in a user-friendly format via push notifications or a special screen within the app.

[1145] User operations

[1146] 1. Wearing the device

[1147] The user first puts on the device and connects it to a smartphone or PC to begin collecting brainwave data.

[1148] 2. Feedback Check

[1149] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1150] 3. Action Practice

[1151] The user can then put the device back on to perform the suggested action and receive feedback.

[1152] Specific examples

[1153] Example 1: Mental health care

[1154] To relieve work-related stress, the user wears the device and brainwave data is collected. The server analyzes the data and determines that the user's alpha waves are high, indicating a state of relaxation. The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[1155] Example 2: Improved concentration

[1156] Students wear a device to measure their concentration levels during breaks from studying. The server analyzes the data and determines that "beta waves are high and you are in a state of concentration." The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[1157] Example input to a generative AI model

[1158] "Please suggest specific actions to relieve stress. Based on the user's brain wave data, we have determined that their alpha waves are currently high."

[1159] In this way, an embodiment of the invention provides a system that supports mental health care and improved concentration by analyzing the user's mental state in real time using electroencephalogram data and suggesting appropriate actions.

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] EEG data collection

[1163] The user wears a device on their head to collect brainwave data. The device uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The input is the user's brain's electrical activity, and the output is digitized brainwave data. Specifically, the electrodes capture the brainwaves, and the built-in A / D converter converts the analog signal into digital data.

[1164] Step 2:

[1165] Transmission of brainwave data to a terminal

[1166] The device transmits the collected EEG data to the user's smartphone or PC via Bluetooth or USB connection. The input is digitized EEG data, and the output is EEG data transmitted to the terminal. Specifically, the communication module inside the device encrypts the data and transmits it to the terminal wirelessly or via a wired connection.

[1167] Step 3:

[1168] Data buffering on the device

[1169] The device temporarily stores the received EEG data in a buffer. The input is the EEG data sent from the device, and the output is the data stored in the buffer. Specifically, a dedicated app on the device runs in the background, receiving and saving data in real time.

[1170] Step 4:

[1171] Sending data to the server

[1172] The device sends the EEG data stored in the buffer to the server using a secure communication protocol. The input is the EEG data stored in the buffer, and the output is the data sent to the server. Specifically, a dedicated app on the device compiles the data and sends it to the cloud server using a protocol such as HTTPS.

[1173] Step 5:

[1174] Data reception

[1175] The server receives the EEG data sent from the device in real time. The input is the EEG data sent from the device, and the output is the data received within the server. Specifically, the server's receiving module decodes the data and stores it in storage.

[1176] Step 6:

[1177] Data Preprocessing

[1178] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it for analysis. The input is the received raw data, and the output is the noise-removed data. Specifically, a pre-processing algorithm in the server filters the data and removes unnecessary frequency components.

[1179] Step 7:

[1180] Frequency Analysis

[1181] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT). The input is noise-removed data, and the output is data classified into each frequency band. Specifically, the FFT algorithm is applied to classify the data into alpha waves, beta waves, theta waves, delta waves, gamma waves, etc.

[1182] Step 8:

[1183] Mental state estimation

[1184] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns. The input is data classified into each frequency band, and the output is an estimated mental state. Specifically, it uses a machine learning model to identify states such as relaxation, concentration, and drowsiness.

[1185] Step 9:

[1186] Proposal Generation

[1187] The server then suggests appropriate actions to the user based on the estimated mental state. The input is the estimated mental state, and the output is the suggested action. Specific operations include using a generative AI model to generate specific suggestions such as "Try taking deep breaths."

[1188] Step 10:

[1189] Proposal Notification

[1190] The server notifies the user's device of the proposed action in real time. The input is the proposed action, and the output is a notification displayed on the user's device. Specifically, the server's notification module converts the proposed action into an appropriate format and sends it to the device.

[1191] Step 11:

[1192] Check and implement user feedback

[1193] The user checks the suggested results displayed on the device and performs specific actions. The input is the notification received from the server, and the output is the change in the user's mental state as a result of the user performing the action. Specific actions include the user taking a deep breath or taking a break.

[1194] (Application example 1)

[1195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1196] In modern society, it is extremely important to accurately grasp a user's psychological state in real time and provide products and services that correspond to that state. However, existing virtual store systems have difficulty accurately analyzing a user's psychological state and recommending products based on that state. In particular, they lack the ability to propose appropriate products and services that respond to subtle changes in the user's psychological state, such as whether the user is relaxed or focused. This reduces the quality of the user experience and hinders improvement in satisfaction. The present invention aims to solve these problems and provide detailed product recommendations in real time based on the user's psychological state.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1198] In this invention, the server includes headgear for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, notification means for notifying the user's terminal of the suggestion result, and recommendation means for recommending products in a virtual store based on the mental state estimated in real time. This enables optimal product recommendations based on the user's psychological state, increasing user satisfaction and providing a personalized experience tailored to each user.

[1199] "Electroencephalogram data" is digital data that indicates the electrical activity of the brain and is collected by headgear worn on the head.

[1200] "Headgear" is a device that is worn on the user's head and collects brain wave data using multiple electrodes.

[1201] A "terminal" is a device such as a smartphone or PC that receives brainwave data sent from the headgear and sends it to a server.

[1202] The "processing means" refers to a device that receives EEG data transmitted from a terminal as input and has the function of preprocessing the data using noise removal, low-pass filters, high-pass filters, etc.

[1203] The "analysis means" is a mechanism used to perform frequency analysis on preprocessed EEG data, specifically, a mechanism that has the function of classifying the data into frequency bands using a Fourier transform (FFT).

[1204] The "estimation means" has a function of estimating the user's mental state from the data frequency-analyzed by the analysis means.

[1205] The "suggestion means" has a function of suggesting appropriate actions to each user based on the mental state estimated by the estimation means.

[1206] The "notification means" has a function of notifying the user terminal of the result of the proposal made by the proposal means.

[1207] The "recommendation means" has the function of recommending products within a virtual store based on the results of real-time estimation of the user's mental state.

[1208] This invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. This system is mainly composed of headgear, a terminal, and a server.

[1209] Basic configuration

[1210] 1. Headgear

[1211] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1212] 2. Terminal

[1213] The brainwave data sent from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB. The device also has the role of sending the received data to a server.

[1214] 3. Server

[1215] The server receives the EEG data sent from the device and performs various processing. It is installed on the cloud and uses a high-performance processor to preprocess and analyze the data. The server-side processing mainly consists of the following steps.

[1216] Server Processing

[1217] 1. Data Reception

[1218] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1219] 2. Data Preprocessing

[1220] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it for analysis. Signal processing libraries such as SciPy are used here.

[1221] 3. Frequency analysis

[1222] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1223] 4. Mental State Estimation

[1224] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1225] 5. Proposal Generation

[1226] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[1227] 6. Notification

[1228] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[1229] User operations

[1230] 1. Put on the headgear

[1231] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1232] 2. Feedback Check

[1233] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1234] 3. Action Practice

[1235] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1236] Specific examples

[1237] As a specific example, by introducing this system into a virtual store, if a user is relaxed, it can recommend products that have a relaxing effect, and if a user is concentrating, it can suggest products that will further enhance concentration. This allows users to select products that best suit their own psychological state, providing a more satisfying shopping experience.

[1238] Prompt Sentence Examples

[1239] While a user is moving around in a virtual store, analyze their brainwave data in real time. If it is determined that the user is in a relaxed state, recommend products that have a relaxing effect to the user.

[1240] In this way, the present invention can realize detailed product recommendations based on the user's psychological state, improving the quality of the user's experience in a virtual store.

[1241] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1242] Step 1:

[1243] Headgear installation and data collection

[1244] The user wears a headgear to collect EEG data. The headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The output data from the headgear is then sent to a terminal.

[1245] Input: User's electrical brain activity

[1246] Output: Digital data sent from the headgear to the device

[1247] Step 2:

[1248] Receiving data

[1249] The device receives the EEG data transmitted from the headgear in real time, and the received data is temporarily stored in a buffer within the device.

[1250] Input: Digital data transmitted from the headgear

[1251] Output: EEG data stored in a temporary buffer on the device

[1252] Step 3:

[1253] Sending data

[1254] The device transmits the buffered EEG data to a server at regular intervals using a secure communication protocol (e.g., HTTPS).

[1255] Input: EEG data stored in a temporary buffer on the device

[1256] Output: EEG data sent to the server

[1257] Step 4:

[1258] Data reception and noise reduction

[1259] The server receives the EEG data sent from the device in real time. Because the received data may contain noise, it is first subjected to noise removal. Specifically, low-pass and high-pass filters are used to remove unnecessary frequency components.

[1260] Input: EEG data sent from the device

[1261] Output: Denoised data

[1262] Step 5:

[1263] Frequency Analysis

[1264] The pre-processed (noise-removed) EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies the data into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1265] Input: Denoised data

[1266] Output: Data classified into each frequency band

[1267] Step 6:

[1268] Mental state estimation

[1269] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's state of mind based on specific patterns. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration. AI models may be used for this estimation.

[1270] Input: Data classified into each frequency band

[1271] Output: Estimated state of mind

[1272] Step 7:

[1273] Proposal Generation

[1274] Based on the estimated mental state, the system suggests appropriate actions to the user. For example, if the user is in a relaxed state, it will recommend relaxation products, and if the user is in a concentrated state, it will recommend products that will improve concentration. These suggestions are generated on the server side.

[1275] Input: Inferred state of mind

[1276] Output: Suggested actions or products

[1277] Step 8:

[1278] Notifications and Recommendations

[1279] The results of the recommendations are sent to the user's device in real time as pop-ups or in-app messages, and product recommendations are also made in the virtual store.

[1280] Input: Suggested action or product

[1281] Output: Notification to user device and product recommendations

[1282] Step 9:

[1283] Reviewing feedback

[1284] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. We recommend the relaxation goods section" is displayed.

[1285] Input: Notification to user terminal

[1286] Output: User confirms the proposal

[1287] Step 10:

[1288] Action implementation and additional data collection

[1289] The user performs the suggested action and receives feedback by putting the headgear back on to collect additional EEG data, a process that allows the system to make even more personalized suggestions.

[1290] Input: Additional EEG data after user action

[1291] Output: Collected data on new headgear worn

[1292] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1293] The present invention is a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and suggests appropriate actions, and specific embodiments thereof will be described below.

[1294] Basic configuration

[1295] 1. Headgear

[1296] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1297] 2. Terminal

[1298] The brainwave data transmitted from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[1299] 3. Server

[1300] A server that receives and processes EEG data sent from the device. This server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1301] Program processing overview

[1302] Server Processing

[1303] 1. Data Reception

[1304] The server receives the EEG data sent from the device in real time, and the received data is preprocessed to remove noise.

[1305] 2. Data Preprocessing

[1306] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[1307] 3. Frequency analysis

[1308] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1309] 4. Emotional state estimation using emotion engine

[1310] Based on the frequency analysis results, an emotion engine is used to estimate the user's emotional state from the EEG data. This emotion engine performs emotion estimation based on a trained model using a machine learning algorithm.

[1311] 5. Mental State Estimation

[1312] The frequency-analyzed data is used to estimate the user's mental state (relaxed, focused, excited, etc.) along with their emotional state.

[1313] 6. Proposal Generation

[1314] Based on the estimated mental and emotional state, the system generates appropriate action suggestions for the user, such as "You are currently in a relaxed state, so please continue to take deep breaths" or "You are feeling stressed, so please take a short walk."

[1315] 7. Notification

[1316] The results of the suggestions are sent to the user's device in real time via push notifications or in-app messages.

[1317] Terminal handling

[1318] 1. Data Reception

[1319] The device receives real-time brainwave data from the headgear and temporarily stores it in a buffer.

[1320] 2. Data Transmission

[1321] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1322] 3. Receiving and displaying notifications

[1323] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[1324] User operations

[1325] 1. Put on the headgear

[1326] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1327] 2. Feedback Check

[1328] The user can check the analysis results and suggestions sent from the server on their device. For example, advice such as "You are currently in a relaxed state" or "You are feeling stressed. Relax" will be displayed.

[1329] 3. Action Practice

[1330] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1331] Specific examples

[1332] Example 1: Mental health care

[1333] To relieve work-related stress, the user wears headgear to collect brain wave data. The server analyzes the data and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[1334] Example 2: Improved concentration

[1335] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[1336] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by analyzing the user's mental state using brain wave data and emotional state and suggesting appropriate actions.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[1340] Step 2:

[1341] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[1342] Step 3:

[1343] The device preprocesses the EEG data stored in the buffer, specifically by applying low-pass and high-pass filters to remove noise.

[1344] Step 4:

[1345] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[1346] Step 5:

[1347] The server receives the EEG data sent from the device, and then performs pre-processing such as filtering and noise removal on the received data.

[1348] Step 6:

[1349] The server applies a Fourier transform (FFT) to the preprocessed EEG data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1350] Step 7:

[1351] The server evaluates the data from each frequency band and uses this to estimate the user's mental state (relaxed, focused, excited, etc.).

[1352] Step 8:

[1353] The server then uses an emotion engine to estimate the user's emotional state (happiness, anxiety, stress, etc.) from the EEG data. The emotion engine uses machine learning algorithms to make predictions based on a trained model.

[1354] Step 9:

[1355] The server generates appropriate action suggestions for the user based on the estimated mental and emotional state, such as "You are currently in a relaxed state. Please continue to take deep breaths" or "You are feeling stressed. It would be good to take a short walk."

[1356] Step 10:

[1357] The server generates suggestions and sends them to the user's device in real time, either as push notifications or in-app messages.

[1358] Step 11:

[1359] The device receives the suggestions and displays them in a format that is easy for the user to understand. Notifications are sent via push notifications or special screens within the app.

[1360] Step 12:

[1361] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[1362] Step 13:

[1363] The user puts on the headgear again and collects EEG data after performing the action. This allows the server to analyze the effect of the user's action again and provide feedback. This data is again sent to the device, where it is analyzed by the server, and the process from step 1 is repeated.

[1364] In this way, the system repeats processing steps from collecting the user's brainwave data to estimating their emotional state, notifying them of suggested actions, and providing feedback on the results of their actions, thereby continuously supporting the user's mental and emotional state.

[1365] Example 2

[1366] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] The present invention aims to provide a system that accurately grasps a user's mental and emotional state based on electroencephalogram (EEG) data and provides appropriate action suggestions in real time based on the results. Existing technologies have problems such as inaccurate analysis of EEG data and action suggestions, and difficulty in real-time notification. In particular, the objective is to solve the problem of low accuracy in noise removal and emotion estimation, making it difficult to provide accurate feedback to the user.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1369] In this invention, the server includes a device for collecting electroencephalogram data, a communication device for receiving the electroencephalogram data transmitted from the device, a calculation device for receiving the electroencephalogram data transmitted from the communication device as input and performing preprocessing, a computation device for frequency analyzing the preprocessed electroencephalogram data, an estimation device for estimating a mental state from the data frequency analyzed by the computation device, a suggestion device for making appropriate action suggestions to individual users based on the mental state estimated by the estimation device, and a notification device for notifying the user's communication device of the suggestion results. This makes it possible to analyze noise-removed electroencephalogram data in real time, accurately estimate the mental state and emotional state, and provide appropriate action suggestions to users in real time.

[1370] "Electroencephalogram data" is information that represents in digital form signals that measure the electrical activity of the brain.

[1371] "Device" refers to equipment including headgear and sensors worn on the user's head to collect brainwave data.

[1372] A "communication device" is a device such as a smartphone or PC that receives the brainwave data sent from the device and sends it to a server.

[1373] A "computing device" is a device that includes a computer or processor for pre-processing electroencephalogram data received from a communication device.

[1374] The "arithmetic unit" is a device for frequency analysis of preprocessed electroencephalogram data, and executes algorithms such as Fourier transform.

[1375] The "estimation device" is a device for estimating the mental state of a user from frequency analysis data obtained by a computing device, and uses a machine learning algorithm.

[1376] The "suggestion device" is a device that suggests appropriate actions to a user based on the mental state estimated by the estimation device.

[1377] The "notification device" is a device that notifies the user's communication device of the proposal results from the proposal device in real time.

[1378] The present invention provides a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and proposes appropriate actions. This system is implemented using the following main hardware and software components.

[1379] 1. Headgear

[1380] The user wears a headgear to collect EEG data. This headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs the signals as digital data, allowing for accurate collection of EEG data.

[1381] 2. Terminal

[1382] The brainwave data sent from the headgear is received by a device such as a smartphone or PC via a connection method such as Bluetooth or USB. The device temporarily stores the received data in a buffer and then transmits it to a server using a secure communication protocol.

[1383] 3. Server

[1384] The server receives the EEG data sent from the device in real time and performs the following processing. First, it applies a low-pass filter and a high-pass filter to remove noise. Next, it uses a Fourier transform (FFT) on the preprocessed EEG data to perform frequency analysis. This allows it to be classified into various frequency bands, such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1385] Furthermore, an emotion engine is used to estimate the user's emotional state based on the analysis results. The emotion engine uses a machine learning algorithm to estimate emotions based on a model. Then, based on the estimated emotional state and frequency analysis data, the user's mental state (relaxed, focused, excited, etc.) is evaluated.

[1386] Based on these evaluation results, the server generates appropriate action suggestions for the user, such as "Keep taking deep breaths" or "Take a short walk." The generated suggestions are sent to the user's device in real time as push notifications or in-app messages.

[1387] Specific examples

[1388] Example 1: Mental health care

[1389] When a user puts on the headgear to relieve work stress, the server analyzes the situation and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[1390] Example 2: Improved concentration

[1391] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[1392] Prompt Sentence Examples

[1393] Sample prompt 1: "After a user wears the headgear and collects EEG data, explain how that data can be analyzed to support mental health care."

[1394] Sample prompt 2: "Please explain with a concrete example how EEG data can be used to generate appropriate action suggestions to improve a user's focus."

[1395] As described above, the present invention provides a specific embodiment that supports mental health care and improved concentration by analyzing the user's mental state using electroencephalogram data and emotional state and suggesting appropriate actions.

[1396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1397] Step 1: Receiving data

[1398] The server receives the EEG data transmitted from the device in real time. As input, the EEG data acquired by the device from the headgear is used. The received data is passed to a pre-processing step for noise reduction. Specifically, the data is temporarily stored in the server's receiving buffer.

[1399] Step 2: Preprocessing the data

[1400] The server preprocesses the received EEG data. In this step, noise is removed by applying low-pass and high-pass filters. The EEG data received in step 1 is used as input, and noise-removed EEG data is obtained as output. Specifically, a fast moving average algorithm is used to reduce noise in the signal.

[1401] Step 3: Frequency analysis

[1402] The server performs a Fourier transform (FFT) on the preprocessed EEG data. The input is the data from which noise was removed in step 2, and the output is data classified into frequency bands such as alpha, beta, theta, delta, and gamma waves. Specifically, the FFT algorithm is run to quantify the intensity of each frequency band.

[1403] Step 4: Estimating emotional state

[1404] The server uses an emotion engine to estimate the user's emotional state based on the results of the frequency analysis. The frequency data classified in step 3 is used as input, and the emotional state, such as relaxation, excitement, or stress, is obtained as output. Specifically, the emotion engine, which uses a machine learning algorithm, estimates the emotional state based on the model.

[1405] Step 5: Mental state estimation

[1406] The server estimates the user's mental state based on the emotional state and the frequency analysis results. The emotional state data from step 4 and the frequency data from step 3 are used as input, and the specific mental state (relaxed, focused, excited, etc.) is obtained as output. Specifically, the mental state is evaluated multidimensionally by combining multiple data indicators.

[1407] Step 6: Generate proposals

[1408] The server generates action suggestions for the user based on the estimated mental state. The mental state data obtained in step 5 is used as input, and specific action suggestions (e.g., "Continue to take deep breaths" or "Take a short break") are obtained as output. Specifically, suggestions are automatically generated according to predefined suggestion generation rules.

[1409] Step 7: Notification

[1410] The server notifies the generated action suggestions to the user's device in real time. The suggestion data generated in step 6 is used as input, and a notification message is sent to the user's device as output. Specifically, the server uses push notification and in-app messaging functions to provide instant feedback to the user.

[1411] Step 8: Put on the headgear

[1412] The user puts on the headgear and connects it to the terminal. The input is to confirm that the headgear is attached according to a specific procedure, and the output is to start collecting brainwave data. Specifically, the headgear's sensors measure the electrical activity of the brainwaves in real time and transmit the data to the terminal.

[1413] Step 9: Feedback confirmation

[1414] The user checks the suggestion displayed on the device and decides on the next action. The suggestion message notified in step 7 is used as input, and the action to be taken by the user is determined as output. Specifically, the device screen displays "Please continue to take deep breaths," and the user follows the advice.

[1415] Step 10: Take Action

[1416] The user puts on the headgear again to practice the suggested behavior and get feedback. The behavior determined in step 9 is used as input, and results such as improved performance or mood are obtained as output. Specifically, the user actually takes a deep breath, and EEG data is collected again to confirm the effect.

[1417] (Application example 2)

[1418] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1419] Systems already exist that use EEG data to understand a user's mental and emotional state and suggest appropriate actions. However, these systems are primarily limited to personal use, mental health care, and improving concentration, and have not been applied to improving customer service in brick-and-mortar stores. Another issue is that if suggestions are not made in real time, it is difficult to improve customer satisfaction.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1421] In this invention, the server includes headgear for collecting electroencephalogram data, an information terminal device for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the information terminal device as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means, notification means for notifying the user's information terminal device of the suggestion results, and suggestion means for suggesting appropriate services and products in real time based on the mental state and emotional state of the customer using an electroencephalogram data collection device temporarily worn by the customer in a physical store. This enables personalized service offerings based on the customer's emotional and mental state in a physical store.

[1422] "Electroencephalogram data" is information that measures the electrical activity of the brain and expresses it in digital form.

[1423] "Headgear" is a device worn on the head to collect the wearer's brain wave data.

[1424] An "information terminal device" is a device for receiving and processing brain wave data transmitted from the headgear, and includes smartphones, personal computers, etc.

[1425] The "processing means" is a means having a function for receiving and preprocessing electroencephalogram data transmitted from an information terminal device.

[1426] The "analysis means" is a mechanism for performing frequency analysis on preprocessed electroencephalogram data.

[1427] "Frequency analysis" is an analytical method that divides EEG data into multiple frequency bands, and uses techniques such as Fourier transform.

[1428] The "estimation means" is a means for estimating a mental state from data frequency-analyzed by the analysis means.

[1429] The "suggestion means" is a means for suggesting an appropriate action to the user based on the mental state estimated by the estimation means.

[1430] The "notification means" is a mechanism for notifying the user's information terminal device of the result of the proposal made by the proposal means.

[1431] "Brick and mortar store" refers to a physical store where customers visit to purchase products.

[1432] An "electroencephalogram data collection device" is a temporarily wearable device for collecting electroencephalogram data.

[1433] A "generative AI model" is an artificial intelligence model that has been trained to perform a specific task using machine learning algorithms.

[1434] A "prompt sentence" is a sentence of guidance or advice that is generated based on the user's mental and emotional state.

[1435] MODE FOR CARRYING OUT THE INVENTION

[1436] The present invention provides a system that analyzes electroencephalogram data collected from a user wearing a headgear, and aims to improve the customer experience, particularly in physical stores. Specific embodiments are described below.

[1437] System configuration

[1438] The system consists of the following main components:

[1439] 1. Headgear

[1440] This is a device that is temporarily worn by the user (customer) to collect brain wave data. It acquires brain wave data in real time and transmits it to an information terminal device.

[1441] 2. Information terminal device

[1442] Mobile devices such as smartphones and tablets are used to receive EEG data sent from the headgear and send it to a server.

[1443] 3. Server

[1444] The cloud-based system uses a high-performance processor to preprocess and analyze EEG data, specifically removing noise from the data, analyzing frequencies, estimating mental and emotional states, and generating appropriate action suggestions, which are then sent to the information terminal device.

[1445] Program processing overview

[1446] The server is equipped with the following processing means:

[1447] 1. Data Reception

[1448] The server receives the EEG data sent from the information terminal device in real time, and the data is first temporarily stored in storage.

[1449] 2. Noise Reduction and Preprocessing

[1450] Low-pass and high-pass filters are used to remove noise from the received EEG data, resulting in clean data suitable for analysis.

[1451] 3. Frequency analysis

[1452] The preprocessed EEG data is then classified by frequency using a Fourier transform (FFT), and waves that reflect psychological states, such as alpha waves, beta waves, and theta waves, are identified.

[1453] 4. Estimating Mental and Emotional States

[1454] Mental and emotional states are estimated from the analysis results using an estimation method based on a generative AI model using machine learning algorithms.

[1455] 5. Generating action suggestions

[1456] Based on the estimated mental and emotional state, appropriate action suggestions (e.g., information about products and services that will help customers relax) are generated.

[1457] 6. Notification

[1458] The generated suggestions are sent to the information terminal device in real time via push notifications or in-app messages.

[1459] Example

[1460] A specific example is given below.

[1461] Example 1: Proposing a relaxing environment

[1462] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[1463] The server analyzes the data and estimates the state of relaxation. Based on the generative AI model, it generates a prompt in real time, such as "You are in a relaxed state. You can use the relaxation area."

[1464] The user's information terminal device is notified.

[1465] Example 2: Proposing a stress-relieving product

[1466] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[1467] The server analyzes the data and estimates the state of stress. Based on the generative AI model, it generates a prompt in real time, such as "You are in a stressful state. Please try some stress relief products."

[1468] The user's information terminal device is notified.

[1469] Prompt Sentence Examples

[1470] An example of a prompt sentence to input to the generative AI model is as follows:

[1471] plaintext

[1472] A customer puts on smart glasses and enters a store. Analyze their brainwave data to estimate their current emotional and mental state. Based on the estimation results, generate a message suggesting appropriate services and products. For example, if the customer is in a relaxed state, introduce them to a relaxation area, or if they are in a stressed state, suggest stress-relieving products.

[1473] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1474] Step 1:

[1475] The user puts on the headgear and establishes a connection with the information terminal device.

[1476] Input: EEG data from the headgear

[1477] Output: Raw EEG data sent to an information terminal device

[1478] Specific operation: Electrodes in the headgear detect brain waves and transmit the data to an information terminal device via Bluetooth or Wi-Fi.

[1479] Step 2:

[1480] The terminal receives the brainwave data transmitted from the headgear and temporarily stores it in a buffer.

[1481] Input: Raw EEG data transmitted from the headgear

[1482] Output: Buffered EEG data for sending to the server

[1483] Specific operation: The information terminal device receives data packets from the headgear and buffers a certain amount of brain wave data.

[1484] Step 3:

[1485] The device sends the buffered EEG data to a server, where the data is transferred using a secure communication protocol.

[1486] Input: EEG data stored in a buffer

[1487] Output: EEG data sent to the server

[1488] Specific operation: The information terminal device uploads the buffered data to the server using the HTTPS or SSH protocol.

[1489] Step 4:

[1490] The server receives the brainwave data in real time and temporarily stores it in storage.

[1491] Input: EEG data sent from the device

[1492] Output: Saved EEG data

[1493] Specific operation: The server uses the data receiving module to continuously receive streams of data from the terminal and store them in storage.

[1494] Step 5:

[1495] The server denoises and preprocesses the EEG data, using low-pass and high-pass filters to remove noise and generate clean data.

[1496] Input: Stored EEG data

[1497] Output: Noise-removed EEG data

[1498] What it does: Preprocessing algorithms remove high- and low-frequency noise to prepare data suitable for analysis.

[1499] Step 6:

[1500] The server performs frequency analysis on the preprocessed EEG data, classifying it into frequency bands using a Fourier transform (FFT).

[1501] Input: EEG data after noise removal

[1502] Output: EEG data categorized by frequency

[1503] Specific operation: Runs the FFT algorithm to decompose brainwave data into alpha waves, beta waves, theta waves, etc.

[1504] Step 7:

[1505] The server uses the frequency analysis results to estimate the mental and emotional state, and utilizes a generative AI model.

[1506] Input: Frequency analysis results

[1507] Output: Inferred mental and emotional state

[1508] Specific operation: The analysis results are input into a generative AI model, which estimates mental state (relaxed, stressed, etc.) and emotional state (happiness, anxiety, etc.) based on a trained algorithm.

[1509] Step 8:

[1510] Based on the estimation results, the server generates appropriate action suggestions and creates prompt sentences.

[1511] Input: Inferred mental and emotional states

[1512] Output: Action suggestion prompt

[1513] Specific operation: A rule-based engine is activated to provide action suggestions based on the inference results, generating prompt statements such as "You are in a relaxed state. You can use the relaxation area."

[1514] Step 9:

[1515] The server notifies the device of the proposed results in the form of a push notification or an in-app message.

[1516] Input: Prompt for suggested action

[1517] Output: Proposal message displayed on the terminal

[1518] Specific operation: The server uses the notification system to send the generated prompt text to the terminal and notify the user.

[1519] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1520] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1521] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1522] [Fourth embodiment]

[1523] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1524] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1525] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1526] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1527] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1528] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1529] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1530] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1531] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1532] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1533] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1534] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1535] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1536] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[1537] Basic configuration

[1538] 1. Headgear

[1539] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1540] 2. Terminal

[1541] The user's smartphone, PC, or other device receives the brainwave data sent from the headgear. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[1542] 3. Server

[1543] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1544] Program processing overview

[1545] Server Processing

[1546] 1. Data Reception

[1547] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1548] 2. Data Preprocessing

[1549] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[1550] 3. Frequency analysis

[1551] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1552] 4. Mental State Estimation

[1553] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1554] 5. Proposal Generation

[1555] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[1556] 6. Notification

[1557] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[1558] Terminal handling

[1559] 1. Data Reception

[1560] The device receives real-time EEG data from the headgear, which is temporarily stored in a buffer within the device.

[1561] 2. Data Transmission

[1562] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1563] 3. Receiving and displaying notifications

[1564] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[1565] User operations

[1566] 1. Put on the headgear

[1567] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1568] 2. Feedback Check

[1569] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1570] 3. Action Practice

[1571] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1572] Specific examples

[1573] Example 1: Mental health care

[1574] To relieve work-related stress, the user wears headgear to collect brainwave data. The server analyzes the data and determines that "alpha waves are high, indicating a state of relaxation." The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[1575] Example 2: Improved concentration

[1576] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that their beta waves are high and they are in a state of concentration. The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[1577] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by using electroencephalogram data to analyze the user's mental state and suggest appropriate actions.

[1578] The processing flow will be explained below.

[1579] Step 1:

[1580] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[1581] Step 2:

[1582] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[1583] Step 3:

[1584] The device removes noise from the EEG data stored in the buffer by applying low-pass and high-pass filters.

[1585] Step 4:

[1586] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[1587] Step 5:

[1588] The server receives the EEG data sent from the device, and the received data undergoes further preprocessing such as noise removal and filtering.

[1589] Step 6:

[1590] The server applies a Fourier transform (FFT) to the preprocessed data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1591] Step 7:

[1592] The server evaluates the intensity of each frequency band and uses this to estimate the user's mental state. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration.

[1593] Step 8:

[1594] The server generates appropriate action suggestions for the user based on the estimated mental state, for example, "You are currently in a relaxed state. Please continue to take deep breaths."

[1595] Step 9:

[1596] The server notifies the user of the generated suggestions in real time, supporting the user's actions.

[1597] Step 10:

[1598] The device receives the suggestions and displays them in an easy-to-understand format to the user. Notifications are sent via push notifications or special screens within the app.

[1599] Step 11:

[1600] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[1601] Step 12:

[1602] The user puts on the headgear again, and EEG data is collected after the behavior is performed. This data is again sent to the terminal, and the process repeats from step 1.

[1603] In this way, the system repeats the processing steps of collecting the user's brainwave data, analyzing it, notifying them of suggestions, and then providing feedback, continuously supporting the user's mental state.

[1604] Example 1

[1605] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1606] Stress and decreased concentration have become problems in modern society. Conventional methods have made it difficult to quickly provide effective solutions to these problems. Therefore, the challenge is to provide a system that can analyze EEG data to grasp the user's mental state in real time and suggest appropriate actions.

[1607] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1608] In this invention, the server includes a device for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the device, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, and notification means for notifying the user's terminal of the suggestion result, thereby making it possible to grasp the user's mental state in real time and suggest appropriate actions.

[1609] "EEG data" refers to digital signals obtained by detecting electrical activity in the brain.

[1610] The "device" refers to a device worn on the head to collect EEG data, and is equipped with multiple electrodes.

[1611] "Terminal" refers to an electronic device that receives the brainwave data transmitted from the device, including smartphones and PCs.

[1612] The "processing means" is a means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing such as noise removal.

[1613] The "analysis means" refers to a means for frequency-analyzing the preprocessed electroencephalogram data, and uses an analysis method such as Fourier transform.

[1614] The "estimation means" is a means for estimating the state of mind from the data frequency-analyzed by the analysis means.

[1615] The "suggestion means" is a means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means.

[1616] The "notification means" is a means for notifying the user terminal of the result of the proposal made by the proposal means.

[1617] The present invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. Specific embodiments of this system will be described below.

[1618] Basic configuration

[1619] 1. Device for collecting EEG data

[1620] The user wears a device on their head to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1621] 2. Terminal

[1622] The EEG data sent from the device is received by the user's smartphone, PC, or other device, which communicates with the device using a connection method such as Bluetooth or USB.

[1623] 3. Server

[1624] A server that receives EEG data sent from the device and performs various processing. The server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1625] Program processing overview

[1626] Server Processing

[1627] 1. Data Reception

[1628] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1629] 2. Data Preprocessing

[1630] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it in a form suitable for analysis.

[1631] 3. Frequency analysis

[1632] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT), which categorizes it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1633] 4. Mental State Estimation

[1634] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1635] 5. Proposal Generation

[1636] The server then suggests appropriate actions to the user based on the estimated mental state. These suggestions are based on past data and general health information. For example, a generative AI model could be used to generate specific suggestions such as "Try taking deep breaths."

[1637] 6. Notification

[1638] The server notifies the user of the results in real time, either as a pop-up or in-app message.

[1639] Terminal handling

[1640] 1. Data Reception

[1641] The terminal receives EEG data from the device in real time, and the received data is temporarily stored in a buffer within the terminal.

[1642] 2. Data Transmission

[1643] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1644] 3. Receiving and displaying notifications

[1645] The device receives the proposed results sent from the server and displays them in a user-friendly format via push notifications or a special screen within the app.

[1646] User operations

[1647] 1. Wearing the device

[1648] The user first puts on the device and connects it to a smartphone or PC to begin collecting brainwave data.

[1649] 2. Feedback Check

[1650] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1651] 3. Action Practice

[1652] The user can then put the device back on to perform the suggested action and receive feedback.

[1653] Specific examples

[1654] Example 1: Mental health care

[1655] To relieve work-related stress, the user wears the device and brainwave data is collected. The server analyzes the data and determines that the user's alpha waves are high, indicating a state of relaxation. The device then displays a message saying, "Please continue to take deep breaths to maintain relaxation."

[1656] Example 2: Improved concentration

[1657] Students wear a device to measure their concentration levels during breaks from studying. The server analyzes the data and determines that "beta waves are high and you are in a state of concentration." The device then displays a message saying, "You are still in a state of concentration. It would be a good idea to continue working."

[1658] Example input to a generative AI model

[1659] "Please suggest specific actions to relieve stress. Based on the user's brain wave data, we have determined that their alpha waves are currently high."

[1660] In this way, an embodiment of the invention provides a system that supports mental health care and improved concentration by analyzing the user's mental state in real time using electroencephalogram data and suggesting appropriate actions.

[1661] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1662] Step 1:

[1663] EEG data collection

[1664] The user wears a device on their head to collect brainwave data. The device uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The input is the user's brain's electrical activity, and the output is digitized brainwave data. Specifically, the electrodes capture the brainwaves, and the built-in A / D converter converts the analog signal into digital data.

[1665] Step 2:

[1666] Transmission of brainwave data to a terminal

[1667] The device transmits the collected EEG data to the user's smartphone or PC via Bluetooth or USB connection. The input is digitized EEG data, and the output is EEG data transmitted to the terminal. Specifically, the communication module inside the device encrypts the data and transmits it to the terminal wirelessly or via a wired connection.

[1668] Step 3:

[1669] Data buffering on the device

[1670] The device temporarily stores the received EEG data in a buffer. The input is the EEG data sent from the device, and the output is the data stored in the buffer. Specifically, a dedicated app on the device runs in the background, receiving and saving data in real time.

[1671] Step 4:

[1672] Sending data to the server

[1673] The device sends the EEG data stored in the buffer to the server using a secure communication protocol. The input is the EEG data stored in the buffer, and the output is the data sent to the server. Specifically, a dedicated app on the device compiles the data and sends it to the cloud server using a protocol such as HTTPS.

[1674] Step 5:

[1675] Data reception

[1676] The server receives the EEG data sent from the device in real time. The input is the EEG data sent from the device, and the output is the data received within the server. Specifically, the server's receiving module decodes the data and stores it in storage.

[1677] Step 6:

[1678] Data Preprocessing

[1679] The server uses low-pass and high-pass filters to remove noise from the received EEG data and prepare it for analysis. The input is the received raw data, and the output is the noise-removed data. Specifically, a pre-processing algorithm in the server filters the data and removes unnecessary frequency components.

[1680] Step 7:

[1681] Frequency Analysis

[1682] The server performs frequency analysis on the preprocessed EEG data using a fast Fourier transform (FFT). The input is noise-removed data, and the output is data classified into each frequency band. Specifically, the FFT algorithm is applied to classify the data into alpha waves, beta waves, theta waves, delta waves, gamma waves, etc.

[1683] Step 8:

[1684] Mental state estimation

[1685] The server evaluates the strength of each frequency band from the analyzed data and estimates the user's mental state based on specific patterns. The input is data classified into each frequency band, and the output is an estimated mental state. Specifically, it uses a machine learning model to identify states such as relaxation, concentration, and drowsiness.

[1686] Step 9:

[1687] Proposal Generation

[1688] The server then suggests appropriate actions to the user based on the estimated mental state. The input is the estimated mental state, and the output is the suggested action. Specific operations include using a generative AI model to generate specific suggestions such as "Try taking deep breaths."

[1689] Step 10:

[1690] Proposal Notification

[1691] The server notifies the user's device of the proposed action in real time. The input is the proposed action, and the output is a notification displayed on the user's device. Specifically, the server's notification module converts the proposed action into an appropriate format and sends it to the device.

[1692] Step 11:

[1693] Check and implement user feedback

[1694] The user checks the suggested results displayed on the device and performs specific actions. The input is the notification received from the server, and the output is the change in the user's mental state as a result of the user performing the action. Specific actions include the user taking a deep breath or taking a break.

[1695] (Application example 1)

[1696] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1697] In modern society, it is extremely important to accurately grasp a user's psychological state in real time and provide products and services that correspond to that state. However, existing virtual store systems have difficulty accurately analyzing a user's psychological state and recommending products based on that state. In particular, they lack the ability to propose appropriate products and services that respond to subtle changes in the user's psychological state, such as whether the user is relaxed or focused. This reduces the quality of the user experience and hinders improvement in satisfaction. The present invention aims to solve these problems and provide detailed product recommendations in real time based on the user's psychological state.

[1698] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1699] In this invention, the server includes headgear for collecting electroencephalogram data, a terminal for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the terminal as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to each user based on the mental state estimated by the estimation means, notification means for notifying the user's terminal of the suggestion result, and recommendation means for recommending products in a virtual store based on the mental state estimated in real time. This enables optimal product recommendations based on the user's psychological state, increasing user satisfaction and providing a personalized experience tailored to each user.

[1700] "Electroencephalogram data" is digital data that indicates the electrical activity of the brain and is collected by headgear worn on the head.

[1701] "Headgear" is a device that is worn on the user's head and collects brain wave data using multiple electrodes.

[1702] A "terminal" is a device such as a smartphone or PC that receives brainwave data sent from the headgear and sends it to a server.

[1703] The "processing means" refers to a device that receives EEG data transmitted from a terminal as input and has the function of preprocessing the data using noise removal, low-pass filters, high-pass filters, etc.

[1704] The "analysis means" is a mechanism used to perform frequency analysis on preprocessed EEG data, specifically, a mechanism that has the function of classifying the data into frequency bands using a Fourier transform (FFT).

[1705] The "estimation means" has a function of estimating the user's mental state from the data frequency-analyzed by the analysis means.

[1706] The "suggestion means" has a function of suggesting appropriate actions to each user based on the mental state estimated by the estimation means.

[1707] The "notification means" has a function of notifying the user terminal of the result of the proposal made by the proposal means.

[1708] The "recommendation means" has the function of recommending products within a virtual store based on the results of real-time estimation of the user's mental state.

[1709] This invention is a system that analyzes electroencephalogram data, grasps the user's mental state, and suggests appropriate actions. This system is mainly composed of headgear, a terminal, and a server.

[1710] Basic configuration

[1711] 1. Headgear

[1712] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1713] 2. Terminal

[1714] The brainwave data sent from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB. The device also has the role of sending the received data to a server.

[1715] 3. Server

[1716] The server receives the EEG data sent from the device and performs various processing. It is installed on the cloud and uses a high-performance processor to preprocess and analyze the data. The server-side processing mainly consists of the following steps.

[1717] Server Processing

[1718] 1. Data Reception

[1719] The server receives the EEG data sent from the device in real time. This data may contain noise, so it is first subjected to noise removal.

[1720] 2. Data Preprocessing

[1721] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it for analysis. Signal processing libraries such as SciPy are used here.

[1722] 3. Frequency analysis

[1723] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1724] 4. Mental State Estimation

[1725] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's mental state based on specific patterns, such as relaxation, concentration, or drowsiness.

[1726] 5. Proposal Generation

[1727] Based on the estimated mental state, the system suggests appropriate actions to the user, based on past data and general health information.

[1728] 6. Notification

[1729] The results of the suggestions are sent to the user's device in real time, either as a pop-up or in-app message.

[1730] User operations

[1731] 1. Put on the headgear

[1732] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1733] 2. Feedback Check

[1734] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. It would be good to take a longer break" is displayed.

[1735] 3. Action Practice

[1736] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1737] Specific examples

[1738] As a specific example, by introducing this system into a virtual store, if a user is relaxed, it can recommend products that have a relaxing effect, and if a user is concentrating, it can suggest products that will further enhance concentration. This allows users to select products that best suit their own psychological state, providing a more satisfying shopping experience.

[1739] Prompt Sentence Examples

[1740] While a user is moving around in a virtual store, analyze their brainwave data in real time. If it is determined that the user is in a relaxed state, recommend products that have a relaxing effect to the user.

[1741] In this way, the present invention can realize detailed product recommendations based on the user's psychological state, improving the quality of the user's experience in a virtual store.

[1742] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1743] Step 1:

[1744] Headgear installation and data collection

[1745] The user wears a headgear to collect EEG data. The headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs it as digital data. The output data from the headgear is then sent to a terminal.

[1746] Input: User's electrical brain activity

[1747] Output: Digital data sent from the headgear to the device

[1748] Step 2:

[1749] Receiving data

[1750] The device receives the EEG data transmitted from the headgear in real time, and the received data is temporarily stored in a buffer within the device.

[1751] Input: Digital data transmitted from the headgear

[1752] Output: EEG data stored in a temporary buffer on the device

[1753] Step 3:

[1754] Sending data

[1755] The device transmits the buffered EEG data to a server at regular intervals using a secure communication protocol (e.g., HTTPS).

[1756] Input: EEG data stored in a temporary buffer on the device

[1757] Output: EEG data sent to the server

[1758] Step 4:

[1759] Data reception and noise reduction

[1760] The server receives the EEG data sent from the device in real time. Because the received data may contain noise, it is first subjected to noise removal. Specifically, low-pass and high-pass filters are used to remove unnecessary frequency components.

[1761] Input: EEG data sent from the device

[1762] Output: Denoised data

[1763] Step 5:

[1764] Frequency Analysis

[1765] The pre-processed (noise-removed) EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies the data into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1766] Input: Denoised data

[1767] Output: Data classified into each frequency band

[1768] Step 6:

[1769] Mental state estimation

[1770] The analyzed data is used to evaluate the strength of each frequency band and estimate the user's state of mind based on specific patterns. For example, a high level of alpha waves indicates relaxation, while a high level of beta waves indicates concentration. AI models may be used for this estimation.

[1771] Input: Data classified into each frequency band

[1772] Output: Estimated state of mind

[1773] Step 7:

[1774] Proposal Generation

[1775] Based on the estimated mental state, the system suggests appropriate actions to the user. For example, if the user is in a relaxed state, it will recommend relaxation products, and if the user is in a concentrated state, it will recommend products that will improve concentration. These suggestions are generated on the server side.

[1776] Input: Inferred state of mind

[1777] Output: Suggested actions or products

[1778] Step 8:

[1779] Notifications and Recommendations

[1780] The results of the recommendations are sent to the user's device in real time as pop-ups or in-app messages, and product recommendations are also made in the virtual store.

[1781] Input: Suggested action or product

[1782] Output: Notification to user device and product recommendations

[1783] Step 9:

[1784] Reviewing feedback

[1785] The user can check the analysis results and suggestions sent from the server on their device. For example, specific advice such as "You are currently in a relaxed state. We recommend the relaxation goods section" is displayed.

[1786] Input: Notification to user terminal

[1787] Output: User confirms the proposal

[1788] Step 10:

[1789] Action implementation and additional data collection

[1790] The user performs the suggested action and receives feedback by putting the headgear back on to collect additional EEG data, a process that allows the system to make even more personalized suggestions.

[1791] Input: Additional EEG data after user action

[1792] Output: Collected data on new headgear worn

[1793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1794] The present invention is a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and suggests appropriate actions, and specific embodiments thereof will be described below.

[1795] Basic configuration

[1796] 1. Headgear

[1797] The user wears a headgear to collect EEG data, which uses multiple electrodes to capture electrical activity in the brain in real time and output it as digital data.

[1798] 2. Terminal

[1799] The brainwave data transmitted from the headgear is received by a device such as a smartphone or PC. The device communicates with the headgear using a connection method such as Bluetooth or USB.

[1800] 3. Server

[1801] A server that receives and processes EEG data sent from the device. This server is installed on the cloud and uses a high-performance processor to preprocess and analyze the data.

[1802] Program processing overview

[1803] Server Processing

[1804] 1. Data Reception

[1805] The server receives the EEG data sent from the device in real time, and the received data is preprocessed to remove noise.

[1806] 2. Data Preprocessing

[1807] The received EEG data is filtered using low-pass and high-pass filters to remove noise and prepare it in a form suitable for analysis.

[1808] 3. Frequency analysis

[1809] The preprocessed EEG data is subjected to frequency analysis using a Fourier transform (FFT), which classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1810] 4. Emotional state estimation using emotion engine

[1811] Based on the frequency analysis results, an emotion engine is used to estimate the user's emotional state from the EEG data. This emotion engine performs emotion estimation based on a trained model using a machine learning algorithm.

[1812] 5. Mental State Estimation

[1813] The frequency-analyzed data is used to estimate the user's mental state (relaxed, focused, excited, etc.) along with their emotional state.

[1814] 6. Proposal Generation

[1815] Based on the estimated mental and emotional state, the system generates appropriate action suggestions for the user, such as "You are currently in a relaxed state, so please continue to take deep breaths" or "You are feeling stressed, so please take a short walk."

[1816] 7. Notification

[1817] The results of the suggestions are sent to the user's device in real time via push notifications or in-app messages.

[1818] Terminal handling

[1819] 1. Data Reception

[1820] The device receives real-time brainwave data from the headgear and temporarily stores it in a buffer.

[1821] 2. Data Transmission

[1822] The device sends the data stored in the buffer to the server using a secure communication protocol.

[1823] 3. Receiving and displaying notifications

[1824] The system receives the proposal results sent from the server and displays them in a format that is easy for users to understand. Notifications are sent via push notifications or special screens within the app.

[1825] User operations

[1826] 1. Put on the headgear

[1827] The user first puts on the headgear and connects it to a smartphone or PC to begin collecting brainwave data.

[1828] 2. Feedback Check

[1829] The user can check the analysis results and suggestions sent from the server on their device. For example, advice such as "You are currently in a relaxed state" or "You are feeling stressed. Relax" will be displayed.

[1830] 3. Action Practice

[1831] The user can then put the headgear back on to perform the suggested actions and receive feedback.

[1832] Specific examples

[1833] Example 1: Mental health care

[1834] To relieve work-related stress, the user wears headgear to collect brain wave data. The server analyzes the data and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[1835] Example 2: Improved concentration

[1836] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[1837] In this way, an embodiment of the invention specifically realizes a system that supports mental health care and improved concentration by analyzing the user's mental state using brain wave data and emotional state and suggesting appropriate actions.

[1838] The processing flow will be explained below.

[1839] Step 1:

[1840] The user wears the headgear and connects it to a device such as a smartphone or PC using a communication method such as Bluetooth or USB.

[1841] Step 2:

[1842] The device receives brainwave data from the headgear in real time and temporarily stores it in a buffer.

[1843] Step 3:

[1844] The device preprocesses the EEG data stored in the buffer, specifically by applying low-pass and high-pass filters to remove noise.

[1845] Step 4:

[1846] The device transmits the pre-processed EEG data to a server using a secure communication protocol.

[1847] Step 5:

[1848] The server receives the EEG data sent from the device, and then performs pre-processing such as filtering and noise removal on the received data.

[1849] Step 6:

[1850] The server applies a Fourier transform (FFT) to the preprocessed EEG data and classifies it into frequency bands such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1851] Step 7:

[1852] The server evaluates the data from each frequency band and uses this to estimate the user's mental state (relaxed, focused, excited, etc.).

[1853] Step 8:

[1854] The server then uses an emotion engine to estimate the user's emotional state (happiness, anxiety, stress, etc.) from the EEG data. The emotion engine uses machine learning algorithms to make predictions based on a trained model.

[1855] Step 9:

[1856] The server generates appropriate action suggestions for the user based on the estimated mental and emotional state, such as "You are currently in a relaxed state. Please continue to take deep breaths" or "You are feeling stressed. It would be good to take a short walk."

[1857] Step 10:

[1858] The server generates suggestions and sends them to the user's device in real time, either as push notifications or in-app messages.

[1859] Step 11:

[1860] The device receives the suggestions and displays them in a format that is easy for the user to understand. Notifications are sent via push notifications or special screens within the app.

[1861] Step 12:

[1862] The user adjusts their behavior according to the advice given, for example, by taking deep breaths or a short break.

[1863] Step 13:

[1864] The user puts on the headgear again and collects EEG data after performing the action. This allows the server to analyze the effect of the user's action again and provide feedback. This data is again sent to the device, where it is analyzed by the server, and the process from step 1 is repeated.

[1865] In this way, the system repeats processing steps from collecting the user's brainwave data to estimating their emotional state, notifying them of suggested actions, and providing feedback on the results of their actions, thereby continuously supporting the user's mental and emotional state.

[1866] Example 2

[1867] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1868] The present invention aims to provide a system that accurately grasps a user's mental and emotional state based on electroencephalogram (EEG) data and provides appropriate action suggestions in real time based on the results. Existing technologies have problems such as inaccurate analysis of EEG data and action suggestions, and difficulty in real-time notification. In particular, the objective is to solve the problem of low accuracy in noise removal and emotion estimation, making it difficult to provide accurate feedback to the user.

[1869] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1870] In this invention, the server includes a device for collecting electroencephalogram data, a communication device for receiving the electroencephalogram data transmitted from the device, a calculation device for receiving the electroencephalogram data transmitted from the communication device as input and performing preprocessing, a computation device for frequency analyzing the preprocessed electroencephalogram data, an estimation device for estimating a mental state from the data frequency analyzed by the computation device, a suggestion device for making appropriate action suggestions to individual users based on the mental state estimated by the estimation device, and a notification device for notifying the user's communication device of the suggestion results. This makes it possible to analyze noise-removed electroencephalogram data in real time, accurately estimate the mental state and emotional state, and provide appropriate action suggestions to users in real time.

[1871] "Electroencephalogram data" is information that represents in digital form signals that measure the electrical activity of the brain.

[1872] "Device" refers to equipment including headgear and sensors worn on the user's head to collect brainwave data.

[1873] A "communication device" is a device such as a smartphone or PC that receives the brainwave data sent from the device and sends it to a server.

[1874] A "computing device" is a device that includes a computer or processor for pre-processing electroencephalogram data received from a communication device.

[1875] The "arithmetic unit" is a device for frequency analysis of preprocessed electroencephalogram data, and executes algorithms such as Fourier transform.

[1876] The "estimation device" is a device for estimating the mental state of a user from frequency analysis data obtained by a computing device, and uses a machine learning algorithm.

[1877] The "suggestion device" is a device that suggests appropriate actions to a user based on the mental state estimated by the estimation device.

[1878] The "notification device" is a device that notifies the user's communication device of the proposal results from the proposal device in real time.

[1879] The present invention provides a system that analyzes electroencephalogram data, grasps the mental and emotional state of a user, and proposes appropriate actions. This system is implemented using the following main hardware and software components.

[1880] 1. Headgear

[1881] The user wears a headgear to collect EEG data. This headgear uses multiple electrodes to capture the brain's electrical activity in real time and outputs the signals as digital data, allowing for accurate collection of EEG data.

[1882] 2. Terminal

[1883] The brainwave data sent from the headgear is received by a device such as a smartphone or PC via a connection method such as Bluetooth or USB. The device temporarily stores the received data in a buffer and then transmits it to a server using a secure communication protocol.

[1884] 3. Server

[1885] The server receives the EEG data sent from the device in real time and performs the following processing. First, it applies a low-pass filter and a high-pass filter to remove noise. Next, it uses a Fourier transform (FFT) on the preprocessed EEG data to perform frequency analysis. This allows it to be classified into various frequency bands, such as alpha waves, beta waves, theta waves, delta waves, and gamma waves.

[1886] Furthermore, an emotion engine is used to estimate the user's emotional state based on the analysis results. The emotion engine uses a machine learning algorithm to estimate emotions based on a model. Then, based on the estimated emotional state and frequency analysis data, the user's mental state (relaxed, focused, excited, etc.) is evaluated.

[1887] Based on these evaluation results, the server generates appropriate action suggestions for the user, such as "Keep taking deep breaths" or "Take a short walk." The generated suggestions are sent to the user's device in real time as push notifications or in-app messages.

[1888] Specific examples

[1889] Example 1: Mental health care

[1890] When a user puts on the headgear to relieve work stress, the server analyzes the situation and estimates that the user's alpha waves are high and they are in a relaxed state. Meanwhile, the emotion engine estimates that the user is in a happy state. The device displays a message saying, "You are in a relaxed state. Please continue to take deep breaths."

[1891] Example 2: Improved concentration

[1892] Students wear headgear to measure their concentration levels during breaks from studying. The server analyzes the data and determines that the student has high beta waves and is in a state of concentration. At the same time, the emotion engine estimates that the student is in a state of slight anxiety. The device displays a message saying, "You are concentrating, but you may be feeling anxious. It would be a good idea to take a short break."

[1893] Prompt Sentence Examples

[1894] Sample prompt 1: "After a user wears the headgear and collects EEG data, explain how that data can be analyzed to support mental health care."

[1895] Sample prompt 2: "Please explain with a concrete example how EEG data can be used to generate appropriate action suggestions to improve a user's focus."

[1896] As described above, the present invention provides a specific embodiment that supports mental health care and improved concentration by analyzing the user's mental state using electroencephalogram data and emotional state and suggesting appropriate actions.

[1897] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1898] Step 1: Receiving data

[1899] The server receives the EEG data transmitted from the device in real time. As input, the EEG data acquired by the device from the headgear is used. The received data is passed to a pre-processing step for noise reduction. Specifically, the data is temporarily stored in the server's receiving buffer.

[1900] Step 2: Preprocessing the data

[1901] The server preprocesses the received EEG data. In this step, noise is removed by applying low-pass and high-pass filters. The EEG data received in step 1 is used as input, and noise-removed EEG data is obtained as output. Specifically, a fast moving average algorithm is used to reduce noise in the signal.

[1902] Step 3: Frequency analysis

[1903] The server performs a Fourier transform (FFT) on the preprocessed EEG data. The input is the data from which noise was removed in step 2, and the output is data classified into frequency bands such as alpha, beta, theta, delta, and gamma waves. Specifically, the FFT algorithm is run to quantify the intensity of each frequency band.

[1904] Step 4: Estimating emotional state

[1905] The server uses an emotion engine to estimate the user's emotional state based on the results of the frequency analysis. The frequency data classified in step 3 is used as input, and the emotional state, such as relaxation, excitement, or stress, is obtained as output. Specifically, the emotion engine, which uses a machine learning algorithm, estimates the emotional state based on the model.

[1906] Step 5: Mental state estimation

[1907] The server estimates the user's mental state based on the emotional state and the frequency analysis results. The emotional state data from step 4 and the frequency data from step 3 are used as input, and the specific mental state (relaxed, focused, excited, etc.) is obtained as output. Specifically, the mental state is evaluated multidimensionally by combining multiple data indicators.

[1908] Step 6: Generate proposals

[1909] The server generates action suggestions for the user based on the estimated mental state. The mental state data obtained in step 5 is used as input, and specific action suggestions (e.g., "Continue to take deep breaths" or "Take a short break") are obtained as output. Specifically, suggestions are automatically generated according to predefined suggestion generation rules.

[1910] Step 7: Notification

[1911] The server notifies the generated action suggestions to the user's device in real time. The suggestion data generated in step 6 is used as input, and a notification message is sent to the user's device as output. Specifically, the server uses push notification and in-app messaging functions to provide instant feedback to the user.

[1912] Step 8: Put on the headgear

[1913] The user puts on the headgear and connects it to the terminal. The input is to confirm that the headgear is attached according to a specific procedure, and the output is to start collecting brainwave data. Specifically, the headgear's sensors measure the electrical activity of the brainwaves in real time and transmit the data to the terminal.

[1914] Step 9: Feedback confirmation

[1915] The user checks the suggestion displayed on the device and decides on the next action. The suggestion message notified in step 7 is used as input, and the action to be taken by the user is determined as output. Specifically, the device screen displays "Please continue to take deep breaths," and the user follows the advice.

[1916] Step 10: Take Action

[1917] The user puts on the headgear again to practice the suggested behavior and get feedback. The behavior determined in step 9 is used as input, and results such as improved performance or mood are obtained as output. Specifically, the user actually takes a deep breath, and EEG data is collected again to confirm the effect.

[1918] (Application example 2)

[1919] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1920] Systems already exist that use EEG data to understand a user's mental and emotional state and suggest appropriate actions. However, these systems are primarily limited to personal use, mental health care, and improving concentration, and have not been applied to improving customer service in brick-and-mortar stores. Another issue is that if suggestions are not made in real time, it is difficult to improve customer satisfaction.

[1921] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1922] In this invention, the server includes headgear for collecting electroencephalogram data, an information terminal device for receiving the electroencephalogram data transmitted from the headgear, processing means for receiving the electroencephalogram data transmitted from the information terminal device as input and performing preprocessing, analysis means for frequency-analyzing the preprocessed electroencephalogram data, estimation means for estimating a mental state from the data frequency-analyzed by the analysis means, suggestion means for suggesting appropriate actions to individual users based on the mental state estimated by the estimation means, notification means for notifying the user's information terminal device of the suggestion results, and suggestion means for suggesting appropriate services and products in real time based on the mental state and emotional state of the customer using an electroencephalogram data collection device temporarily worn by the customer in a physical store. This enables personalized service offerings based on the customer's emotional and mental state in a physical store.

[1923] "Electroencephalogram data" is information that measures the electrical activity of the brain and expresses it in digital form.

[1924] "Headgear" is a device worn on the head to collect the wearer's brain wave data.

[1925] An "information terminal device" is a device for receiving and processing brain wave data transmitted from the headgear, and includes smartphones, personal computers, etc.

[1926] The "processing means" is a means having a function for receiving and preprocessing electroencephalogram data transmitted from an information terminal device.

[1927] The "analysis means" is a mechanism for performing frequency analysis on preprocessed electroencephalogram data.

[1928] "Frequency analysis" is an analytical method that divides EEG data into multiple frequency bands, and uses techniques such as Fourier transform.

[1929] The "estimation means" is a means for estimating a mental state from data frequency-analyzed by the analysis means.

[1930] The "suggestion means" is a means for suggesting an appropriate action to the user based on the mental state estimated by the estimation means.

[1931] The "notification means" is a mechanism for notifying the user's information terminal device of the result of the proposal made by the proposal means.

[1932] "Brick and mortar store" refers to a physical store where customers visit to purchase products.

[1933] An "electroencephalogram data collection device" is a temporarily wearable device for collecting electroencephalogram data.

[1934] A "generative AI model" is an artificial intelligence model that has been trained to perform a specific task using machine learning algorithms.

[1935] A "prompt sentence" is a sentence of guidance or advice that is generated based on the user's mental and emotional state.

[1936] MODE FOR CARRYING OUT THE INVENTION

[1937] The present invention provides a system that analyzes electroencephalogram data collected from a user wearing a headgear, and aims to improve the customer experience, particularly in physical stores. Specific embodiments are described below.

[1938] System configuration

[1939] The system consists of the following main components:

[1940] 1. Headgear

[1941] This is a device that is temporarily worn by the user (customer) to collect brain wave data. It acquires brain wave data in real time and transmits it to an information terminal device.

[1942] 2. Information terminal device

[1943] Mobile devices such as smartphones and tablets are used to receive EEG data sent from the headgear and send it to a server.

[1944] 3. Server

[1945] The cloud-based system uses a high-performance processor to preprocess and analyze EEG data, specifically removing noise from the data, analyzing frequencies, estimating mental and emotional states, and generating appropriate action suggestions, which are then sent to the information terminal device.

[1946] Program processing overview

[1947] The server is equipped with the following processing means:

[1948] 1. Data Reception

[1949] The server receives the EEG data sent from the information terminal device in real time, and the data is first temporarily stored in storage.

[1950] 2. Noise Reduction and Preprocessing

[1951] Low-pass and high-pass filters are used to remove noise from the received EEG data, resulting in clean data suitable for analysis.

[1952] 3. Frequency analysis

[1953] The preprocessed EEG data is then classified by frequency using a Fourier transform (FFT), and waves that reflect psychological states, such as alpha waves, beta waves, and theta waves, are identified.

[1954] 4. Estimating Mental and Emotional States

[1955] Mental and emotional states are estimated from the analysis results using an estimation method based on a generative AI model using machine learning algorithms.

[1956] 5. Generating action suggestions

[1957] Based on the estimated mental and emotional state, appropriate action suggestions (e.g., information about products and services that will help customers relax) are generated.

[1958] 6. Notification

[1959] The generated suggestions are sent to the information terminal device in real time via push notifications or in-app messages.

[1960] Example

[1961] A specific example is given below.

[1962] Example 1: Proposing a relaxing environment

[1963] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[1964] The server analyzes the data and estimates the state of relaxation. Based on the generative AI model, it generates a prompt in real time, such as "You are in a relaxed state. You can use the relaxation area."

[1965] The user's information terminal device is notified.

[1966] Example 2: Proposing a stress-relieving product

[1967] A user wearing the headgear enters a physical store. The information terminal device transmits brain wave data to the server.

[1968] The server analyzes the data and estimates the state of stress. Based on the generative AI model, it generates a prompt in real time, such as "You are in a stressful state. Please try some stress relief products."

[1969] The user's information terminal device is notified.

[1970] Prompt Sentence Examples

[1971] An example of a prompt sentence to input to the generative AI model is as follows:

[1972] plaintext

[1973] A customer puts on smart glasses and enters a store. Analyze their brainwave data to estimate their current emotional and mental state. Based on the estimation results, generate a message suggesting appropriate services and products. For example, if the customer is in a relaxed state, introduce them to a relaxation area, or if they are in a stressed state, suggest stress-relieving products.

[1974] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1975] Step 1:

[1976] The user puts on the headgear and establishes a connection with the information terminal device.

[1977] Input: EEG data from the headgear

[1978] Output: Raw EEG data sent to an information terminal device

[1979] Specific operation: Electrodes in the headgear detect brain waves and transmit the data to an information terminal device via Bluetooth or Wi-Fi.

[1980] Step 2:

[1981] The terminal receives the brainwave data transmitted from the headgear and temporarily stores it in a buffer.

[1982] Input: Raw EEG data transmitted from the headgear

[1983] Output: Buffered EEG data for sending to the server

[1984] Specific operation: The information terminal device receives data packets from the headgear and buffers a certain amount of brain wave data.

[1985] Step 3:

[1986] The device sends the buffered EEG data to a server, where the data is transferred using a secure communication protocol.

[1987] Input: EEG data stored in a buffer

[1988] Output: EEG data sent to the server

[1989] Specific operation: The information terminal device uploads the buffered data to the server using the HTTPS or SSH protocol.

[1990] Step 4:

[1991] The server receives the brainwave data in real time and temporarily stores it in storage.

[1992] Input: EEG data sent from the device

[1993] Output: Saved EEG data

[1994] Specific operation: The server uses the data receiving module to continuously receive streams of data from the terminal and store them in storage.

[1995] Step 5:

[1996] The server denoises and preprocesses the EEG data, using low-pass and high-pass filters to remove noise and generate clean data.

[1997] Input: Stored EEG data

[1998] Output: Noise-removed EEG data

[1999] What it does: Preprocessing algorithms remove high- and low-frequency noise to prepare data suitable for analysis.

[2000] Step 6:

[2001] The server performs frequency analysis on the preprocessed EEG data, classifying it into frequency bands using a Fourier transform (FFT).

[2002] Input: EEG data after noise removal

[2003] Output: EEG data categorized by frequency

[2004] Specific operation: Runs the FFT algorithm to decompose brainwave data into alpha waves, beta waves, theta waves, etc.

[2005] Step 7:

[2006] The server uses the frequency analysis results to estimate the mental and emotional state, and utilizes a generative AI model.

[2007] Input: Frequency analysis results

[2008] Output: Inferred mental and emotional state

[2009] Specific operation: The analysis results are input into a generative AI model, which estimates mental state (relaxed, stressed, etc.) and emotional state (happiness, anxiety, etc.) based on a trained algorithm.

[2010] Step 8:

[2011] Based on the estimation results, the server generates appropriate action suggestions and creates prompt sentences.

[2012] Input: Inferred mental and emotional states

[2013] Output: Action suggestion prompt

[2014] Specific operation: A rule-based engine is activated to provide action suggestions based on the inference results, generating prompt statements such as "You are in a relaxed state. You can use the relaxation area."

[2015] Step 9:

[2016] The server notifies the device of the proposed results in the form of a push notification or an in-app message.

[2017] Input: Prompt for suggested action

[2018] Output: Proposal message displayed on the terminal

[2019] Specific operation: The server uses the notification system to send the generated prompt text to the terminal and notify the user.

[2020] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2021] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2022] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2023] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2024] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2025] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2026] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2027] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2028] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2029] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2030] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2031] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2032] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2033] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2034] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2035] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2036] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2037] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2038] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2039] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2040] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2041] The following is further disclosed regarding the above embodiment.

[2042] (Claim 1)

[2043] Headgear that collects brainwave data,

[2044] a terminal for receiving the electroencephalogram data transmitted from the headgear;

[2045] a processing means for receiving as input the electroencephalogram data transmitted from the terminal and performing preprocessing;

[2046] an analysis means for frequency-analyzing the preprocessed electroencephalogram data;

[2047] an estimation means for estimating a state of mind from the data frequency-analyzed by the analysis means;

[2048] a suggestion means for suggesting an appropriate action to each user based on the state of mind estimated by the estimation means;

[2049] a notification means for notifying a user terminal of a result of the proposal by the proposal means;

[2050] A system including:

[2051] (Claim 2)

[2052] 2. The system according to claim 1, wherein the analyzing means classifies the electroencephalogram data into a plurality of frequency bands using a Fourier transform.

[2053] (Claim 3)

[2054] 2. The system according to claim 1, wherein the suggestion means compares the electroencephalogram data with the user's past electroencephalogram data to evaluate fluctuations and trends.

[2055] "Example 1"

[2056] (Claim 1)

[2057] a device for collecting electroencephalogram data;

[2058] a terminal for receiving the electroencephalogram data transmitted from the device;

[2059] a processing means for receiving as input the electroencephalogram data transmitted from the terminal and performing preprocessing;

[2060] an analysis means for frequency-analyzing the preprocessed electroencephalogram data;

[2061] an estimation means for estimating a state of mind from the data frequency-analyzed by the analysis means;

[2062] a suggestion means for suggesting an appropriate action to each user based on the state of mind estimated by the estimation means;

[2063] a notification means for notifying a user terminal of a result of the proposal by the proposal means;

[2064] A system including:

[2065] (Claim 2)

[2066] 2. The system according to claim 1, wherein the analyzing means classifies the electroencephalogram data into a plurality of frequency bands using a Fourier transform.

[2067] (Claim 3)

[2068] 2. The system according to claim 1, wherein the suggestion means compares the electroencephalogram data with the user's past electroencephalogram data to evaluate fluctuations and trends.

[2069] "Application Example 1"

[2070] (Claim 1)

[2071] Headgear that collects brainwave data,

[2072] a terminal for receiving the electroencephalogram data transmitted from the headgear;

[2073] a processing means for receiving as input the electroencephalogram data transmitted from the terminal and performing preprocessing;

[2074] an analysis means for frequency-analyzing the preprocessed electroencephalogram data;

[2075] an estimation means for estimating a state of mind from the data frequency-analyzed by the analysis means;

[2076] a suggestion means for suggesting an appropriate action to each user based on the state of mind estimated by the estimation means;

[2077] a notification means for notifying a user terminal of a result of the proposal by the proposal means;

[2078] a recommendation means for recommending products in a virtual store based on the estimated mental state in real time;

[2079] A system including:

[2080] (Claim 2)

[2081] 2. The system according to claim 1, wherein the analyzing means classifies the electroencephalogram data into a plurality of frequency bands using a Fourier transform.

[2082] (Claim 3)

[2083] 2. The system according to claim 1, wherein the suggestion means compares the electroencephalogram data with the user's past electroencephalogram data to evaluate fluctuations and trends.

[2084] "Example 2: Combining Emotion Engines"

[2085] (Claim 1)

[2086] a device for collecting electroencephalogram data;

[2087] a communication device for receiving the electroencephalogram data transmitted from the device;

[2088] a computing device for receiving as input the electroencephalogram data transmitted from the communication device and performing preprocessing;

[2089] a computing device for frequency-analyzing the preprocessed electroencephalogram data;

[2090] an estimation device that estimates a mental state from data frequency-analyzed by the arithmetic device;

[2091] a suggestion device that suggests appropriate actions to individual users based on the mental state estimated by the estimation device;

[2092] a notification device that notifies a communication device of a user of a proposal result by the proposal device;

[2093] A system including:

[2094] (Claim 2)

[2095] 2. The system according to claim 1, wherein the arithmetic unit uses a Fourier transform for frequency analysis.

[2096] (Claim 3)

[2097] The system according to claim 1, characterized in that the suggestion device compares the user's past electroencephalogram data to evaluate fluctuations and trends.

[2098] (Claim 4)

[2099] 10. The system of claim 1, wherein the computing device uses a low-pass filter and a high-pass filter for noise removal.

[2100] (Claim 5)

[2101] 2. The system of claim 1, wherein the estimation device uses an emotion engine that estimates an emotional state based on frequency analysis data.

[2102] (Claim 6)

[2103] 2. The system according to claim 1, wherein the notification device notifies the communication device of the proposal result in real time.

[2104] "Application example 2 when combining emotion engines"

[2105] (Claim 1)

[2106] Headgear that collects brainwave data,

[2107] an information terminal device for receiving the electroencephalogram data transmitted from the headgear;

[2108] a processing means for receiving as input the electroencephalogram data transmitted from the information terminal device and performing preprocessing;

[2109] an analysis means for frequency-analyzing the preprocessed electroencephalogram data;

[2110] an estimation means for estimating a mental state from the data frequency-analyzed by the analysis means;

[2111] a suggestion means for suggesting an appropriate action to each user based on the mental state estimated by the estimation means;

[2112] a notification means for notifying a user of a result of the proposal made by said proposal means to an information terminal device of the user;

[2113] a suggestion means for suggesting appropriate services and products in real time based on the mental and emotional states of customers using an electroencephalogram data collection device temporarily worn by customers in a physical store;

[2114] A system including:

[2115] (Claim 2)

[2116] 2. The system according to claim 1, wherein the analyzing means classifies the electroencephalogram data into a plurality of frequency bands using a Fourier transform.

[2117] (Claim 3)

[2118] 2. The system according to claim 1, wherein the suggestion means compares the electroencephalogram data with the user's past electroencephalogram data to evaluate fluctuations and trends.

[2119] (Claim 4)

[2120] The system according to claim 1, characterized in that the suggestion means suggests appropriate services and products in real time based on the mental state of the customer in the physical store.

[2121] (Clai...

Claims

1. Headgear that collects brainwave data, a terminal for receiving the electroencephalogram data transmitted from the headgear; a processing means for receiving as input the electroencephalogram data transmitted from the terminal and performing preprocessing; an analysis means for frequency-analyzing the preprocessed electroencephalogram data; an estimation means for estimating a state of mind from the data frequency-analyzed by the analysis means; a suggestion means for suggesting an appropriate action to each user based on the state of mind estimated by the estimation means; a notification means for notifying a user terminal of a result of the proposal by the proposal means; A system including:

2. 2. The system according to claim 1, wherein said analyzing means classifies the electroencephalogram data into a plurality of frequency bands using a Fourier transform.

3. 2. The system according to claim 1, wherein the suggestion means compares the electroencephalogram data with the user's past electroencephalogram data to evaluate fluctuations and trends.

Images