System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring brainwave signals of a user using a non-invasive brainwave sensor; means for pre-processing the brainwave signals and performing denoising and filtering; means for transmitting the pre-processed brainwave signals to a server using a secure protocol; means for extracting features from the brainwave signals and analyzing the features using a machine-learning model at the server; and means for generating outputs such as text, images, audio, and video using a generative AI based on the analysis results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional IoT technology connects things to things, and new technologies that seamlessly connect people are needed. In particular, people with physical disabilities have limited means to express their thoughts efficiently, and business people lack ways to quickly and accurately communicate important information. Furthermore, there are no adequate solutions to problems such as decreased concentration, stress management, and preventing accidents caused by drowsiness. New technologies are needed to solve these issues. [Means for solving the problem]

[0005] This invention relates to a system that acquires a user's EEG signals using a non-invasive EEG sensor, preprocesses them, and then transmits the EEG data to a server using a secure protocol. The server extracts features from the received EEG data and analyzes it using a machine learning model. The analysis results are converted into output such as text, images, audio, and video using a generative AI and sent to a storage device. The storage device then provides feedback to the user. Furthermore, the system also includes a means for physically paralyzed individuals to mentally control assistive devices and a means for monitoring the user's concentration level and generating a warning if it falls below a threshold, thereby achieving seamless communication and efficient work performance.

[0006] A "non-invasive brainwave sensor" is a sensor that directly measures the user's brainwaves, but does not penetrate the skin or skull; it can obtain brainwave signals simply by contacting the outside.

[0007] "Electroencephalographic signals" are weak electrical signals generated by neural activity in the brain, and are generally recorded as EEG (Electroencephalography).

[0008] "Preprocessing" refers to a process of removing noise from the acquired EEG signal and improving the quality of the signal by filtering, etc.

[0009] A "secure protocol" is a communication method that protects the confidentiality and integrity of data by using methods such as encryption and authentication when sending and receiving data.

[0010] A "server" is a computer system that provides computing resources for analyzing and processing data.

[0011] "Feature extraction" is the process of extracting useful information and patterns from data, specifically the extraction of frequency components and amplitude of EEG signals.

[0012] A "machine learning model" is an algorithm or mathematical model that learns patterns from data and performs predictions, classifications, and other tasks.

[0013] "Generative AI" is an artificial intelligence technology that generates output such as text, images, audio, and video from input data. Examples include natural language processing models and image generation models.

[0014] A "storage device" is an electronic device that can store data and retrieve it as needed.

[0015] "Providing feedback to the user" refers to informing or displaying the generated output to the user in an appropriate format.

[0016] An "assistive device" is a device used to assist with a bodily function, examples of which include a wheelchair or a prosthetic limb.

[0017] "Monitoring concentration" is the process of analyzing the user's brainwave signals to assess their current concentration and attention.

[0018] "Generating an alert" refers to creating a message or notification to alert the user based on the assessment of the user's concentration and stress level. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that analyzes EEG signals acquired by an EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[0041] EEG data collection and preprocessing

[0042] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device then performs pre-processing on the received EEG signals, such as noise reduction and filtering, to improve signal quality and increase analysis accuracy.

[0043] Sending data to the server

[0044] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[0045] Analysis of EEG data

[0046] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's brainwave data. The analysis results in the user's thoughts and emotional state.

[0047] Realizing output through generative AI

[0048] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. The generative AI uses various generative models (e.g., natural language processing models and image generation models) to create highly accurate output.

[0049] Feedback of results

[0050] Server: The generated output data is formatted into an appropriate format, encrypted again, and sent to the terminal. The terminal decrypts the sent data and saves it to a storage device. The storage device notifies or displays the saved result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[0051] Specific examples

[0052] Example of assistive device operation for physically paralyzed people

[0053] User: A paralyzed person thinks about operating a wheelchair.

[0054] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0055] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." It then uses a generation AI to generate an operation command.

[0056] Storage device: Sends the generated instructions to the wheelchair control system, causing the wheelchair to perform the specified actions.

[0057] Example of concentration monitoring during a meeting

[0058] Users: Business people attending the meeting.

[0059] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0060] Server: Receives and analyzes the data. Based on the analysis results, the user's concentration level is evaluated. If it falls below a certain threshold, the generation AI generates a warning message.

[0061] Storage device: The generated warning message is sent to the device, which notifies the user, for example, by displaying "Your concentration level is decreasing" on the display.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[0065] Step 2:

[0066] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[0067] Step 3:

[0068] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[0069] Step 4:

[0070] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[0071] Step 5:

[0072] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[0073] Step 6:

[0074] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[0075] Step 7:

[0076] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[0077] Step 8:

[0078] Server: Based on the analysis results, generative AI is used to generate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[0079] Step 9:

[0080] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[0081] Step 10:

[0082] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[0083] Step 11:

[0084] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[0085] As a concrete example, the case of a physically paralyzed person operating a wheelchair is shown below.

[0086] Step 1:

[0087] User: A paralyzed person wears a brainwave sensor on their head to help them think about operating a wheelchair.

[0088] Step 2:

[0089] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0090] Step 3:

[0091] Terminal: Encrypts the preprocessed data and sends it to the server.

[0092] Step 4:

[0093] Server: Receives the data and extracts features after decoding.

[0094] Step 5:

[0095] Server: A machine learning model analyzes the user's thoughts, and the generation AI generates operation commands (e.g., "go forward," "go backward," "turn left," "turn right").

[0096] Step 6:

[0097] Server: Encrypts and transmits commands to the terminal.

[0098] Step 7:

[0099] Terminal: Decodes commands and sends instructions to the wheelchair control system.

[0100] Step 8:

[0101] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[0102] Through these specific process steps, the invention can be effectively implemented.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] The present invention aims to provide a system that acquires and analyzes a user's EEG signals in real time to accurately grasp the user's thoughts and emotions and provide appropriate feedback based on the acquired signals. Current systems face challenges in removing noise from EEG signals obtained from non-invasive EEG sensors, securely transmitting data, improving analytical accuracy, and providing feedback in a variety of output formats using generative AI. These challenges must be addressed, for example, when a physically paralyzed person needs to operate an assistive device by thought, or when monitoring a user's concentration level and generating appropriate warnings.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for decoding the EEG data and performing a fast Fourier transform to extract features, means for inputting the features into a machine learning model and analyzing them, and means for generating output such as text, images, audio, and video using a generative AI based on the analysis results. This enables highly accurate analysis based on the user's EEG signals and the generation of appropriate output.

[0108] A "non-invasive EEG sensor" is a sensor that is worn on the user's head to acquire EEG signals in real time, and is a device that can be used without damaging the user's skin.

[0109] "Preprocessing" refers to processing that removes noise and filters the EEG signal to improve the quality of the signal.

[0110] "Noise reduction" is the process of removing unnecessary noise and external interference from EEG signals.

[0111] "Filtering" is a process of extracting specific frequency components from an EEG signal or removing unnecessary frequency components.

[0112] A "secure protocol" is a communication method that ensures confidentiality and integrity of data and is a protocol that has encryption and authentication functions.

[0113] "Encryption" is a process of converting data to prevent the contents of the data being transmitted from being read by a third party.

[0114] "Decryption" is the process of restoring encrypted data to its original state.

[0115] "Feature extraction" is the process of finding the characteristics and patterns necessary for analysis from EEG signals.

[0116] The "Fast Fourier Transform (FFT)" is a mathematical technique for converting time-domain signals into the frequency domain for analysis.

[0117] A "machine learning model" is a model that learns rules and patterns from data and makes predictions and classifications.

[0118] "Generative AI" is a general term for artificial intelligence technology that generates output such as text, images, audio, and video based on analysis results.

[0119] "Text, image, audio, video, etc. output" refers to the form of information generated based on the user's thoughts and feelings.

[0120] "Formatting" is the process of putting data into a specific form.

[0121] "Storage Device" means a device that stores the generated output and communicates or displays it to the user when necessary.

[0122] This invention is a system that analyzes EEG signals acquired using a non-invasive EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[0123] Hardware and software used

[0124] Brainwave sensor: A non-invasive brainwave sensor is worn on the user's head to capture brainwave signals in real time. This sensor has wireless communication capabilities and transmits the brainwave signals to a device.

[0125] Terminal: The terminal processes the EEG signals received from the sensor. Pre-processing such as noise reduction and filtering is performed on the terminal. Specifically, band-pass filters and artifact removal algorithms are used to make the signals clearer.

[0126] Server: The server receives and decrypts the encrypted EEG data sent from the device. It then extracts features using an FFT (Fast Fourier Transform). The extracted features are input into a machine learning model to analyze the user's EEG data. Based on the analysis results, a generative AI generates output such as text, images, audio, and video. The generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[0127] Storage Device: A storage device is a device that stores the generated output data and provides feedback to the user, for example, by displaying a message on a display or playing an audio message through a speaker.

[0128] Specific examples

[0129] Example of assistive device operation for physically paralyzed people

[0130] User: A paralyzed person thinks about operating a wheelchair.

[0131] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0132] Server: Receives the data and decrypts the encrypted data. Analyzes the decrypted data and identifies the thought content, which is either "go forward," "go backward," "turn left," or "turn right." Then, generates an operation command using a generation AI. The generated command is re-encrypted and sent to the terminal.

[0133] Terminal: Decodes the command and sends it to the wheelchair control system.

[0134] Wheelchair: Follows the received operation command and performs the specified action (forward, backward, etc.).

[0135] Example prompt: "Get thoughts to move the wheelchair forward, then use generative AI to convert those thoughts into commands to control the wheelchair."

[0136] Example of concentration monitoring during a meeting

[0137] User: Business people attending a meeting.

[0138] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0139] Server: Receives data and decrypts the encrypted data. Analyzes the decrypted data and evaluates the user's concentration. If it falls below a certain threshold, the AI ​​generates a warning message. The generated message is re-encrypted and sent to the device.

[0140] On your device: Decrypt the warning message and save it to your storage device.

[0141] Storage device: Reads the saved warning message and notifies the user by displaying "Your concentration level is decreasing" on the display, for example.

[0142] Example prompt: "Collect EEG data to monitor concentration during meetings, and evaluate the concentration level based on the analysis results. Generate a warning message if the concentration level falls below the threshold."

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1: Collecting EEG data

[0145] User: The user wears a non-invasive EEG sensor on their head, which captures real-time EEG signals and transmits them wirelessly to a device.

[0146] Input: User's EEG signal

[0147] Output: Raw EEG data sent from the sensor to the device

[0148] Step 2: Preprocessing the EEG data

[0149] Device: The device performs noise reduction and filtering on the EEG signals received from the sensor. Specifically, it uses a bandpass filter to extract specific frequency components and an artifact removal algorithm to remove unnecessary components.

[0150] Input: Raw EEG data sent from the sensor

[0151] Output: Pre-processed, high-quality EEG data

[0152] Step 3: Sending data to the server

[0153] Terminal: The pre-processed EEG data is encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[0154] Input: Preprocessed EEG data

[0155] Output: Encrypted EEG data is sent to the server

[0156] Step 4: Decoding EEG data and extracting features

[0157] Server: The server receives the encrypted EEG data, decrypts it, and extracts features from the decrypted data using FFT (Fast Fourier Transform).

[0158] Input: Encrypted EEG data

[0159] Output: Features extracted by FFT

[0160] Step 5: Analyze the EEG data

[0161] Server: The extracted features are input into a machine learning model to analyze the user's thoughts and emotional state. The machine learning model uses SVM, neural networks, etc.

[0162] Input: Extracted features

[0163] Output: Typical analysis results (thoughts and emotional state)

[0164] Step 6: Realizing the output of generative AI

[0165] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. This generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[0166] Input: Analysis results

[0167] Output: The generated output, such as text, images, audio, or video.

[0168] Step 7: Encrypt data and send it to your device

[0169] Server: Formats the generated output data into the appropriate format, encrypts it again, and sends it to the device.

[0170] Input: Generated output data

[0171] Output: The encrypted output data.

[0172] Step 8: Decrypt and save the data

[0173] Terminal: Decrypts the received encrypted output data and stores it on a storage device.

[0174] Input: Encrypted output data

[0175] Output: The decoded output data

[0176] Step 9: Feedback on results

[0177] Storage device: Reads the saved output data and notifies or displays it to the user, for example, by displaying a message on the display or playing an audio message through the speaker.

[0178] Input: Decoded output data

[0179] Output: what is displayed or notified to the user

[0180] (Application example 1)

[0181] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0182] In conventional brick-and-mortar stores, users had to proactively inquire about products to obtain information and receive appropriate advice. Furthermore, there were few ways to grasp in real time which products users were interested in and what their purchasing intentions were, making it difficult to provide effective customer service. The present invention aims to provide a new solution for analyzing users' brainwave data and improving the shopping experience in brick-and-mortar stores.

[0183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0184] In this invention, the server includes: means for acquiring a user's electroencephalogram (EEG) signal using a non-invasive EEG sensor; means for preprocessing the EEG signal and performing noise reduction and filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing it using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for feeding back the output to the user in the storage device; means for identifying from the EEG data whether the user is interested in a product in a physical store or has a purchase intent; and means for providing the user with relevant product information and advice based on the identified interest and purchase intent. This makes it possible to provide optimal product information and advice in real time based on the user's interest and purchase intent, thereby improving the shopping experience.

[0185] A "non-invasive EEG sensor" is a sensor that can acquire EEG signals simply by wearing it, without physically invading the body.

[0186] "EEG signals" are electrical signals that reflect the user's brain activity.

[0187] "Preprocessing" refers to processing that removes noise and filters the EEG signals to improve the accuracy of analysis.

[0188] "Noise removal" is a process for removing unnecessary components contained in an electroencephalogram signal.

[0189] "Filtering" is a signal processing technique that extracts or suppresses only specific frequency bands.

[0190] A "secure protocol" is a communication protocol that ensures confidentiality and integrity in data communication.

[0191] "Features" are important information extracted from electroencephalogram data for analysis.

[0192] A "machine learning model" is an algorithm that learns patterns based on given data and makes predictions and classifications for future data.

[0193] "Generative AI" is a system that uses artificial intelligence technology to automatically generate output such as text, images, audio, and video.

[0194] A "storage device" is a device that stores data and provides feedback to the user as needed.

[0195] "Interest" refers to the attention or concern a user has towards a particular product.

[0196] "Purchase intent" refers to a user's desire or motivation to purchase a particular product.

[0197] "Product information" is information about the characteristics and features of a product.

[0198] "Advice" means any purchasing advice or recommendation provided to a User.

[0199] A system for implementing the present invention includes the following hardware and software.

[0200] Hardware

[0201] 1. EEG sensor:

[0202] Non-invasive EEG sensors: For example, by wearing specific headgear, the user's brainwave signals are acquired in real time. For example, high-performance EEG sensors can be used to accurately reflect the user's mental state and interests.

[0203] 2. Terminal:

[0204] Smart glasses: Devices equipped with displays and speakers that display information and provide audio feedback to users, allowing them to obtain the information they need in real time while shopping in physical stores.

[0205] This device has the processing power to receive data transmitted from the EEG sensor and perform initial pre-processing.

[0206] software

[0207] 1. Server:

[0208] Data processing and analysis: The server receives the data sent from the EEG sensor and the device, decrypts it, and extracts features from the preprocessed EEG data using algorithms such as FFT (Fast Fourier Transform).

[0209] Machine learning model: The extracted features are input into a machine learning model to analyze the user's brainwave data, which can determine whether the user is interested in a particular product and whether they have the intention to purchase it.

[0210] Generative AI: Based on the analysis results, generative AI generates output such as text, images, audio, and video. For example, it uses natural language processing models and image generation models (e.g., GPT-3 and DALL-E).

[0211] Example

[0212] 1. Data Preprocessing:

[0213] The user wears an EEG sensor, and the EEG signal is transmitted to a device, which then performs noise reduction and filtering to obtain high-quality data.

[0214] 2. Data transmission and analysis:

[0215] The preprocessed EEG data is sent to a server using a secure protocol (e.g., HTTPS). The server decrypts the data and extracts features using algorithms such as FFT. The user's EEG data is then analyzed by a machine learning model to identify the user's thoughts and emotional state.

[0216] 3. Generative AI output:

[0217] Based on the analysis results, generative AI generates output in the form of text, images, audio, video, etc. For example, if a user shows interest in a particular product, detailed information about that product and related product advice will be generated.

[0218] Specific examples from physical stores

[0219] Brainwave sensors: Users wearing high-performance, non-invasive brainwave sensors enter physical stores.

[0220] Smart glasses: When a user expresses interest in a particular product, the smart glasses display details about that product and offer advice on related products, and may also provide audio information through the speakers.

[0221] Example prompt for the generative AI:

[0222] "Users are interested in specific products. Based on this analysis, please provide product details and related products."

[0223] The system will significantly enhance the in-store shopping experience, allowing users to get real-time information and advice on products they are interested in or interested in purchasing.

[0224] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0225] Step 1:

[0226] The user wears a non-invasive EEG sensor. The sensor captures the user's EEG signals in real time and transmits them to a device. The input is the user's EEG, and the output is raw EEG data. The device receives this data.

[0227] Step 2:

[0228] The device performs preprocessing on the received EEG signals, specifically noise removal and filtering. The input is raw EEG data, and the output is clear, preprocessed EEG data. This improves the quality of the data and the accuracy of subsequent analysis.

[0229] Step 3:

[0230] The terminal encrypts the preprocessed EEG data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the preprocessed EEG data, and the output is the encrypted EEG data. The server receives this data.

[0231] Step 4:

[0232] The server decrypts the received data and extracts features using algorithms such as FFT (Fast Fourier Transform). The input is encrypted EEG data, and the output is features decomposed into specific frequency components. This makes it easier to analyze the data.

[0233] Step 5:

[0234] The extracted features are input into a machine learning model. The server analyzes this data using the machine learning model to identify the user's interests and purchasing intent. The input is the features, and the output is the identified user's interests and purchasing intent. This analysis reveals the user's state of mind.

[0235] Step 6:

[0236] The server uses a generative AI model based on the analysis results to generate output such as text, images, audio, and video. The input is the analysis results such as the user's interests and purchasing intentions, and the output is the generated content. An example of a prompt for the generative AI model is, "The user is interested in a specific product. Based on this analysis result, please provide product details and related products."

[0237] Step 7:

[0238] The generated output data is formatted into an appropriate format by the server, encrypted again, and then sent to the terminal. The input is the generated content, and the output is the encrypted content data.

[0239] Step 8:

[0240] The device decrypts the transmitted data and provides feedback to the user through the smart glasses' display and speaker. The input is encrypted content data, and the output is information and advice provided to the user. This allows users to obtain information and advice about products of interest in real time.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion engine, and a storage device that provides feedback on the output results to the user.

[0243] EEG data collection and preprocessing

[0244] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device performs pre-processing such as noise reduction and filtering on the received EEG signals to improve the signal quality.

[0245] Sending data to the server

[0246] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[0247] Analysis of EEG data

[0248] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) and wavelet transform to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's EEG data.

[0249] Emotion recognition by emotion engine

[0250] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses trained models to identify the user's emotions from brainwave patterns and physiological data.

[0251] Realizing output through generative AI

[0252] Server: Based on the emotional state recognized by the emotion engine, the generative AI generates appropriate output such as text, images, audio, and video. The generative AI uses advanced models such as GPT-3 and DALL-E to create highly accurate output.

[0253] Feedback of results

[0254] Server: The generated output data is formatted appropriately, encrypted again, and sent to the device. The device decrypts the data and stores it in a storage device. The storage device notifies or displays the stored result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[0255] Specific examples

[0256] Example of assistive device operation for physically paralyzed people

[0257] User: A paralyzed person thinks about operating a wheelchair.

[0258] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0259] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." At the same time, the emotion engine recognizes the user's emotional state. Then, it uses generative AI to generate operational commands and feedback based on the user's emotional state.

[0260] Storage device: Sends the generated command to the wheelchair control system, which then executes the specified action. At the same time, the wheelchair provides feedback based on the action results and the user's emotions.

[0261] Example of concentration monitoring during a meeting

[0262] Users: Business people attending the meeting.

[0263] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0264] Server: Receives and analyzes the data. Based on the analysis results, the user's level of concentration is evaluated. If it falls below a certain threshold, the emotion engine recognizes the user's emotion, and the generative AI generates optimal feedback.

[0265] Storage device: Generated alerts based on the concentration level and emotions are sent to the device, which then notifies the user. For example, the device may display "Your concentration is declining" on the display, along with advice on how to reduce stress.

[0266] In this way, the system of the present invention analyzes the user's EEG data and generates appropriate feedback based on the emotional state recognized by the emotion engine, thereby enriching the user's experience and supporting efficient communication and work performance.

[0267] The processing flow will be explained below.

[0268] Step 1:

[0269] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[0270] Step 2:

[0271] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[0272] Step 3:

[0273] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[0274] Step 4:

[0275] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[0276] Step 5:

[0277] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[0278] Step 6:

[0279] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[0280] Step 7:

[0281] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[0282] Step 8:

[0283] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses algorithms to identify emotions from brainwave patterns and other physiological data.

[0284] Step 9:

[0285] Server: Based on the emotional state recognized by the emotion engine, generative AI is used to generate appropriate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[0286] Step 10:

[0287] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[0288] Step 11:

[0289] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[0290] Step 12:

[0291] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[0292] Example: Operating assistive devices for physically paralyzed people

[0293] Step 1:

[0294] User: A paralyzed person wears an EEG sensor on their head.

[0295] Step 2:

[0296] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0297] Step 3:

[0298] Terminal: Encrypts the pre-processed data and sends it to the server.

[0299] Step 4:

[0300] Server: Receives the data and extracts features after decoding.

[0301] Step 5:

[0302] Server: A machine learning model analyzes the user's thoughts and generates operational commands such as "go forward" or "go backward." At the same time, an emotion engine recognizes the user's emotional state.

[0303] Step 6:

[0304] Server: The generation AI generates feedback based on the control command and emotional state.

[0305] Step 7:

[0306] Server: Encrypts the generated commands and sends them to the device using a secure protocol.

[0307] Step 8:

[0308] Terminal: Decodes the received commands and sends them to the wheelchair control system.

[0309] Step 9:

[0310] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[0311] Example: Monitoring concentration during a meeting

[0312] Step 1:

[0313] User: Business people participating in a meeting wear brainwave sensors.

[0314] Step 2:

[0315] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0316] Step 3:

[0317] Terminal: Encrypts the pre-processed data and sends it to the server.

[0318] Step 4:

[0319] Server: Receives the data and extracts features after decoding.

[0320] Step 5:

[0321] Server: Analyzes and evaluates the user's concentration level using a machine learning model, while simultaneously recognizing their emotional state using an emotion engine.

[0322] Step 6:

[0323] Server: When the threshold is exceeded, the AI ​​generates an appropriate warning message and alert based on the emotion.

[0324] Step 7:

[0325] Server: Encrypts the generated message and sends it to the device using a secure protocol.

[0326] Step 8:

[0327] Device: Decodes the received message and notifies the user, for example by displaying "Your concentration is declining" on the display and providing emotion-based stress reduction advice.

[0328] Example 2

[0329] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Existing EEG analysis systems lack a unified process from acquiring EEG data to recognizing emotions and providing feedback to the user, making it difficult to accurately and quickly reflect the user's emotional state. Furthermore, for specific applications such as assisting physically paralyzed people or monitoring concentration levels during meetings, there was a lack of a system that provided appropriate feedback in real time according to user needs, making it difficult to improve user experience and work efficiency.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0332] In this invention, the server includes means for extracting features from the EEG data and analyzing them using a machine learning model, means for recognizing the user's emotional state using an emotion recognition engine based on the analysis results, and means for generating output such as text, images, audio, and video using a generative AI model based on the recognized emotional state. This enables emotion recognition and the generation of appropriate feedback to be performed quickly and accurately based on the user's EEG data, making it possible to improve user experience and business efficiency in specific applications.

[0333] A "non-invasive EEG sensor" is a sensor worn on the head to acquire the user's brainwave signals, but collects data without invading the body.

[0334] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[0335] A "secure protocol" is a communication protocol that uses encryption technology to securely transmit data to another party in order to ensure confidentiality and integrity of the data.

[0336] "Features" are important data points or indicators extracted from EEG data and used for analysis.

[0337] A "machine learning model" is an algorithm or method used to analyze EEG data, primarily for pattern recognition, classification, and regression of data.

[0338] An "emotion recognition engine" is a model trained to identify and recognize a user's emotional state by analyzing EEG data and other physiological data.

[0339] A "generative AI model" is an AI model that generates output such as text, images, audio, or video based on a recognized emotional state. Representative examples include GPT-3 and DALL-E.

[0340] "Storage Device" means a device used to provide generated output back to the user, for storing and displaying data.

[0341] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion recognition engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion recognition engine, and a storage device that provides feedback on the output results to the user.

[0342] Hardware and software used

[0343] User: The user wears a non-invasive EEG sensor on their head, which converts electrical activity in the brain into digital signals and transmits them to a device in real time.

[0344] Terminal: Receives EEG signals sent from the EEG sensor. The terminal is equipped with software to remove noise and filter the received signals, and to perform preprocessing. This mainly involves band-pass filtering and detrending.

[0345] Server: The server receives the preprocessed EEG data via an encrypted protocol (e.g., HTTPS). The server extracts features using FFT (Fast Fourier Transform) or wavelet transform, and analyzes the data using a machine learning model. Machine learning libraries such as Python and TensorFlow are used here.

[0346] Emotion Recognition Engine: Recognizes the user's emotions based on the analysis of EEG data. The emotion recognition engine uses algorithms such as neural networks and support vector machines to identify emotions based on pre-trained models and analyzing EEG patterns and physiological data.

[0347] Generative AI model: Based on the emotional state recognized by the emotion recognition engine, the generative AI generates output such as text, images, audio, and video. For example, advanced generative AI models such as GPT-3 and DALL-E are used, which generate highly accurate output.

[0348] Storage device: Used to feed back the generated output data to the user. The data is encrypted again and sent to the terminal, where it is decrypted and stored in the storage device. The generated text or voice message is notified to the user through devices such as a display or speaker.

[0349] Specific examples

[0350] Example of assistive device operation for physically paralyzed people

[0351] User: A paralyzed person thinks about operating a wheelchair, for example, "Go forward."

[0352] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0353] Server: Receives and analyzes the data. From the analysis results, it determines that a "move forward" command has been issued. At the same time, it uses an emotion recognition engine to recognize the user's emotional state. It then uses a generative AI to generate feedback based on the operation command and emotion. A message such as "The wheelchair will move forward. Please relax" is generated.

[0354] Storage device: Sends the generated commands to the wheelchair control system, causing the wheelchair to perform the specified actions, while simultaneously displaying feedback messages to the user.

[0355] Example of concentration monitoring during a meeting

[0356] Users: Business people attending the meeting.

[0357] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0358] Server: Receives and analyzes the data. The level of concentration is evaluated based on the analysis results, and if it falls below a certain threshold, the emotion recognition engine recognizes the user's emotions. An alert message such as "Your concentration is declining. Take a deep breath and refresh yourself" is generated.

[0359] Storage device: Sends the generated alert message to the device, which notifies the user.

[0360] Prompt Sentence Examples

[0361] "Recommend music that users listen to when they are in a relaxed state."

[0362] "Generate an alert message when someone loses focus during a meeting."

[0363] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0364] Step 1:

[0365] User: Wears a non-invasive EEG sensor on the head, which measures the user's brainwave signals in real time and transmits the digital signal to the device.

[0366] Input: EEG signal

[0367] Output: Digital signal

[0368] Step 2:

[0369] Terminal: Pre-processing the received digital signal, specifically applying a band-pass filter to remove high frequency noise, and detrending to remove baseline drift.

[0370] Input: Digital signal

[0371] Output: Preprocessed EEG data

[0372] Step 3:

[0373] Terminal: Encrypt the pre-processed EEG data. Send the encrypted data to the server using a secure protocol (e.g., HTTPS).

[0374] Input: Preprocessed EEG data

[0375] Output: Encrypted EEG data

[0376] Step 4:

[0377] Server: Receives and decrypts the encrypted data sent from the device. Then, it extracts frequency components using FFT (Fast Fourier Transform) and obtains features with high time-frequency resolution using wavelet transform.

[0378] Input: Encrypted EEG data

[0379] Output: Feature data

[0380] Step 5:

[0381] Server: The feature data is input into a machine learning model for analysis. Here, analysis is performed using support vector machines and neural networks, and the user's EEG data is analyzed.

[0382] Input: Feature data

[0383] Output: Analysis results of EEG data

[0384] Step 6:

[0385] Server: The analysis results are input into the emotion recognition engine to recognize the user's emotional state. The emotion recognition engine analyzes the user's brainwave patterns based on a pre-trained dataset to identify emotions.

[0386] Input: Analysis results

[0387] Output: User's emotional state

[0388] Step 7:

[0389] Server: Based on the recognized emotional state, a generative AI model (e.g., GPT-3 or DALL-E) is used to generate output data (text, images, audio, video, etc.). For example, a prompt such as "Please recommend music that the user would listen to when in a relaxed state" is input into the generative AI, and output data is generated.

[0390] Input: User's emotional state

[0391] Output: The generated output data

[0392] Step 8:

[0393] Server: The generated output data is re-encrypted and sent to the terminal.

[0394] Input: Generated output data

[0395] Output: The encrypted output data.

[0396] Step 9:

[0397] Terminal: Decrypts the transmitted encrypted data, stores it on a storage device, and notifies the user of generated text and voice messages using devices such as a display and speaker.

[0398] Input: Encrypted output data

[0399] Output: The decoded output data

[0400] These are the specific processing steps of the program. Each step forms a series of steps based on the user's EEG data: data acquisition, preprocessing, analysis, generation, and feedback.

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Conventional non-invasive EEG sensor systems have the technology to acquire users' EEG signals and provide feedback based on the analysis results, but they have not yet fully developed mechanisms for evaluating fatigue and concentration levels in real time and providing appropriate feedback based on that evaluation. Furthermore, they lack the functionality to link with visual devices worn by workers and manage the data in real time under specific work environments. This has made it difficult to efficiently manage worker health and safety and improve labor productivity.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring a user's EEG signal using a non-invasive EEG sensor; means for preprocessing the EEG signal to remove noise and perform filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing them using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for providing feedback of the output to the user in the storage device; and means for evaluating a worker's fatigue level and concentration level in real time in cooperation with a visual device worn by the worker and providing appropriate feedback. This makes it possible to evaluate a worker's fatigue level and concentration level in real time, thereby improving work efficiency while managing their health and safety.

[0405] A "non-invasive brainwave sensor" is a device that acquires a user's brainwave signals without invading the skin.

[0406] "User" refers to a person who operates or uses the system.

[0407] "Electroencephalogram signals" refer to changes in electrical potential generated by electrical activity in the brain.

[0408] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[0409] "Noise removal" is a process for removing unnecessary noise contained in the electroencephalogram signal.

[0410] "Filtering" is the process of conditioning a signal by passing only specific frequency bands.

[0411] A "secure protocol" is a secure communication protocol that ensures confidentiality and integrity of communication data.

[0412] A "server" is a central processing unit for analyzing and processing data.

[0413] "Features" refer to important data characteristics extracted from EEG signals.

[0414] A "machine learning model" is an algorithm that learns patterns based on data and makes predictions and classifications.

[0415] "Analysis" is a process for obtaining detailed information based on the acquired data.

[0416] "Generative AI" refers to the ability to generate output such as text, images, audio, and video using artificial intelligence technology.

[0417] "Storage device" refers to a device for storing data.

[0418] "Feedback" refers to returning analysis results or generated output to the user.

[0419] "Visual device" refers to a device worn by a user that provides information visually.

[0420] "Fatigue level" is an index for evaluating the degree of fatigue felt by the user.

[0421] "Level of concentration" is an index for evaluating the level of concentration of a user.

[0422] The present invention provides a system for acquiring a user's electroencephalogram (EEG) signal using a non-invasive electroencephalogram sensor, analyzing the signal, and providing appropriate feedback in real time. Specific embodiments of the present invention will be described in detail below.

[0423] System program description

[0424] EEG data collection and preprocessing

[0425] The user wears a non-invasive EEG sensor on their head, which captures EEG signals in real time and performs noise reduction and filtering. After pre-processing, the data is encrypted and sent to a server via a secure protocol. This process ensures data quality and confidentiality.

[0426] Sending data to the server and analyzing it

[0427] The server decodes the received preprocessed EEG data and uses algorithms such as fast Fourier transform (FFT) and wavelet transform to extract features, which are then input into a machine learning model to analyze the EEG data.

[0428] Emotion recognition by emotion engine

[0429] Based on the analysis results, the server recognizes the user's emotional state using an emotion engine, which uses a trained model to identify the user's emotional state from numerous brainwave patterns and physiological data.

[0430] Feedback generation by generative AI

[0431] Based on the emotional state recognized by the emotion engine, the server uses generative AI to create appropriate feedback, using, for example, advanced text and image generation models to generate highly accurate output.

[0432] Feedback of results

[0433] The generated feedback data is then encrypted again and sent via a secure protocol to the device, which decrypts the data and presents it to the user via a visual device (such as smart glasses).

[0434] Hardware and software used

[0435] Brainwave sensors: Non-invasive devices such as Emotiv and Muse.

[0436] Server: A central processing unit for running data analysis and generative AI models, using Python and libraries such as SciPy and Sklearn.

[0437] Visual devices: smart glasses (e.g., Google Glass, Microsoft HoloLens).

[0438] Communication protocol: Encrypted communication (e.g. HTTPS).

[0439] Specific examples

[0440] Application example: Factory worker monitoring

[0441] Factory workers wear smart glasses, and their brainwaves are monitored in real time using an EEG sensor. If a worker continues working for an extended period of time, data is collected to assess their level of fatigue and concentration. Based on the analysis results, the server uses a generative AI to provide appropriate feedback. For example, if the worker's level of fatigue is high, the smart glasses will display a message saying, "We recommend taking a break."

[0442] Example prompt sentence:

[0443] "Workers working long hours on an assembly line were monitored for brainwaves to assess their fatigue levels. Appropriate feedback was then generated for the workers."

[0444] As such, the present invention is a system that analyzes the user's brainwave data in real time and provides appropriate feedback using an emotion engine and generative AI, thereby improving work efficiency while managing the health and safety of workers.

[0445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0446] Step 1:

[0447] The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time. The captured EEG signals are sent to a terminal for noise reduction and filtering. The input is the raw EEG signal, and the output is pre-processed clean EEG data.

[0448] Step 2:

[0449] The device performs noise reduction and bandpass filtering on the received EEG signal. Specifically, it uses a Butterworth filter to pass only the frequency band between 0.5 Hz and 30 Hz. This processing improves the signal quality and removes unnecessary noise. The input is the raw EEG signal, and the output is preprocessed EEG data.

[0450] Step 3:

[0451] The preprocessed EEG data is encrypted and sent from the device to the server using the HTTPS protocol. The server decrypts the received data and prepares it for analysis. The input is the preprocessed EEG data, and the output is the decrypted EEG data.

[0452] Step 4:

[0453] The server extracts features from the decoded EEG data. Specifically, it uses FFT (Fast Fourier Transform) and wavelet transform to separate different frequency components of the EEG signal. The extracted features are input into a machine learning model for further analysis. The input is the decoded EEG data, and the output is the extracted features.

[0454] Step 5:

[0455] The server inputs the extracted features into a machine learning model and analyzes the EEG data. The analysis results are processed by an emotion engine to identify the appropriate emotional state based on information obtained from the user's EEG patterns. The input is feature data, and the output is an evaluation of the emotional state.

[0456] Step 6:

[0457] The server uses generative AI to generate optimal feedback based on the emotional state recognized by the emotion engine. Specifically, it uses GPT-3 or a similar generative AI model to generate appropriate output such as text, images, audio, and video. The input is the evaluation result of the emotional state, and the output is the generated feedback.

[0458] Step 7:

[0459] The generated feedback data is encrypted and sent back to the terminal from the server. The terminal decrypts the data and stores it in a storage device. The input is the generated feedback data, and the output is the decrypted feedback data.

[0460] Step 8:

[0461] The device then notifies the user of the decoded feedback via a visual device such as smart glasses. For example, if the fatigue level is high, a message such as "We recommend you take a break" is displayed. The input is the decoded feedback data, and the output is a notification message displayed to the user.

[0462] This makes it possible to analyze the user's brain wave data in real time and provide appropriate feedback.

[0463] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0464] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0465] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0466] [Second embodiment]

[0467] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0468] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0470] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0472] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0474] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0475] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0476] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0477] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0478] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0479] This invention is a system that analyzes EEG signals acquired by an EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[0480] EEG data collection and preprocessing

[0481] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device then performs pre-processing on the received EEG signals, such as noise reduction and filtering, to improve signal quality and increase analysis accuracy.

[0482] Sending data to the server

[0483] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[0484] Analysis of EEG data

[0485] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's brainwave data. The analysis results in the user's thoughts and emotional state.

[0486] Realizing output through generative AI

[0487] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. The generative AI uses various generative models (e.g., natural language processing models and image generation models) to create highly accurate output.

[0488] Feedback of results

[0489] Server: The generated output data is formatted into an appropriate format, encrypted again, and sent to the terminal. The terminal decrypts the sent data and saves it to a storage device. The storage device notifies or displays the saved result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[0490] Specific examples

[0491] Example of assistive device operation for physically paralyzed people

[0492] User: A paralyzed person thinks about operating a wheelchair.

[0493] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0494] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." It then uses a generation AI to generate an operation command.

[0495] Storage device: Sends the generated instructions to the wheelchair control system, causing the wheelchair to perform the specified actions.

[0496] Example of concentration monitoring during a meeting

[0497] Users: Business people attending the meeting.

[0498] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0499] Server: Receives and analyzes the data. Based on the analysis results, the user's concentration level is evaluated. If it falls below a certain threshold, the generation AI generates a warning message.

[0500] Storage device: The generated warning message is sent to the device, which notifies the user, for example, by displaying "Your concentration level is decreasing" on the display.

[0501] The processing flow will be explained below.

[0502] Step 1:

[0503] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[0504] Step 2:

[0505] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[0506] Step 3:

[0507] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[0508] Step 4:

[0509] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[0510] Step 5:

[0511] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[0512] Step 6:

[0513] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[0514] Step 7:

[0515] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[0516] Step 8:

[0517] Server: Based on the analysis results, generative AI is used to generate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[0518] Step 9:

[0519] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[0520] Step 10:

[0521] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[0522] Step 11:

[0523] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[0524] As a concrete example, the case of a physically paralyzed person operating a wheelchair is shown below.

[0525] Step 1:

[0526] User: A paralyzed person wears a brainwave sensor on their head to help them think about operating a wheelchair.

[0527] Step 2:

[0528] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0529] Step 3:

[0530] Terminal: Encrypts the preprocessed data and sends it to the server.

[0531] Step 4:

[0532] Server: Receives the data and extracts features after decoding.

[0533] Step 5:

[0534] Server: A machine learning model analyzes the user's thoughts, and the generation AI generates operation commands (e.g., "go forward," "go backward," "turn left," "turn right").

[0535] Step 6:

[0536] Server: Encrypts and transmits commands to the terminal.

[0537] Step 7:

[0538] Terminal: Decodes commands and sends instructions to the wheelchair control system.

[0539] Step 8:

[0540] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[0541] Through these specific process steps, the invention can be effectively implemented.

[0542] Example 1

[0543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] The present invention aims to provide a system that acquires and analyzes a user's EEG signals in real time to accurately grasp the user's thoughts and emotions and provide appropriate feedback based on the acquired signals. Current systems face challenges in removing noise from EEG signals obtained from non-invasive EEG sensors, securely transmitting data, improving analytical accuracy, and providing feedback in a variety of output formats using generative AI. These challenges must be addressed, for example, when a physically paralyzed person needs to operate an assistive device by thought, or when monitoring a user's concentration level and generating appropriate warnings.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0546] In this invention, the server includes means for decoding the EEG data and performing a fast Fourier transform to extract features, means for inputting the features into a machine learning model and analyzing them, and means for generating output such as text, images, audio, and video using a generative AI based on the analysis results. This enables highly accurate analysis based on the user's EEG signals and the generation of appropriate output.

[0547] A "non-invasive EEG sensor" is a sensor that is worn on the user's head to acquire EEG signals in real time, and is a device that can be used without damaging the user's skin.

[0548] "Preprocessing" refers to processing that removes noise and filters the EEG signal to improve the quality of the signal.

[0549] "Noise reduction" is the process of removing unnecessary noise and external interference from EEG signals.

[0550] "Filtering" is a process of extracting specific frequency components from an EEG signal or removing unnecessary frequency components.

[0551] A "secure protocol" is a communication method that ensures confidentiality and integrity of data and is a protocol that has encryption and authentication functions.

[0552] "Encryption" is a process of converting data to prevent the contents of the data being transmitted from being read by a third party.

[0553] "Decryption" is the process of restoring encrypted data to its original state.

[0554] "Feature extraction" is the process of finding the characteristics and patterns necessary for analysis from EEG signals.

[0555] The "Fast Fourier Transform (FFT)" is a mathematical technique for converting time-domain signals into the frequency domain for analysis.

[0556] A "machine learning model" is a model that learns rules and patterns from data and makes predictions and classifications.

[0557] "Generative AI" is a general term for artificial intelligence technology that generates output such as text, images, audio, and video based on analysis results.

[0558] "Text, image, audio, video, etc. output" refers to the form of information generated based on the user's thoughts and feelings.

[0559] "Formatting" is the process of putting data into a specific form.

[0560] "Storage Device" means a device that stores the generated output and communicates or displays it to the user when necessary.

[0561] This invention is a system that analyzes EEG signals acquired using a non-invasive EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[0562] Hardware and software used

[0563] Brainwave sensor: A non-invasive brainwave sensor is worn on the user's head to capture brainwave signals in real time. This sensor has wireless communication capabilities and transmits the brainwave signals to a device.

[0564] Terminal: The terminal processes the EEG signals received from the sensor. Pre-processing such as noise reduction and filtering is performed on the terminal. Specifically, band-pass filters and artifact removal algorithms are used to make the signals clearer.

[0565] Server: The server receives and decrypts the encrypted EEG data sent from the device. It then extracts features using an FFT (Fast Fourier Transform). The extracted features are input into a machine learning model to analyze the user's EEG data. Based on the analysis results, a generative AI generates output such as text, images, audio, and video. The generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[0566] Storage Device: A storage device is a device that stores the generated output data and provides feedback to the user, for example, by displaying a message on a display or playing an audio message through a speaker.

[0567] Specific examples

[0568] Example of assistive device operation for physically paralyzed people

[0569] User: A paralyzed person thinks about operating a wheelchair.

[0570] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0571] Server: Receives the data and decrypts the encrypted data. Analyzes the decrypted data and identifies the thought content, which is either "go forward," "go backward," "turn left," or "turn right." Then, generates an operation command using a generation AI. The generated command is re-encrypted and sent to the terminal.

[0572] Terminal: Decodes the command and sends it to the wheelchair control system.

[0573] Wheelchair: Follows the received operation command and performs the specified action (forward, backward, etc.).

[0574] Example prompt: "Get thoughts to move the wheelchair forward, then use generative AI to convert those thoughts into commands to control the wheelchair."

[0575] Example of concentration monitoring during a meeting

[0576] User: Business people attending a meeting.

[0577] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0578] Server: Receives data and decrypts the encrypted data. Analyzes the decrypted data and evaluates the user's concentration. If it falls below a certain threshold, the AI ​​generates a warning message. The generated message is re-encrypted and sent to the device.

[0579] On your device: Decrypt the warning message and save it to your storage device.

[0580] Storage device: Reads the saved warning message and notifies the user by displaying "Your concentration level is decreasing" on the display, for example.

[0581] Example prompt: "Collect EEG data to monitor concentration during meetings, and evaluate the concentration level based on the analysis results. Generate a warning message if the concentration level falls below the threshold."

[0582] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0583] Step 1: Collecting EEG data

[0584] User: The user wears a non-invasive EEG sensor on their head, which captures real-time EEG signals and transmits them wirelessly to a device.

[0585] Input: User's EEG signal

[0586] Output: Raw EEG data sent from the sensor to the device

[0587] Step 2: Preprocessing the EEG data

[0588] Device: The device performs noise reduction and filtering on the EEG signals received from the sensor. Specifically, it uses a bandpass filter to extract specific frequency components and an artifact removal algorithm to remove unnecessary components.

[0589] Input: Raw EEG data sent from the sensor

[0590] Output: Pre-processed, high-quality EEG data

[0591] Step 3: Sending data to the server

[0592] Terminal: The pre-processed EEG data is encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[0593] Input: Preprocessed EEG data

[0594] Output: Encrypted EEG data is sent to the server

[0595] Step 4: Decoding EEG data and extracting features

[0596] Server: The server receives the encrypted EEG data, decrypts it, and extracts features from the decrypted data using FFT (Fast Fourier Transform).

[0597] Input: Encrypted EEG data

[0598] Output: Features extracted by FFT

[0599] Step 5: Analyze the EEG data

[0600] Server: The extracted features are input into a machine learning model to analyze the user's thoughts and emotional state. The machine learning model uses SVM, neural networks, etc.

[0601] Input: Extracted features

[0602] Output: Typical analysis results (thoughts and emotional state)

[0603] Step 6: Realizing the output of generative AI

[0604] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. This generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[0605] Input: Analysis results

[0606] Output: The generated output, such as text, images, audio, or video.

[0607] Step 7: Encrypt data and send it to your device

[0608] Server: Formats the generated output data into the appropriate format, encrypts it again, and sends it to the device.

[0609] Input: Generated output data

[0610] Output: The encrypted output data.

[0611] Step 8: Decrypt and save the data

[0612] Terminal: Decrypts the received encrypted output data and stores it on a storage device.

[0613] Input: Encrypted output data

[0614] Output: The decoded output data

[0615] Step 9: Feedback on results

[0616] Storage device: Reads the saved output data and notifies or displays it to the user, for example, by displaying a message on the display or playing an audio message through the speaker.

[0617] Input: Decoded output data

[0618] Output: what is displayed or notified to the user

[0619] (Application example 1)

[0620] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0621] In conventional brick-and-mortar stores, users had to proactively inquire about products to obtain information and receive appropriate advice. Furthermore, there were few ways to grasp in real time which products users were interested in and what their purchasing intentions were, making it difficult to provide effective customer service. The present invention aims to provide a new solution for analyzing users' brainwave data and improving the shopping experience in brick-and-mortar stores.

[0622] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0623] In this invention, the server includes: means for acquiring a user's electroencephalogram (EEG) signal using a non-invasive EEG sensor; means for preprocessing the EEG signal and performing noise reduction and filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing it using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for feeding back the output to the user in the storage device; means for identifying from the EEG data whether the user is interested in a product in a physical store or has a purchase intent; and means for providing the user with relevant product information and advice based on the identified interest and purchase intent. This makes it possible to provide optimal product information and advice in real time based on the user's interest and purchase intent, thereby improving the shopping experience.

[0624] A "non-invasive EEG sensor" is a sensor that can acquire EEG signals simply by wearing it, without physically invading the body.

[0625] "EEG signals" are electrical signals that reflect the user's brain activity.

[0626] "Preprocessing" refers to processing that removes noise and filters the EEG signals to improve the accuracy of analysis.

[0627] "Noise removal" is a process for removing unnecessary components contained in an electroencephalogram signal.

[0628] "Filtering" is a signal processing technique that extracts or suppresses only specific frequency bands.

[0629] A "secure protocol" is a communication protocol that ensures confidentiality and integrity in data communication.

[0630] "Features" are important information extracted from electroencephalogram data for analysis.

[0631] A "machine learning model" is an algorithm that learns patterns based on given data and makes predictions and classifications for future data.

[0632] "Generative AI" is a system that uses artificial intelligence technology to automatically generate output such as text, images, audio, and video.

[0633] A "storage device" is a device that stores data and provides feedback to the user as needed.

[0634] "Interest" refers to the attention or concern a user has towards a particular product.

[0635] "Purchase intent" refers to a user's desire or motivation to purchase a particular product.

[0636] "Product information" is information about the characteristics and features of a product.

[0637] "Advice" means any purchasing advice or recommendation provided to a User.

[0638] A system for implementing the present invention includes the following hardware and software.

[0639] Hardware

[0640] 1. EEG sensor:

[0641] Non-invasive EEG sensors: For example, by wearing specific headgear, the user's brainwave signals are acquired in real time. For example, high-performance EEG sensors can be used to accurately reflect the user's mental state and interests.

[0642] 2. Terminal:

[0643] Smart glasses: Devices equipped with displays and speakers that display information and provide audio feedback to users, allowing them to obtain the information they need in real time while shopping in physical stores.

[0644] This device has the processing power to receive data transmitted from the EEG sensor and perform initial pre-processing.

[0645] software

[0646] 1. Server:

[0647] Data processing and analysis: The server receives the data sent from the EEG sensor and the device, decrypts it, and extracts features from the preprocessed EEG data using algorithms such as FFT (Fast Fourier Transform).

[0648] Machine learning model: The extracted features are input into a machine learning model to analyze the user's brainwave data, which can determine whether the user is interested in a particular product and whether they have the intention to purchase it.

[0649] Generative AI: Based on the analysis results, generative AI generates output such as text, images, audio, and video. For example, it uses natural language processing models and image generation models (e.g., GPT-3 and DALL-E).

[0650] Example

[0651] 1. Data Preprocessing:

[0652] The user wears an EEG sensor, and the EEG signal is transmitted to a device, which then performs noise reduction and filtering to obtain high-quality data.

[0653] 2. Data transmission and analysis:

[0654] The preprocessed EEG data is sent to a server using a secure protocol (e.g., HTTPS). The server decrypts the data and extracts features using algorithms such as FFT. The user's EEG data is then analyzed by a machine learning model to identify the user's thoughts and emotional state.

[0655] 3. Generative AI output:

[0656] Based on the analysis results, generative AI generates output in the form of text, images, audio, video, etc. For example, if a user shows interest in a particular product, detailed information about that product and related product advice will be generated.

[0657] Specific examples from physical stores

[0658] Brainwave sensors: Users wearing high-performance, non-invasive brainwave sensors enter physical stores.

[0659] Smart glasses: When a user expresses interest in a particular product, the smart glasses display details about that product and offer advice on related products, and may also provide audio information through the speakers.

[0660] Example prompt for the generative AI:

[0661] "Users are interested in specific products. Based on this analysis, please provide product details and related products."

[0662] The system will significantly enhance the in-store shopping experience, allowing users to get real-time information and advice on products they are interested in or interested in purchasing.

[0663] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0664] Step 1:

[0665] The user wears a non-invasive EEG sensor. The sensor captures the user's EEG signals in real time and transmits them to a device. The input is the user's EEG, and the output is raw EEG data. The device receives this data.

[0666] Step 2:

[0667] The device performs preprocessing on the received EEG signals, specifically noise removal and filtering. The input is raw EEG data, and the output is clear, preprocessed EEG data. This improves the quality of the data and the accuracy of subsequent analysis.

[0668] Step 3:

[0669] The terminal encrypts the preprocessed EEG data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the preprocessed EEG data, and the output is the encrypted EEG data. The server receives this data.

[0670] Step 4:

[0671] The server decrypts the received data and extracts features using algorithms such as FFT (Fast Fourier Transform). The input is encrypted EEG data, and the output is features decomposed into specific frequency components. This makes it easier to analyze the data.

[0672] Step 5:

[0673] The extracted features are input into a machine learning model. The server analyzes this data using the machine learning model to identify the user's interests and purchasing intent. The input is the features, and the output is the identified user's interests and purchasing intent. This analysis reveals the user's state of mind.

[0674] Step 6:

[0675] The server uses a generative AI model based on the analysis results to generate output such as text, images, audio, and video. The input is the analysis results such as the user's interests and purchasing intentions, and the output is the generated content. An example of a prompt for the generative AI model is, "The user is interested in a specific product. Based on this analysis result, please provide product details and related products."

[0676] Step 7:

[0677] The generated output data is formatted into an appropriate format by the server, encrypted again, and then sent to the terminal. The input is the generated content, and the output is the encrypted content data.

[0678] Step 8:

[0679] The device decrypts the transmitted data and provides feedback to the user through the smart glasses' display and speaker. The input is encrypted content data, and the output is information and advice provided to the user. This allows users to obtain information and advice about products of interest in real time.

[0680] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0681] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion engine, and a storage device that provides feedback on the output results to the user.

[0682] EEG data collection and preprocessing

[0683] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device performs pre-processing such as noise reduction and filtering on the received EEG signals to improve the signal quality.

[0684] Sending data to the server

[0685] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[0686] Analysis of EEG data

[0687] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) and wavelet transform to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's EEG data.

[0688] Emotion recognition by emotion engine

[0689] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses trained models to identify the user's emotions from brainwave patterns and physiological data.

[0690] Realizing output through generative AI

[0691] Server: Based on the emotional state recognized by the emotion engine, the generative AI generates appropriate output such as text, images, audio, and video. The generative AI uses advanced models such as GPT-3 and DALL-E to create highly accurate output.

[0692] Feedback of results

[0693] Server: The generated output data is formatted appropriately, encrypted again, and sent to the device. The device decrypts the data and stores it in a storage device. The storage device notifies or displays the stored result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[0694] Specific examples

[0695] Example of assistive device operation for physically paralyzed people

[0696] User: A paralyzed person thinks about operating a wheelchair.

[0697] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0698] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." At the same time, the emotion engine recognizes the user's emotional state. Then, it uses generative AI to generate operational commands and feedback based on the user's emotional state.

[0699] Storage device: Sends the generated command to the wheelchair control system, which then executes the specified action. At the same time, the wheelchair provides feedback based on the action results and the user's emotions.

[0700] Example of concentration monitoring during a meeting

[0701] Users: Business people attending the meeting.

[0702] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0703] Server: Receives and analyzes the data. Based on the analysis results, the user's level of concentration is evaluated. If it falls below a certain threshold, the emotion engine recognizes the user's emotion, and the generative AI generates optimal feedback.

[0704] Storage device: Generated alerts based on the concentration level and emotions are sent to the device, which then notifies the user. For example, the device may display "Your concentration is declining" on the display, along with advice on how to reduce stress.

[0705] In this way, the system of the present invention analyzes the user's EEG data and generates appropriate feedback based on the emotional state recognized by the emotion engine, thereby enriching the user's experience and supporting efficient communication and work performance.

[0706] The processing flow will be explained below.

[0707] Step 1:

[0708] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[0709] Step 2:

[0710] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[0711] Step 3:

[0712] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[0713] Step 4:

[0714] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[0715] Step 5:

[0716] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[0717] Step 6:

[0718] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[0719] Step 7:

[0720] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[0721] Step 8:

[0722] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses algorithms to identify emotions from brainwave patterns and other physiological data.

[0723] Step 9:

[0724] Server: Based on the emotional state recognized by the emotion engine, generative AI is used to generate appropriate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[0725] Step 10:

[0726] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[0727] Step 11:

[0728] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[0729] Step 12:

[0730] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[0731] Example: Operating assistive devices for physically paralyzed people

[0732] Step 1:

[0733] User: A paralyzed person wears an EEG sensor on their head.

[0734] Step 2:

[0735] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0736] Step 3:

[0737] Terminal: Encrypts the pre-processed data and sends it to the server.

[0738] Step 4:

[0739] Server: Receives the data and extracts features after decoding.

[0740] Step 5:

[0741] Server: A machine learning model analyzes the user's thoughts and generates operational commands such as "go forward" or "go backward." At the same time, an emotion engine recognizes the user's emotional state.

[0742] Step 6:

[0743] Server: The generation AI generates feedback based on the control command and emotional state.

[0744] Step 7:

[0745] Server: Encrypts the generated commands and sends them to the device using a secure protocol.

[0746] Step 8:

[0747] Terminal: Decodes the received commands and sends them to the wheelchair control system.

[0748] Step 9:

[0749] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[0750] Example: Monitoring concentration during a meeting

[0751] Step 1:

[0752] User: Business people participating in a meeting wear brainwave sensors.

[0753] Step 2:

[0754] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0755] Step 3:

[0756] Terminal: Encrypts the pre-processed data and sends it to the server.

[0757] Step 4:

[0758] Server: Receives the data and extracts features after decoding.

[0759] Step 5:

[0760] Server: Analyzes and evaluates the user's concentration level using a machine learning model, while simultaneously recognizing their emotional state using an emotion engine.

[0761] Step 6:

[0762] Server: When the threshold is exceeded, the AI ​​generates an appropriate warning message and alert based on the emotion.

[0763] Step 7:

[0764] Server: Encrypts the generated message and sends it to the device using a secure protocol.

[0765] Step 8:

[0766] Device: Decodes the received message and notifies the user, for example by displaying "Your concentration is declining" on the display and providing emotion-based stress reduction advice.

[0767] Example 2

[0768] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0769] Existing EEG analysis systems lack a unified process from acquiring EEG data to recognizing emotions and providing feedback to the user, making it difficult to accurately and quickly reflect the user's emotional state. Furthermore, for specific applications such as assisting physically paralyzed people or monitoring concentration levels during meetings, there was a lack of a system that provided appropriate feedback in real time according to user needs, making it difficult to improve user experience and work efficiency.

[0770] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0771] In this invention, the server includes means for extracting features from the EEG data and analyzing them using a machine learning model, means for recognizing the user's emotional state using an emotion recognition engine based on the analysis results, and means for generating output such as text, images, audio, and video using a generative AI model based on the recognized emotional state. This enables emotion recognition and the generation of appropriate feedback to be performed quickly and accurately based on the user's EEG data, making it possible to improve user experience and business efficiency in specific applications.

[0772] A "non-invasive EEG sensor" is a sensor worn on the head to acquire the user's brainwave signals, but collects data without invading the body.

[0773] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[0774] A "secure protocol" is a communication protocol that uses encryption technology to securely transmit data to another party in order to ensure confidentiality and integrity of the data.

[0775] "Features" are important data points or indicators extracted from EEG data and used for analysis.

[0776] A "machine learning model" is an algorithm or method used to analyze EEG data, primarily for pattern recognition, classification, and regression of data.

[0777] An "emotion recognition engine" is a model trained to identify and recognize a user's emotional state by analyzing EEG data and other physiological data.

[0778] A "generative AI model" is an AI model that generates output such as text, images, audio, or video based on a recognized emotional state. Representative examples include GPT-3 and DALL-E.

[0779] "Storage Device" means a device used to provide generated output back to the user, for storing and displaying data.

[0780] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion recognition engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion recognition engine, and a storage device that provides feedback on the output results to the user.

[0781] Hardware and software used

[0782] User: The user wears a non-invasive EEG sensor on their head, which converts electrical activity in the brain into digital signals and transmits them to a device in real time.

[0783] Terminal: Receives EEG signals sent from the EEG sensor. The terminal is equipped with software to remove noise and filter the received signals, and to perform preprocessing. This mainly involves band-pass filtering and detrending.

[0784] Server: The server receives the preprocessed EEG data via an encrypted protocol (e.g., HTTPS). The server extracts features using FFT (Fast Fourier Transform) or wavelet transform, and analyzes the data using a machine learning model. Machine learning libraries such as Python and TensorFlow are used here.

[0785] Emotion Recognition Engine: Recognizes the user's emotions based on the analysis of EEG data. The emotion recognition engine uses algorithms such as neural networks and support vector machines to identify emotions based on pre-trained models and analyzing EEG patterns and physiological data.

[0786] Generative AI model: Based on the emotional state recognized by the emotion recognition engine, the generative AI generates output such as text, images, audio, and video. For example, advanced generative AI models such as GPT-3 and DALL-E are used, which generate highly accurate output.

[0787] Storage device: Used to feed back the generated output data to the user. The data is encrypted again and sent to the terminal, where it is decrypted and stored in the storage device. The generated text or voice message is notified to the user through devices such as a display or speaker.

[0788] Specific examples

[0789] Example of assistive device operation for physically paralyzed people

[0790] User: A paralyzed person thinks about operating a wheelchair, for example, "Go forward."

[0791] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0792] Server: Receives and analyzes the data. From the analysis results, it determines that a "move forward" command has been issued. At the same time, it uses an emotion recognition engine to recognize the user's emotional state. It then uses a generative AI to generate feedback based on the operation command and emotion. A message such as "The wheelchair will move forward. Please relax" is generated.

[0793] Storage device: Sends the generated commands to the wheelchair control system, causing the wheelchair to perform the specified actions, while simultaneously displaying feedback messages to the user.

[0794] Example of concentration monitoring during a meeting

[0795] Users: Business people attending the meeting.

[0796] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0797] Server: Receives and analyzes the data. The level of concentration is evaluated based on the analysis results, and if it falls below a certain threshold, the emotion recognition engine recognizes the user's emotions. An alert message such as "Your concentration is declining. Take a deep breath and refresh yourself" is generated.

[0798] Storage device: Sends the generated alert message to the device, which notifies the user.

[0799] Prompt Sentence Examples

[0800] "Recommend music that users listen to when they are in a relaxed state."

[0801] "Generate an alert message when someone loses focus during a meeting."

[0802] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0803] Step 1:

[0804] User: Wears a non-invasive EEG sensor on the head, which measures the user's brainwave signals in real time and transmits the digital signal to the device.

[0805] Input: EEG signal

[0806] Output: Digital signal

[0807] Step 2:

[0808] Terminal: Pre-processing the received digital signal, specifically applying a band-pass filter to remove high frequency noise, and detrending to remove baseline drift.

[0809] Input: Digital signal

[0810] Output: Preprocessed EEG data

[0811] Step 3:

[0812] Terminal: Encrypt the pre-processed EEG data. Send the encrypted data to the server using a secure protocol (e.g., HTTPS).

[0813] Input: Preprocessed EEG data

[0814] Output: Encrypted EEG data

[0815] Step 4:

[0816] Server: Receives and decrypts the encrypted data sent from the device. Then, it extracts frequency components using FFT (Fast Fourier Transform) and obtains features with high time-frequency resolution using wavelet transform.

[0817] Input: Encrypted EEG data

[0818] Output: Feature data

[0819] Step 5:

[0820] Server: The feature data is input into a machine learning model for analysis. Here, analysis is performed using support vector machines and neural networks, and the user's EEG data is analyzed.

[0821] Input: Feature data

[0822] Output: Analysis results of EEG data

[0823] Step 6:

[0824] Server: The analysis results are input into the emotion recognition engine to recognize the user's emotional state. The emotion recognition engine analyzes the user's brainwave patterns based on a pre-trained dataset to identify emotions.

[0825] Input: Analysis results

[0826] Output: User's emotional state

[0827] Step 7:

[0828] Server: Based on the recognized emotional state, a generative AI model (e.g., GPT-3 or DALL-E) is used to generate output data (text, images, audio, video, etc.). For example, a prompt such as "Please recommend music that the user would listen to when in a relaxed state" is input into the generative AI, and output data is generated.

[0829] Input: User's emotional state

[0830] Output: The generated output data

[0831] Step 8:

[0832] Server: The generated output data is re-encrypted and sent to the terminal.

[0833] Input: Generated output data

[0834] Output: The encrypted output data.

[0835] Step 9:

[0836] Terminal: Decrypts the transmitted encrypted data, stores it on a storage device, and notifies the user of generated text and voice messages using devices such as a display and speaker.

[0837] Input: Encrypted output data

[0838] Output: The decoded output data

[0839] These are the specific processing steps of the program. Each step forms a series of steps based on the user's EEG data: data acquisition, preprocessing, analysis, generation, and feedback.

[0840] (Application example 2)

[0841] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0842] Conventional non-invasive EEG sensor systems have the technology to acquire users' EEG signals and provide feedback based on the analysis results, but they have not yet fully developed mechanisms for evaluating fatigue and concentration levels in real time and providing appropriate feedback based on that evaluation. Furthermore, they lack the functionality to link with visual devices worn by workers and manage the data in real time under specific work environments. This has made it difficult to efficiently manage worker health and safety and improve labor productivity.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring a user's EEG signal using a non-invasive EEG sensor; means for preprocessing the EEG signal to remove noise and perform filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing them using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for providing feedback of the output to the user in the storage device; and means for evaluating a worker's fatigue level and concentration level in real time in cooperation with a visual device worn by the worker and providing appropriate feedback. This makes it possible to evaluate a worker's fatigue level and concentration level in real time, thereby improving work efficiency while managing their health and safety.

[0844] A "non-invasive brainwave sensor" is a device that acquires a user's brainwave signals without invading the skin.

[0845] "User" refers to a person who operates or uses the system.

[0846] "Electroencephalogram signals" refer to changes in electrical potential generated by electrical activity in the brain.

[0847] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[0848] "Noise removal" is a process for removing unnecessary noise contained in the electroencephalogram signal.

[0849] "Filtering" is the process of conditioning a signal by passing only specific frequency bands.

[0850] A "secure protocol" is a secure communication protocol that ensures confidentiality and integrity of communication data.

[0851] A "server" is a central processing unit for analyzing and processing data.

[0852] "Features" refer to important data characteristics extracted from EEG signals.

[0853] A "machine learning model" is an algorithm that learns patterns based on data and makes predictions and classifications.

[0854] "Analysis" is a process for obtaining detailed information based on the acquired data.

[0855] "Generative AI" refers to the ability to generate output such as text, images, audio, and video using artificial intelligence technology.

[0856] "Storage device" refers to a device for storing data.

[0857] "Feedback" refers to returning analysis results or generated output to the user.

[0858] "Visual device" refers to a device worn by a user that provides information visually.

[0859] "Fatigue level" is an index for evaluating the degree of fatigue felt by the user.

[0860] "Level of concentration" is an index for evaluating the level of concentration of a user.

[0861] The present invention provides a system for acquiring a user's electroencephalogram (EEG) signal using a non-invasive electroencephalogram sensor, analyzing the signal, and providing appropriate feedback in real time. Specific embodiments of the present invention will be described in detail below.

[0862] System program description

[0863] EEG data collection and preprocessing

[0864] The user wears a non-invasive EEG sensor on their head, which captures EEG signals in real time and performs noise reduction and filtering. After pre-processing, the data is encrypted and sent to a server via a secure protocol. This process ensures data quality and confidentiality.

[0865] Sending data to the server and analyzing it

[0866] The server decodes the received preprocessed EEG data and uses algorithms such as fast Fourier transform (FFT) and wavelet transform to extract features, which are then input into a machine learning model to analyze the EEG data.

[0867] Emotion recognition by emotion engine

[0868] Based on the analysis results, the server recognizes the user's emotional state using an emotion engine, which uses a trained model to identify the user's emotional state from numerous brainwave patterns and physiological data.

[0869] Feedback generation by generative AI

[0870] Based on the emotional state recognized by the emotion engine, the server uses generative AI to create appropriate feedback, using, for example, advanced text and image generation models to generate highly accurate output.

[0871] Feedback of results

[0872] The generated feedback data is then encrypted again and sent via a secure protocol to the device, which decrypts the data and presents it to the user via a visual device (such as smart glasses).

[0873] Hardware and software used

[0874] Brainwave sensors: Non-invasive devices such as Emotiv and Muse.

[0875] Server: A central processing unit for running data analysis and generative AI models, using Python and libraries such as SciPy and Sklearn.

[0876] Visual devices: smart glasses (e.g., Google Glass, Microsoft HoloLens).

[0877] Communication protocol: Encrypted communication (e.g. HTTPS).

[0878] Specific examples

[0879] Application example: Factory worker monitoring

[0880] Factory workers wear smart glasses, and their brainwaves are monitored in real time using an EEG sensor. If a worker continues working for an extended period of time, data is collected to assess their level of fatigue and concentration. Based on the analysis results, the server uses a generative AI to provide appropriate feedback. For example, if the worker's level of fatigue is high, the smart glasses will display a message saying, "We recommend taking a break."

[0881] Example prompt sentence:

[0882] "Workers working long hours on an assembly line were monitored for brainwaves to assess their fatigue levels. Appropriate feedback was then generated for the workers."

[0883] As such, the present invention is a system that analyzes the user's brainwave data in real time and provides appropriate feedback using an emotion engine and generative AI, thereby improving work efficiency while managing the health and safety of workers.

[0884] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0885] Step 1:

[0886] The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time. The captured EEG signals are sent to a terminal for noise reduction and filtering. The input is the raw EEG signal, and the output is pre-processed clean EEG data.

[0887] Step 2:

[0888] The device performs noise reduction and bandpass filtering on the received EEG signal. Specifically, it uses a Butterworth filter to pass only the frequency band between 0.5 Hz and 30 Hz. This processing improves the signal quality and removes unnecessary noise. The input is the raw EEG signal, and the output is preprocessed EEG data.

[0889] Step 3:

[0890] The preprocessed EEG data is encrypted and sent from the device to the server using the HTTPS protocol. The server decrypts the received data and prepares it for analysis. The input is the preprocessed EEG data, and the output is the decrypted EEG data.

[0891] Step 4:

[0892] The server extracts features from the decoded EEG data. Specifically, it uses FFT (Fast Fourier Transform) and wavelet transform to separate different frequency components of the EEG signal. The extracted features are input into a machine learning model for further analysis. The input is the decoded EEG data, and the output is the extracted features.

[0893] Step 5:

[0894] The server inputs the extracted features into a machine learning model and analyzes the EEG data. The analysis results are processed by an emotion engine to identify the appropriate emotional state based on information obtained from the user's EEG patterns. The input is feature data, and the output is an evaluation of the emotional state.

[0895] Step 6:

[0896] The server uses generative AI to generate optimal feedback based on the emotional state recognized by the emotion engine. Specifically, it uses GPT-3 or a similar generative AI model to generate appropriate output such as text, images, audio, and video. The input is the evaluation result of the emotional state, and the output is the generated feedback.

[0897] Step 7:

[0898] The generated feedback data is encrypted and sent back to the terminal from the server. The terminal decrypts the data and stores it in a storage device. The input is the generated feedback data, and the output is the decrypted feedback data.

[0899] Step 8:

[0900] The device then notifies the user of the decoded feedback via a visual device such as smart glasses. For example, if the fatigue level is high, a message such as "We recommend you take a break" is displayed. The input is the decoded feedback data, and the output is a notification message displayed to the user.

[0901] This makes it possible to analyze the user's brain wave data in real time and provide appropriate feedback.

[0902] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0903] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0904] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0905] [Third embodiment]

[0906] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0907] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0908] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0909] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0910] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0911] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0912] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0913] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0914] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0915] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0916] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0917] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0918] This invention is a system that analyzes EEG signals acquired by an EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[0919] EEG data collection and preprocessing

[0920] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device then performs pre-processing on the received EEG signals, such as noise reduction and filtering, to improve signal quality and increase analysis accuracy.

[0921] Sending data to the server

[0922] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[0923] Analysis of EEG data

[0924] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's brainwave data. The analysis results in the user's thoughts and emotional state.

[0925] Realizing output through generative AI

[0926] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. The generative AI uses various generative models (e.g., natural language processing models and image generation models) to create highly accurate output.

[0927] Feedback of results

[0928] Server: The generated output data is formatted into an appropriate format, encrypted again, and sent to the terminal. The terminal decrypts the sent data and saves it to a storage device. The storage device notifies or displays the saved result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[0929] Specific examples

[0930] Example of assistive device operation for physically paralyzed people

[0931] User: A paralyzed person thinks about operating a wheelchair.

[0932] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0933] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." It then uses a generation AI to generate an operation command.

[0934] Storage device: Sends the generated instructions to the wheelchair control system, causing the wheelchair to perform the specified actions.

[0935] Example of concentration monitoring during a meeting

[0936] Users: Business people attending the meeting.

[0937] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[0938] Server: Receives and analyzes the data. Based on the analysis results, the user's concentration level is evaluated. If it falls below a certain threshold, the generation AI generates a warning message.

[0939] Storage device: The generated warning message is sent to the device, which notifies the user, for example, by displaying "Your concentration level is decreasing" on the display.

[0940] The processing flow will be explained below.

[0941] Step 1:

[0942] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[0943] Step 2:

[0944] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[0945] Step 3:

[0946] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[0947] Step 4:

[0948] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[0949] Step 5:

[0950] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[0951] Step 6:

[0952] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[0953] Step 7:

[0954] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[0955] Step 8:

[0956] Server: Based on the analysis results, generative AI is used to generate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[0957] Step 9:

[0958] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[0959] Step 10:

[0960] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[0961] Step 11:

[0962] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[0963] As a concrete example, the case of a physically paralyzed person operating a wheelchair is shown below.

[0964] Step 1:

[0965] User: A paralyzed person wears a brainwave sensor on their head to help them think about operating a wheelchair.

[0966] Step 2:

[0967] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[0968] Step 3:

[0969] Terminal: Encrypts the preprocessed data and sends it to the server.

[0970] Step 4:

[0971] Server: Receives the data and extracts features after decoding.

[0972] Step 5:

[0973] Server: A machine learning model analyzes the user's thoughts, and the generation AI generates operation commands (e.g., "go forward," "go backward," "turn left," "turn right").

[0974] Step 6:

[0975] Server: Encrypts and transmits commands to the terminal.

[0976] Step 7:

[0977] Terminal: Decodes commands and sends instructions to the wheelchair control system.

[0978] Step 8:

[0979] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[0980] Through these specific process steps, the invention can be effectively implemented.

[0981] Example 1

[0982] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0983] The present invention aims to provide a system that acquires and analyzes a user's EEG signals in real time to accurately grasp the user's thoughts and emotions and provide appropriate feedback based on the acquired signals. Current systems face challenges in removing noise from EEG signals obtained from non-invasive EEG sensors, securely transmitting data, improving analytical accuracy, and providing feedback in a variety of output formats using generative AI. These challenges must be addressed, for example, when a physically paralyzed person needs to operate an assistive device by thought, or when monitoring a user's concentration level and generating appropriate warnings.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0985] In this invention, the server includes means for decoding the EEG data and performing a fast Fourier transform to extract features, means for inputting the features into a machine learning model and analyzing them, and means for generating output such as text, images, audio, and video using a generative AI based on the analysis results. This enables highly accurate analysis based on the user's EEG signals and the generation of appropriate output.

[0986] A "non-invasive EEG sensor" is a sensor that is worn on the user's head to acquire EEG signals in real time, and is a device that can be used without damaging the user's skin.

[0987] "Preprocessing" refers to processing that removes noise and filters the EEG signal to improve the quality of the signal.

[0988] "Noise reduction" is the process of removing unnecessary noise and external interference from EEG signals.

[0989] "Filtering" is a process of extracting specific frequency components from an EEG signal or removing unnecessary frequency components.

[0990] A "secure protocol" is a communication method that ensures confidentiality and integrity of data and is a protocol that has encryption and authentication functions.

[0991] "Encryption" is a process of converting data to prevent the contents of the data being transmitted from being read by a third party.

[0992] "Decryption" is the process of restoring encrypted data to its original state.

[0993] "Feature extraction" is the process of finding the characteristics and patterns necessary for analysis from EEG signals.

[0994] The "Fast Fourier Transform (FFT)" is a mathematical technique for converting time-domain signals into the frequency domain for analysis.

[0995] A "machine learning model" is a model that learns rules and patterns from data and makes predictions and classifications.

[0996] "Generative AI" is a general term for artificial intelligence technology that generates output such as text, images, audio, and video based on analysis results.

[0997] "Text, image, audio, video, etc. output" refers to the form of information generated based on the user's thoughts and feelings.

[0998] "Formatting" is the process of putting data into a specific form.

[0999] "Storage Device" means a device that stores the generated output and communicates or displays it to the user when necessary.

[1000] This invention is a system that analyzes EEG signals acquired using a non-invasive EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[1001] Hardware and software used

[1002] Brainwave sensor: A non-invasive brainwave sensor is worn on the user's head to capture brainwave signals in real time. This sensor has wireless communication capabilities and transmits the brainwave signals to a device.

[1003] Terminal: The terminal processes the EEG signals received from the sensor. Pre-processing such as noise reduction and filtering is performed on the terminal. Specifically, band-pass filters and artifact removal algorithms are used to make the signals clearer.

[1004] Server: The server receives and decrypts the encrypted EEG data sent from the device. It then extracts features using an FFT (Fast Fourier Transform). The extracted features are input into a machine learning model to analyze the user's EEG data. Based on the analysis results, a generative AI generates output such as text, images, audio, and video. The generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[1005] Storage Device: A storage device is a device that stores the generated output data and provides feedback to the user, for example, by displaying a message on a display or playing an audio message through a speaker.

[1006] Specific examples

[1007] Example of assistive device operation for physically paralyzed people

[1008] User: A paralyzed person thinks about operating a wheelchair.

[1009] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1010] Server: Receives the data and decrypts the encrypted data. Analyzes the decrypted data and identifies the thought content, which is either "go forward," "go backward," "turn left," or "turn right." Then, generates an operation command using a generation AI. The generated command is re-encrypted and sent to the terminal.

[1011] Terminal: Decodes the command and sends it to the wheelchair control system.

[1012] Wheelchair: Follows the received operation command and performs the specified action (forward, backward, etc.).

[1013] Example prompt: "Get thoughts to move the wheelchair forward, then use generative AI to convert those thoughts into commands to control the wheelchair."

[1014] Example of concentration monitoring during a meeting

[1015] User: Business people attending a meeting.

[1016] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1017] Server: Receives data and decrypts the encrypted data. Analyzes the decrypted data and evaluates the user's concentration. If it falls below a certain threshold, the AI ​​generates a warning message. The generated message is re-encrypted and sent to the device.

[1018] On your device: Decrypt the warning message and save it to your storage device.

[1019] Storage device: Reads the saved warning message and notifies the user by displaying "Your concentration level is decreasing" on the display, for example.

[1020] Example prompt: "Collect EEG data to monitor concentration during meetings, and evaluate the concentration level based on the analysis results. Generate a warning message if the concentration level falls below the threshold."

[1021] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1022] Step 1: Collecting EEG data

[1023] User: The user wears a non-invasive EEG sensor on their head, which captures real-time EEG signals and transmits them wirelessly to a device.

[1024] Input: User's EEG signal

[1025] Output: Raw EEG data sent from the sensor to the device

[1026] Step 2: Preprocessing the EEG data

[1027] Device: The device performs noise reduction and filtering on the EEG signals received from the sensor. Specifically, it uses a bandpass filter to extract specific frequency components and an artifact removal algorithm to remove unnecessary components.

[1028] Input: Raw EEG data sent from the sensor

[1029] Output: Pre-processed, high-quality EEG data

[1030] Step 3: Sending data to the server

[1031] Terminal: The pre-processed EEG data is encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[1032] Input: Preprocessed EEG data

[1033] Output: Encrypted EEG data is sent to the server

[1034] Step 4: Decoding EEG data and extracting features

[1035] Server: The server receives the encrypted EEG data, decrypts it, and extracts features from the decrypted data using FFT (Fast Fourier Transform).

[1036] Input: Encrypted EEG data

[1037] Output: Features extracted by FFT

[1038] Step 5: Analyze the EEG data

[1039] Server: The extracted features are input into a machine learning model to analyze the user's thoughts and emotional state. The machine learning model uses SVM, neural networks, etc.

[1040] Input: Extracted features

[1041] Output: Typical analysis results (thoughts and emotional state)

[1042] Step 6: Realizing the output of generative AI

[1043] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. This generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[1044] Input: Analysis results

[1045] Output: The generated output, such as text, images, audio, or video.

[1046] Step 7: Encrypt data and send it to your device

[1047] Server: Formats the generated output data into the appropriate format, encrypts it again, and sends it to the device.

[1048] Input: Generated output data

[1049] Output: The encrypted output data.

[1050] Step 8: Decrypt and save the data

[1051] Terminal: Decrypts the received encrypted output data and stores it on a storage device.

[1052] Input: Encrypted output data

[1053] Output: The decoded output data

[1054] Step 9: Feedback on results

[1055] Storage device: Reads the saved output data and notifies or displays it to the user, for example, by displaying a message on the display or playing an audio message through the speaker.

[1056] Input: Decoded output data

[1057] Output: what is displayed or notified to the user

[1058] (Application example 1)

[1059] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1060] In conventional brick-and-mortar stores, users had to proactively inquire about products to obtain information and receive appropriate advice. Furthermore, there were few ways to grasp in real time which products users were interested in and what their purchasing intentions were, making it difficult to provide effective customer service. The present invention aims to provide a new solution for analyzing users' brainwave data and improving the shopping experience in brick-and-mortar stores.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1062] In this invention, the server includes: means for acquiring a user's electroencephalogram (EEG) signal using a non-invasive EEG sensor; means for preprocessing the EEG signal and performing noise reduction and filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing it using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for feeding back the output to the user in the storage device; means for identifying from the EEG data whether the user is interested in a product in a physical store or has a purchase intent; and means for providing the user with relevant product information and advice based on the identified interest and purchase intent. This makes it possible to provide optimal product information and advice in real time based on the user's interest and purchase intent, thereby improving the shopping experience.

[1063] A "non-invasive EEG sensor" is a sensor that can acquire EEG signals simply by wearing it, without physically invading the body.

[1064] "EEG signals" are electrical signals that reflect the user's brain activity.

[1065] "Preprocessing" refers to processing that removes noise and filters the EEG signals to improve the accuracy of analysis.

[1066] "Noise removal" is a process for removing unnecessary components contained in an electroencephalogram signal.

[1067] "Filtering" is a signal processing technique that extracts or suppresses only specific frequency bands.

[1068] A "secure protocol" is a communication protocol that ensures confidentiality and integrity in data communication.

[1069] "Features" are important information extracted from electroencephalogram data for analysis.

[1070] A "machine learning model" is an algorithm that learns patterns based on given data and makes predictions and classifications for future data.

[1071] "Generative AI" is a system that uses artificial intelligence technology to automatically generate output such as text, images, audio, and video.

[1072] A "storage device" is a device that stores data and provides feedback to the user as needed.

[1073] "Interest" refers to the attention or concern a user has towards a particular product.

[1074] "Purchase intent" refers to a user's desire or motivation to purchase a particular product.

[1075] "Product information" is information about the characteristics and features of a product.

[1076] "Advice" means any purchasing advice or recommendation provided to a User.

[1077] A system for implementing the present invention includes the following hardware and software.

[1078] Hardware

[1079] 1. EEG sensor:

[1080] Non-invasive EEG sensors: For example, by wearing specific headgear, the user's brainwave signals are acquired in real time. For example, high-performance EEG sensors can be used to accurately reflect the user's mental state and interests.

[1081] 2. Terminal:

[1082] Smart glasses: Devices equipped with displays and speakers that display information and provide audio feedback to users, allowing them to obtain the information they need in real time while shopping in physical stores.

[1083] This device has the processing power to receive data transmitted from the EEG sensor and perform initial pre-processing.

[1084] software

[1085] 1. Server:

[1086] Data processing and analysis: The server receives the data sent from the EEG sensor and the device, decrypts it, and extracts features from the preprocessed EEG data using algorithms such as FFT (Fast Fourier Transform).

[1087] Machine learning model: The extracted features are input into a machine learning model to analyze the user's brainwave data, which can determine whether the user is interested in a particular product and whether they have the intention to purchase it.

[1088] Generative AI: Based on the analysis results, generative AI generates output such as text, images, audio, and video. For example, it uses natural language processing models and image generation models (e.g., GPT-3 and DALL-E).

[1089] Example

[1090] 1. Data Preprocessing:

[1091] The user wears an EEG sensor, and the EEG signal is transmitted to a device, which then performs noise reduction and filtering to obtain high-quality data.

[1092] 2. Data transmission and analysis:

[1093] The preprocessed EEG data is sent to a server using a secure protocol (e.g., HTTPS). The server decrypts the data and extracts features using algorithms such as FFT. The user's EEG data is then analyzed by a machine learning model to identify the user's thoughts and emotional state.

[1094] 3. Generative AI output:

[1095] Based on the analysis results, generative AI generates output in the form of text, images, audio, video, etc. For example, if a user shows interest in a particular product, detailed information about that product and related product advice will be generated.

[1096] Specific examples from physical stores

[1097] Brainwave sensors: Users wearing high-performance, non-invasive brainwave sensors enter physical stores.

[1098] Smart glasses: When a user expresses interest in a particular product, the smart glasses display details about that product and offer advice on related products, and may also provide audio information through the speakers.

[1099] Example prompt for the generative AI:

[1100] "Users are interested in specific products. Based on this analysis, please provide product details and related products."

[1101] The system will significantly enhance the in-store shopping experience, allowing users to get real-time information and advice on products they are interested in or interested in purchasing.

[1102] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1103] Step 1:

[1104] The user wears a non-invasive EEG sensor. The sensor captures the user's EEG signals in real time and transmits them to a device. The input is the user's EEG, and the output is raw EEG data. The device receives this data.

[1105] Step 2:

[1106] The device performs preprocessing on the received EEG signals, specifically noise removal and filtering. The input is raw EEG data, and the output is clear, preprocessed EEG data. This improves the quality of the data and the accuracy of subsequent analysis.

[1107] Step 3:

[1108] The terminal encrypts the preprocessed EEG data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the preprocessed EEG data, and the output is the encrypted EEG data. The server receives this data.

[1109] Step 4:

[1110] The server decrypts the received data and extracts features using algorithms such as FFT (Fast Fourier Transform). The input is encrypted EEG data, and the output is features decomposed into specific frequency components. This makes it easier to analyze the data.

[1111] Step 5:

[1112] The extracted features are input into a machine learning model. The server analyzes this data using the machine learning model to identify the user's interests and purchasing intent. The input is the features, and the output is the identified user's interests and purchasing intent. This analysis reveals the user's state of mind.

[1113] Step 6:

[1114] The server uses a generative AI model based on the analysis results to generate output such as text, images, audio, and video. The input is the analysis results such as the user's interests and purchasing intentions, and the output is the generated content. An example of a prompt for the generative AI model is, "The user is interested in a specific product. Based on this analysis result, please provide product details and related products."

[1115] Step 7:

[1116] The generated output data is formatted into an appropriate format by the server, encrypted again, and then sent to the terminal. The input is the generated content, and the output is the encrypted content data.

[1117] Step 8:

[1118] The device decrypts the transmitted data and provides feedback to the user through the smart glasses' display and speaker. The input is encrypted content data, and the output is information and advice provided to the user. This allows users to obtain information and advice about products of interest in real time.

[1119] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1120] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion engine, and a storage device that provides feedback on the output results to the user.

[1121] EEG data collection and preprocessing

[1122] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device performs pre-processing such as noise reduction and filtering on the received EEG signals to improve the signal quality.

[1123] Sending data to the server

[1124] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[1125] Analysis of EEG data

[1126] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) and wavelet transform to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's EEG data.

[1127] Emotion recognition by emotion engine

[1128] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses trained models to identify the user's emotions from brainwave patterns and physiological data.

[1129] Realizing output through generative AI

[1130] Server: Based on the emotional state recognized by the emotion engine, the generative AI generates appropriate output such as text, images, audio, and video. The generative AI uses advanced models such as GPT-3 and DALL-E to create highly accurate output.

[1131] Feedback of results

[1132] Server: The generated output data is formatted appropriately, encrypted again, and sent to the device. The device decrypts the data and stores it in a storage device. The storage device notifies or displays the stored result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[1133] Specific examples

[1134] Example of assistive device operation for physically paralyzed people

[1135] User: A paralyzed person thinks about operating a wheelchair.

[1136] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1137] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." At the same time, the emotion engine recognizes the user's emotional state. Then, it uses generative AI to generate operational commands and feedback based on the user's emotional state.

[1138] Storage device: Sends the generated command to the wheelchair control system, which then executes the specified action. At the same time, the wheelchair provides feedback based on the action results and the user's emotions.

[1139] Example of concentration monitoring during a meeting

[1140] Users: Business people attending the meeting.

[1141] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1142] Server: Receives and analyzes the data. Based on the analysis results, the user's level of concentration is evaluated. If it falls below a certain threshold, the emotion engine recognizes the user's emotion, and the generative AI generates optimal feedback.

[1143] Storage device: Generated alerts based on the concentration level and emotions are sent to the device, which then notifies the user. For example, the device may display "Your concentration is declining" on the display, along with advice on how to reduce stress.

[1144] In this way, the system of the present invention analyzes the user's EEG data and generates appropriate feedback based on the emotional state recognized by the emotion engine, thereby enriching the user's experience and supporting efficient communication and work performance.

[1145] The processing flow will be explained below.

[1146] Step 1:

[1147] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[1148] Step 2:

[1149] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[1150] Step 3:

[1151] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[1152] Step 4:

[1153] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[1154] Step 5:

[1155] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[1156] Step 6:

[1157] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[1158] Step 7:

[1159] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[1160] Step 8:

[1161] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses algorithms to identify emotions from brainwave patterns and other physiological data.

[1162] Step 9:

[1163] Server: Based on the emotional state recognized by the emotion engine, generative AI is used to generate appropriate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[1164] Step 10:

[1165] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[1166] Step 11:

[1167] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[1168] Step 12:

[1169] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[1170] Example: Operating assistive devices for physically paralyzed people

[1171] Step 1:

[1172] User: A paralyzed person wears an EEG sensor on their head.

[1173] Step 2:

[1174] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[1175] Step 3:

[1176] Terminal: Encrypts the pre-processed data and sends it to the server.

[1177] Step 4:

[1178] Server: Receives the data and extracts features after decoding.

[1179] Step 5:

[1180] Server: A machine learning model analyzes the user's thoughts and generates operational commands such as "go forward" or "go backward." At the same time, an emotion engine recognizes the user's emotional state.

[1181] Step 6:

[1182] Server: The generation AI generates feedback based on the control command and emotional state.

[1183] Step 7:

[1184] Server: Encrypts the generated commands and sends them to the device using a secure protocol.

[1185] Step 8:

[1186] Terminal: Decodes the received commands and sends them to the wheelchair control system.

[1187] Step 9:

[1188] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[1189] Example: Monitoring concentration during a meeting

[1190] Step 1:

[1191] User: Business people participating in a meeting wear brainwave sensors.

[1192] Step 2:

[1193] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[1194] Step 3:

[1195] Terminal: Encrypts the pre-processed data and sends it to the server.

[1196] Step 4:

[1197] Server: Receives the data and extracts features after decoding.

[1198] Step 5:

[1199] Server: Analyzes and evaluates the user's concentration level using a machine learning model, while simultaneously recognizing their emotional state using an emotion engine.

[1200] Step 6:

[1201] Server: When the threshold is exceeded, the AI ​​generates an appropriate warning message and alert based on the emotion.

[1202] Step 7:

[1203] Server: Encrypts the generated message and sends it to the device using a secure protocol.

[1204] Step 8:

[1205] Device: Decodes the received message and notifies the user, for example by displaying "Your concentration is declining" on the display and providing emotion-based stress reduction advice.

[1206] Example 2

[1207] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] Existing EEG analysis systems lack a unified process from acquiring EEG data to recognizing emotions and providing feedback to the user, making it difficult to accurately and quickly reflect the user's emotional state. Furthermore, for specific applications such as assisting physically paralyzed people or monitoring concentration levels during meetings, there was a lack of a system that provided appropriate feedback in real time according to user needs, making it difficult to improve user experience and work efficiency.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1210] In this invention, the server includes means for extracting features from the EEG data and analyzing them using a machine learning model, means for recognizing the user's emotional state using an emotion recognition engine based on the analysis results, and means for generating output such as text, images, audio, and video using a generative AI model based on the recognized emotional state. This enables emotion recognition and the generation of appropriate feedback to be performed quickly and accurately based on the user's EEG data, making it possible to improve user experience and business efficiency in specific applications.

[1211] A "non-invasive EEG sensor" is a sensor worn on the head to acquire the user's brainwave signals, but collects data without invading the body.

[1212] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[1213] A "secure protocol" is a communication protocol that uses encryption technology to securely transmit data to another party in order to ensure confidentiality and integrity of the data.

[1214] "Features" are important data points or indicators extracted from EEG data and used for analysis.

[1215] A "machine learning model" is an algorithm or method used to analyze EEG data, primarily for pattern recognition, classification, and regression of data.

[1216] An "emotion recognition engine" is a model trained to identify and recognize a user's emotional state by analyzing EEG data and other physiological data.

[1217] A "generative AI model" is an AI model that generates output such as text, images, audio, or video based on a recognized emotional state. Representative examples include GPT-3 and DALL-E.

[1218] "Storage Device" means a device used to provide generated output back to the user, for storing and displaying data.

[1219] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion recognition engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion recognition engine, and a storage device that provides feedback on the output results to the user.

[1220] Hardware and software used

[1221] User: The user wears a non-invasive EEG sensor on their head, which converts electrical activity in the brain into digital signals and transmits them to a device in real time.

[1222] Terminal: Receives EEG signals sent from the EEG sensor. The terminal is equipped with software to remove noise and filter the received signals, and to perform preprocessing. This mainly involves band-pass filtering and detrending.

[1223] Server: The server receives the preprocessed EEG data via an encrypted protocol (e.g., HTTPS). The server extracts features using FFT (Fast Fourier Transform) or wavelet transform, and analyzes the data using a machine learning model. Machine learning libraries such as Python and TensorFlow are used here.

[1224] Emotion Recognition Engine: Recognizes the user's emotions based on the analysis of EEG data. The emotion recognition engine uses algorithms such as neural networks and support vector machines to identify emotions based on pre-trained models and analyzing EEG patterns and physiological data.

[1225] Generative AI model: Based on the emotional state recognized by the emotion recognition engine, the generative AI generates output such as text, images, audio, and video. For example, advanced generative AI models such as GPT-3 and DALL-E are used, which generate highly accurate output.

[1226] Storage device: Used to feed back the generated output data to the user. The data is encrypted again and sent to the terminal, where it is decrypted and stored in the storage device. The generated text or voice message is notified to the user through devices such as a display or speaker.

[1227] Specific examples

[1228] Example of assistive device operation for physically paralyzed people

[1229] User: A paralyzed person thinks about operating a wheelchair, for example, "Go forward."

[1230] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1231] Server: Receives and analyzes the data. From the analysis results, it determines that a "move forward" command has been issued. At the same time, it uses an emotion recognition engine to recognize the user's emotional state. It then uses a generative AI to generate feedback based on the operation command and emotion. A message such as "The wheelchair will move forward. Please relax" is generated.

[1232] Storage device: Sends the generated commands to the wheelchair control system, causing the wheelchair to perform the specified actions, while simultaneously displaying feedback messages to the user.

[1233] Example of concentration monitoring during a meeting

[1234] Users: Business people attending the meeting.

[1235] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1236] Server: Receives and analyzes the data. The level of concentration is evaluated based on the analysis results, and if it falls below a certain threshold, the emotion recognition engine recognizes the user's emotions. An alert message such as "Your concentration is declining. Take a deep breath and refresh yourself" is generated.

[1237] Storage device: Sends the generated alert message to the device, which notifies the user.

[1238] Prompt Sentence Examples

[1239] "Recommend music that users listen to when they are in a relaxed state."

[1240] "Generate an alert message when someone loses focus during a meeting."

[1241] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] User: Wears a non-invasive EEG sensor on the head, which measures the user's brainwave signals in real time and transmits the digital signal to the device.

[1244] Input: EEG signal

[1245] Output: Digital signal

[1246] Step 2:

[1247] Terminal: Pre-processing the received digital signal, specifically applying a band-pass filter to remove high frequency noise, and detrending to remove baseline drift.

[1248] Input: Digital signal

[1249] Output: Preprocessed EEG data

[1250] Step 3:

[1251] Terminal: Encrypt the pre-processed EEG data. Send the encrypted data to the server using a secure protocol (e.g., HTTPS).

[1252] Input: Preprocessed EEG data

[1253] Output: Encrypted EEG data

[1254] Step 4:

[1255] Server: Receives and decrypts the encrypted data sent from the device. Then, it extracts frequency components using FFT (Fast Fourier Transform) and obtains features with high time-frequency resolution using wavelet transform.

[1256] Input: Encrypted EEG data

[1257] Output: Feature data

[1258] Step 5:

[1259] Server: The feature data is input into a machine learning model for analysis. Here, analysis is performed using support vector machines and neural networks, and the user's EEG data is analyzed.

[1260] Input: Feature data

[1261] Output: Analysis results of EEG data

[1262] Step 6:

[1263] Server: The analysis results are input into the emotion recognition engine to recognize the user's emotional state. The emotion recognition engine analyzes the user's brainwave patterns based on a pre-trained dataset to identify emotions.

[1264] Input: Analysis results

[1265] Output: User's emotional state

[1266] Step 7:

[1267] Server: Based on the recognized emotional state, a generative AI model (e.g., GPT-3 or DALL-E) is used to generate output data (text, images, audio, video, etc.). For example, a prompt such as "Please recommend music that the user would listen to when in a relaxed state" is input into the generative AI, and output data is generated.

[1268] Input: User's emotional state

[1269] Output: The generated output data

[1270] Step 8:

[1271] Server: The generated output data is re-encrypted and sent to the terminal.

[1272] Input: Generated output data

[1273] Output: The encrypted output data.

[1274] Step 9:

[1275] Terminal: Decrypts the transmitted encrypted data, stores it on a storage device, and notifies the user of generated text and voice messages using devices such as a display and speaker.

[1276] Input: Encrypted output data

[1277] Output: The decoded output data

[1278] These are the specific processing steps of the program. Each step forms a series of steps based on the user's EEG data: data acquisition, preprocessing, analysis, generation, and feedback.

[1279] (Application example 2)

[1280] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1281] Conventional non-invasive EEG sensor systems have the technology to acquire users' EEG signals and provide feedback based on the analysis results, but they have not yet fully developed mechanisms for evaluating fatigue and concentration levels in real time and providing appropriate feedback based on that evaluation. Furthermore, they lack the functionality to link with visual devices worn by workers and manage the data in real time under specific work environments. This has made it difficult to efficiently manage worker health and safety and improve labor productivity.

[1282] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring a user's EEG signal using a non-invasive EEG sensor; means for preprocessing the EEG signal to remove noise and perform filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing them using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for providing feedback of the output to the user in the storage device; and means for evaluating a worker's fatigue level and concentration level in real time in cooperation with a visual device worn by the worker and providing appropriate feedback. This makes it possible to evaluate a worker's fatigue level and concentration level in real time, thereby improving work efficiency while managing their health and safety.

[1283] A "non-invasive brainwave sensor" is a device that acquires a user's brainwave signals without invading the skin.

[1284] "User" refers to a person who operates or uses the system.

[1285] "Electroencephalogram signals" refer to changes in electrical potential generated by electrical activity in the brain.

[1286] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[1287] "Noise removal" is a process for removing unnecessary noise contained in the electroencephalogram signal.

[1288] "Filtering" is the process of conditioning a signal by passing only specific frequency bands.

[1289] A "secure protocol" is a secure communication protocol that ensures confidentiality and integrity of communication data.

[1290] A "server" is a central processing unit for analyzing and processing data.

[1291] "Features" refer to important data characteristics extracted from EEG signals.

[1292] A "machine learning model" is an algorithm that learns patterns based on data and makes predictions and classifications.

[1293] "Analysis" is a process for obtaining detailed information based on the acquired data.

[1294] "Generative AI" refers to the ability to generate output such as text, images, audio, and video using artificial intelligence technology.

[1295] "Storage device" refers to a device for storing data.

[1296] "Feedback" refers to returning analysis results or generated output to the user.

[1297] "Visual device" refers to a device worn by a user that provides information visually.

[1298] "Fatigue level" is an index for evaluating the degree of fatigue felt by the user.

[1299] "Level of concentration" is an index for evaluating the level of concentration of a user.

[1300] The present invention provides a system for acquiring a user's electroencephalogram (EEG) signal using a non-invasive electroencephalogram sensor, analyzing the signal, and providing appropriate feedback in real time. Specific embodiments of the present invention will be described in detail below.

[1301] System program description

[1302] EEG data collection and preprocessing

[1303] The user wears a non-invasive EEG sensor on their head, which captures EEG signals in real time and performs noise reduction and filtering. After pre-processing, the data is encrypted and sent to a server via a secure protocol. This process ensures data quality and confidentiality.

[1304] Sending data to the server and analyzing it

[1305] The server decodes the received preprocessed EEG data and uses algorithms such as fast Fourier transform (FFT) and wavelet transform to extract features, which are then input into a machine learning model to analyze the EEG data.

[1306] Emotion recognition by emotion engine

[1307] Based on the analysis results, the server recognizes the user's emotional state using an emotion engine, which uses a trained model to identify the user's emotional state from numerous brainwave patterns and physiological data.

[1308] Feedback generation by generative AI

[1309] Based on the emotional state recognized by the emotion engine, the server uses generative AI to create appropriate feedback, using, for example, advanced text and image generation models to generate highly accurate output.

[1310] Feedback of results

[1311] The generated feedback data is then encrypted again and sent via a secure protocol to the device, which decrypts the data and presents it to the user via a visual device (such as smart glasses).

[1312] Hardware and software used

[1313] Brainwave sensors: Non-invasive devices such as Emotiv and Muse.

[1314] Server: A central processing unit for running data analysis and generative AI models, using Python and libraries such as SciPy and Sklearn.

[1315] Visual devices: smart glasses (e.g., Google Glass, Microsoft HoloLens).

[1316] Communication protocol: Encrypted communication (e.g. HTTPS).

[1317] Specific examples

[1318] Application example: Factory worker monitoring

[1319] Factory workers wear smart glasses, and their brainwaves are monitored in real time using an EEG sensor. If a worker continues working for an extended period of time, data is collected to assess their level of fatigue and concentration. Based on the analysis results, the server uses a generative AI to provide appropriate feedback. For example, if the worker's level of fatigue is high, the smart glasses will display a message saying, "We recommend taking a break."

[1320] Example prompt sentence:

[1321] "Workers working long hours on an assembly line were monitored for brainwaves to assess their fatigue levels. Appropriate feedback was then generated for the workers."

[1322] As such, the present invention is a system that analyzes the user's brainwave data in real time and provides appropriate feedback using an emotion engine and generative AI, thereby improving work efficiency while managing the health and safety of workers.

[1323] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1324] Step 1:

[1325] The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time. The captured EEG signals are sent to a terminal for noise reduction and filtering. The input is the raw EEG signal, and the output is pre-processed clean EEG data.

[1326] Step 2:

[1327] The device performs noise reduction and bandpass filtering on the received EEG signal. Specifically, it uses a Butterworth filter to pass only the frequency band between 0.5 Hz and 30 Hz. This processing improves the signal quality and removes unnecessary noise. The input is the raw EEG signal, and the output is preprocessed EEG data.

[1328] Step 3:

[1329] The preprocessed EEG data is encrypted and sent from the device to the server using the HTTPS protocol. The server decrypts the received data and prepares it for analysis. The input is the preprocessed EEG data, and the output is the decrypted EEG data.

[1330] Step 4:

[1331] The server extracts features from the decoded EEG data. Specifically, it uses FFT (Fast Fourier Transform) and wavelet transform to separate different frequency components of the EEG signal. The extracted features are input into a machine learning model for further analysis. The input is the decoded EEG data, and the output is the extracted features.

[1332] Step 5:

[1333] The server inputs the extracted features into a machine learning model and analyzes the EEG data. The analysis results are processed by an emotion engine to identify the appropriate emotional state based on information obtained from the user's EEG patterns. The input is feature data, and the output is an evaluation of the emotional state.

[1334] Step 6:

[1335] The server uses generative AI to generate optimal feedback based on the emotional state recognized by the emotion engine. Specifically, it uses GPT-3 or a similar generative AI model to generate appropriate output such as text, images, audio, and video. The input is the evaluation result of the emotional state, and the output is the generated feedback.

[1336] Step 7:

[1337] The generated feedback data is encrypted and sent back to the terminal from the server. The terminal decrypts the data and stores it in a storage device. The input is the generated feedback data, and the output is the decrypted feedback data.

[1338] Step 8:

[1339] The device then notifies the user of the decoded feedback via a visual device such as smart glasses. For example, if the fatigue level is high, a message such as "We recommend you take a break" is displayed. The input is the decoded feedback data, and the output is a notification message displayed to the user.

[1340] This makes it possible to analyze the user's brain wave data in real time and provide appropriate feedback.

[1341] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1342] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1343] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1344] [Fourth embodiment]

[1345] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1346] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1347] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1348] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1349] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1350] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1351] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1352] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1353] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1354] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1355] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1356] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1357] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1358] This invention is a system that analyzes EEG signals acquired by an EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[1359] EEG data collection and preprocessing

[1360] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device then performs pre-processing on the received EEG signals, such as noise reduction and filtering, to improve signal quality and increase analysis accuracy.

[1361] Sending data to the server

[1362] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[1363] Analysis of EEG data

[1364] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's brainwave data. The analysis results in the user's thoughts and emotional state.

[1365] Realizing output through generative AI

[1366] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. The generative AI uses various generative models (e.g., natural language processing models and image generation models) to create highly accurate output.

[1367] Feedback of results

[1368] Server: The generated output data is formatted into an appropriate format, encrypted again, and sent to the terminal. The terminal decrypts the sent data and saves it to a storage device. The storage device notifies or displays the saved result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[1369] Specific examples

[1370] Example of assistive device operation for physically paralyzed people

[1371] User: A paralyzed person thinks about operating a wheelchair.

[1372] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1373] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." It then uses a generation AI to generate an operation command.

[1374] Storage device: Sends the generated instructions to the wheelchair control system, causing the wheelchair to perform the specified actions.

[1375] Example of concentration monitoring during a meeting

[1376] Users: Business people attending the meeting.

[1377] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1378] Server: Receives and analyzes the data. Based on the analysis results, the user's concentration level is evaluated. If it falls below a certain threshold, the generation AI generates a warning message.

[1379] Storage device: The generated warning message is sent to the device, which notifies the user, for example, by displaying "Your concentration level is decreasing" on the display.

[1380] The processing flow will be explained below.

[1381] Step 1:

[1382] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[1383] Step 2:

[1384] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[1385] Step 3:

[1386] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[1387] Step 4:

[1388] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[1389] Step 5:

[1390] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[1391] Step 6:

[1392] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[1393] Step 7:

[1394] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[1395] Step 8:

[1396] Server: Based on the analysis results, generative AI is used to generate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[1397] Step 9:

[1398] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[1399] Step 10:

[1400] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[1401] Step 11:

[1402] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[1403] As a concrete example, the case of a physically paralyzed person operating a wheelchair is shown below.

[1404] Step 1:

[1405] User: A paralyzed person wears a brainwave sensor on their head to help them think about operating a wheelchair.

[1406] Step 2:

[1407] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[1408] Step 3:

[1409] Terminal: Encrypts the preprocessed data and sends it to the server.

[1410] Step 4:

[1411] Server: Receives the data and extracts features after decoding.

[1412] Step 5:

[1413] Server: A machine learning model analyzes the user's thoughts, and the generation AI generates operation commands (e.g., "go forward," "go backward," "turn left," "turn right").

[1414] Step 6:

[1415] Server: Encrypts and transmits commands to the terminal.

[1416] Step 7:

[1417] Terminal: Decodes commands and sends instructions to the wheelchair control system.

[1418] Step 8:

[1419] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[1420] Through these specific process steps, the invention can be effectively implemented.

[1421] Example 1

[1422] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1423] The present invention aims to provide a system that acquires and analyzes a user's EEG signals in real time to accurately grasp the user's thoughts and emotions and provide appropriate feedback based on the acquired signals. Current systems face challenges in removing noise from EEG signals obtained from non-invasive EEG sensors, securely transmitting data, improving analytical accuracy, and providing feedback in a variety of output formats using generative AI. These challenges must be addressed, for example, when a physically paralyzed person needs to operate an assistive device by thought, or when monitoring a user's concentration level and generating appropriate warnings.

[1424] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1425] In this invention, the server includes means for decoding the EEG data and performing a fast Fourier transform to extract features, means for inputting the features into a machine learning model and analyzing them, and means for generating output such as text, images, audio, and video using a generative AI based on the analysis results. This enables highly accurate analysis based on the user's EEG signals and the generation of appropriate output.

[1426] A "non-invasive EEG sensor" is a sensor that is worn on the user's head to acquire EEG signals in real time, and is a device that can be used without damaging the user's skin.

[1427] "Preprocessing" refers to processing that removes noise and filters the EEG signal to improve the quality of the signal.

[1428] "Noise reduction" is the process of removing unnecessary noise and external interference from EEG signals.

[1429] "Filtering" is a process of extracting specific frequency components from an EEG signal or removing unnecessary frequency components.

[1430] A "secure protocol" is a communication method that ensures confidentiality and integrity of data and is a protocol that has encryption and authentication functions.

[1431] "Encryption" is a process of converting data to prevent the contents of the data being transmitted from being read by a third party.

[1432] "Decryption" is the process of restoring encrypted data to its original state.

[1433] "Feature extraction" is the process of finding the characteristics and patterns necessary for analysis from EEG signals.

[1434] The "Fast Fourier Transform (FFT)" is a mathematical technique for converting time-domain signals into the frequency domain for analysis.

[1435] A "machine learning model" is a model that learns rules and patterns from data and makes predictions and classifications.

[1436] "Generative AI" is a general term for artificial intelligence technology that generates output such as text, images, audio, and video based on analysis results.

[1437] "Text, image, audio, video, etc. output" refers to the form of information generated based on the user's thoughts and feelings.

[1438] "Formatting" is the process of putting data into a specific form.

[1439] "Storage Device" means a device that stores the generated output and communicates or displays it to the user when necessary.

[1440] This invention is a system that analyzes EEG signals acquired using a non-invasive EEG sensor and outputs the results using a generative AI. This system consists of a non-invasive EEG sensor, a terminal that processes the data, a server that runs the data analysis and generative AI, and a storage device that feeds back the output results to the user.

[1441] Hardware and software used

[1442] Brainwave sensor: A non-invasive brainwave sensor is worn on the user's head to capture brainwave signals in real time. This sensor has wireless communication capabilities and transmits the brainwave signals to a device.

[1443] Terminal: The terminal processes the EEG signals received from the sensor. Pre-processing such as noise reduction and filtering is performed on the terminal. Specifically, band-pass filters and artifact removal algorithms are used to make the signals clearer.

[1444] Server: The server receives and decrypts the encrypted EEG data sent from the device. It then extracts features using an FFT (Fast Fourier Transform). The extracted features are input into a machine learning model to analyze the user's EEG data. Based on the analysis results, a generative AI generates output such as text, images, audio, and video. The generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[1445] Storage Device: A storage device is a device that stores the generated output data and provides feedback to the user, for example, by displaying a message on a display or playing an audio message through a speaker.

[1446] Specific examples

[1447] Example of assistive device operation for physically paralyzed people

[1448] User: A paralyzed person thinks about operating a wheelchair.

[1449] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1450] Server: Receives the data and decrypts the encrypted data. Analyzes the decrypted data and identifies the thought content, which is either "go forward," "go backward," "turn left," or "turn right." Then, generates an operation command using a generation AI. The generated command is re-encrypted and sent to the terminal.

[1451] Terminal: Decodes the command and sends it to the wheelchair control system.

[1452] Wheelchair: Follows the received operation command and performs the specified action (forward, backward, etc.).

[1453] Example prompt: "Get thoughts to move the wheelchair forward, then use generative AI to convert those thoughts into commands to control the wheelchair."

[1454] Example of concentration monitoring during a meeting

[1455] User: Business people attending a meeting.

[1456] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1457] Server: Receives data and decrypts the encrypted data. Analyzes the decrypted data and evaluates the user's concentration. If it falls below a certain threshold, the AI ​​generates a warning message. The generated message is re-encrypted and sent to the device.

[1458] On your device: Decrypt the warning message and save it to your storage device.

[1459] Storage device: Reads the saved warning message and notifies the user by displaying "Your concentration level is decreasing" on the display, for example.

[1460] Example prompt: "Collect EEG data to monitor concentration during meetings, and evaluate the concentration level based on the analysis results. Generate a warning message if the concentration level falls below the threshold."

[1461] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1462] Step 1: Collecting EEG data

[1463] User: The user wears a non-invasive EEG sensor on their head, which captures real-time EEG signals and transmits them wirelessly to a device.

[1464] Input: User's EEG signal

[1465] Output: Raw EEG data sent from the sensor to the device

[1466] Step 2: Preprocessing the EEG data

[1467] Device: The device performs noise reduction and filtering on the EEG signals received from the sensor. Specifically, it uses a bandpass filter to extract specific frequency components and an artifact removal algorithm to remove unnecessary components.

[1468] Input: Raw EEG data sent from the sensor

[1469] Output: Pre-processed, high-quality EEG data

[1470] Step 3: Sending data to the server

[1471] Terminal: The pre-processed EEG data is encrypted and sent to the server using a secure protocol (e.g., HTTPS).

[1472] Input: Preprocessed EEG data

[1473] Output: Encrypted EEG data is sent to the server

[1474] Step 4: Decoding EEG data and extracting features

[1475] Server: The server receives the encrypted EEG data, decrypts it, and extracts features from the decrypted data using FFT (Fast Fourier Transform).

[1476] Input: Encrypted EEG data

[1477] Output: Features extracted by FFT

[1478] Step 5: Analyze the EEG data

[1479] Server: The extracted features are input into a machine learning model to analyze the user's thoughts and emotional state. The machine learning model uses SVM, neural networks, etc.

[1480] Input: Extracted features

[1481] Output: Typical analysis results (thoughts and emotional state)

[1482] Step 6: Realizing the output of generative AI

[1483] Server: Based on the analysis results, the generative AI generates output such as text, images, audio, and video. This generative AI uses a natural language processing model (e.g., GPT-3) or an image generation model (e.g., GAN).

[1484] Input: Analysis results

[1485] Output: The generated output, such as text, images, audio, or video.

[1486] Step 7: Encrypt data and send it to your device

[1487] Server: Formats the generated output data into the appropriate format, encrypts it again, and sends it to the device.

[1488] Input: Generated output data

[1489] Output: The encrypted output data.

[1490] Step 8: Decrypt and save the data

[1491] Terminal: Decrypts the received encrypted output data and stores it on a storage device.

[1492] Input: Encrypted output data

[1493] Output: The decoded output data

[1494] Step 9: Feedback on results

[1495] Storage device: Reads the saved output data and notifies or displays it to the user, for example, by displaying a message on the display or playing an audio message through the speaker.

[1496] Input: Decoded output data

[1497] Output: what is displayed or notified to the user

[1498] (Application example 1)

[1499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1500] In conventional brick-and-mortar stores, users had to proactively inquire about products to obtain information and receive appropriate advice. Furthermore, there were few ways to grasp in real time which products users were interested in and what their purchasing intentions were, making it difficult to provide effective customer service. The present invention aims to provide a new solution for analyzing users' brainwave data and improving the shopping experience in brick-and-mortar stores.

[1501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1502] In this invention, the server includes: means for acquiring a user's electroencephalogram (EEG) signal using a non-invasive EEG sensor; means for preprocessing the EEG signal and performing noise reduction and filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing it using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for feeding back the output to the user in the storage device; means for identifying from the EEG data whether the user is interested in a product in a physical store or has a purchase intent; and means for providing the user with relevant product information and advice based on the identified interest and purchase intent. This makes it possible to provide optimal product information and advice in real time based on the user's interest and purchase intent, thereby improving the shopping experience.

[1503] A "non-invasive EEG sensor" is a sensor that can acquire EEG signals simply by wearing it, without physically invading the body.

[1504] "EEG signals" are electrical signals that reflect the user's brain activity.

[1505] "Preprocessing" refers to processing that removes noise and filters the EEG signals to improve the accuracy of analysis.

[1506] "Noise removal" is a process for removing unnecessary components contained in an electroencephalogram signal.

[1507] "Filtering" is a signal processing technique that extracts or suppresses only specific frequency bands.

[1508] A "secure protocol" is a communication protocol that ensures confidentiality and integrity in data communication.

[1509] "Features" are important information extracted from electroencephalogram data for analysis.

[1510] A "machine learning model" is an algorithm that learns patterns based on given data and makes predictions and classifications for future data.

[1511] "Generative AI" is a system that uses artificial intelligence technology to automatically generate output such as text, images, audio, and video.

[1512] A "storage device" is a device that stores data and provides feedback to the user as needed.

[1513] "Interest" refers to the attention or concern a user has towards a particular product.

[1514] "Purchase intent" refers to a user's desire or motivation to purchase a particular product.

[1515] "Product information" is information about the characteristics and features of a product.

[1516] "Advice" means any purchasing advice or recommendation provided to a User.

[1517] A system for implementing the present invention includes the following hardware and software.

[1518] Hardware

[1519] 1. EEG sensor:

[1520] Non-invasive EEG sensors: For example, by wearing specific headgear, the user's brainwave signals are acquired in real time. For example, high-performance EEG sensors can be used to accurately reflect the user's mental state and interests.

[1521] 2. Terminal:

[1522] Smart glasses: Devices equipped with displays and speakers that display information and provide audio feedback to users, allowing them to obtain the information they need in real time while shopping in physical stores.

[1523] This device has the processing power to receive data transmitted from the EEG sensor and perform initial pre-processing.

[1524] software

[1525] 1. Server:

[1526] Data processing and analysis: The server receives the data sent from the EEG sensor and the device, decrypts it, and extracts features from the preprocessed EEG data using algorithms such as FFT (Fast Fourier Transform).

[1527] Machine learning model: The extracted features are input into a machine learning model to analyze the user's brainwave data, which can determine whether the user is interested in a particular product and whether they have the intention to purchase it.

[1528] Generative AI: Based on the analysis results, generative AI generates output such as text, images, audio, and video. For example, it uses natural language processing models and image generation models (e.g., GPT-3 and DALL-E).

[1529] Example

[1530] 1. Data Preprocessing:

[1531] The user wears an EEG sensor, and the EEG signal is transmitted to a device, which then performs noise reduction and filtering to obtain high-quality data.

[1532] 2. Data transmission and analysis:

[1533] The preprocessed EEG data is sent to a server using a secure protocol (e.g., HTTPS). The server decrypts the data and extracts features using algorithms such as FFT. The user's EEG data is then analyzed by a machine learning model to identify the user's thoughts and emotional state.

[1534] 3. Generative AI output:

[1535] Based on the analysis results, generative AI generates output in the form of text, images, audio, video, etc. For example, if a user shows interest in a particular product, detailed information about that product and related product advice will be generated.

[1536] Specific examples from physical stores

[1537] Brainwave sensors: Users wearing high-performance, non-invasive brainwave sensors enter physical stores.

[1538] Smart glasses: When a user expresses interest in a particular product, the smart glasses display details about that product and offer advice on related products, and may also provide audio information through the speakers.

[1539] Example prompt for the generative AI:

[1540] "Users are interested in specific products. Based on this analysis, please provide product details and related products."

[1541] The system will significantly enhance the in-store shopping experience, allowing users to get real-time information and advice on products they are interested in or interested in purchasing.

[1542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1543] Step 1:

[1544] The user wears a non-invasive EEG sensor. The sensor captures the user's EEG signals in real time and transmits them to a device. The input is the user's EEG, and the output is raw EEG data. The device receives this data.

[1545] Step 2:

[1546] The device performs preprocessing on the received EEG signals, specifically noise removal and filtering. The input is raw EEG data, and the output is clear, preprocessed EEG data. This improves the quality of the data and the accuracy of subsequent analysis.

[1547] Step 3:

[1548] The terminal encrypts the preprocessed EEG data and sends it to the server using a secure protocol (e.g., HTTPS). The input is the preprocessed EEG data, and the output is the encrypted EEG data. The server receives this data.

[1549] Step 4:

[1550] The server decrypts the received data and extracts features using algorithms such as FFT (Fast Fourier Transform). The input is encrypted EEG data, and the output is features decomposed into specific frequency components. This makes it easier to analyze the data.

[1551] Step 5:

[1552] The extracted features are input into a machine learning model. The server analyzes this data using the machine learning model to identify the user's interests and purchasing intent. The input is the features, and the output is the identified user's interests and purchasing intent. This analysis reveals the user's state of mind.

[1553] Step 6:

[1554] The server uses a generative AI model based on the analysis results to generate output such as text, images, audio, and video. The input is the analysis results such as the user's interests and purchasing intentions, and the output is the generated content. An example of a prompt for the generative AI model is, "The user is interested in a specific product. Based on this analysis result, please provide product details and related products."

[1555] Step 7:

[1556] The generated output data is formatted into an appropriate format by the server, encrypted again, and then sent to the terminal. The input is the generated content, and the output is the encrypted content data.

[1557] Step 8:

[1558] The device decrypts the transmitted data and provides feedback to the user through the smart glasses' display and speaker. The input is encrypted content data, and the output is information and advice provided to the user. This allows users to obtain information and advice about products of interest in real time.

[1559] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1560] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion engine, and a storage device that provides feedback on the output results to the user.

[1561] EEG data collection and preprocessing

[1562] Device: The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time and transmits them to the device. The device performs pre-processing such as noise reduction and filtering on the received EEG signals to improve the signal quality.

[1563] Sending data to the server

[1564] Terminal: Preprocessed EEG data is encrypted and transmitted to the server using a secure protocol (e.g., HTTPS), which ensures data confidentiality and integrity.

[1565] Analysis of EEG data

[1566] Server: The server receives the data sent from the device and decrypts it. It uses algorithms such as FFT (Fast Fourier Transform) and wavelet transform to extract features from the decrypted data. The extracted features are input into a machine learning model, which analyzes the user's EEG data.

[1567] Emotion recognition by emotion engine

[1568] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses trained models to identify the user's emotions from brainwave patterns and physiological data.

[1569] Realizing output through generative AI

[1570] Server: Based on the emotional state recognized by the emotion engine, the generative AI generates appropriate output such as text, images, audio, and video. The generative AI uses advanced models such as GPT-3 and DALL-E to create highly accurate output.

[1571] Feedback of results

[1572] Server: The generated output data is formatted appropriately, encrypted again, and sent to the device. The device decrypts the data and stores it in a storage device. The storage device notifies or displays the stored result to the user. For example, a text message may be displayed on the screen or an audio message may be played through the speaker.

[1573] Specific examples

[1574] Example of assistive device operation for physically paralyzed people

[1575] User: A paralyzed person thinks about operating a wheelchair.

[1576] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1577] Server: Receives and analyzes the data. From the analysis results, it determines whether the user is thinking "go forward," "go backward," "turn left," or "turn right." At the same time, the emotion engine recognizes the user's emotional state. Then, it uses generative AI to generate operational commands and feedback based on the user's emotional state.

[1578] Storage device: Sends the generated command to the wheelchair control system, which then executes the specified action. At the same time, the wheelchair provides feedback based on the action results and the user's emotions.

[1579] Example of concentration monitoring during a meeting

[1580] Users: Business people attending the meeting.

[1581] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1582] Server: Receives and analyzes the data. Based on the analysis results, the user's level of concentration is evaluated. If it falls below a certain threshold, the emotion engine recognizes the user's emotion, and the generative AI generates optimal feedback.

[1583] Storage device: Generated alerts based on the concentration level and emotions are sent to the device, which then notifies the user. For example, the device may display "Your concentration is declining" on the display, along with advice on how to reduce stress.

[1584] In this way, the system of the present invention analyzes the user's EEG data and generates appropriate feedback based on the emotional state recognized by the emotion engine, thereby enriching the user's experience and supporting efficient communication and work performance.

[1585] The processing flow will be explained below.

[1586] Step 1:

[1587] User: The user wears a non-invasive EEG sensor on their head, which captures the user's brainwave signals in real time.

[1588] Step 2:

[1589] Terminal: The acquired EEG signal is sent to the terminal, which performs pre-processing, including noise reduction and filtering, to improve the signal quality.

[1590] Step 3:

[1591] Terminal: Pre-processed EEG data is encrypted using a strong encryption algorithm to ensure data security.

[1592] Step 4:

[1593] Terminal: Sends encrypted data to the server using a secure protocol (e.g., HTTPS).

[1594] Step 5:

[1595] Server: Receives the data sent from the device and decrypts it, returning it to a format that can be analyzed.

[1596] Step 6:

[1597] Server: Extracts features from EEG data using algorithms such as FFT (Fast Fourier Transform) and wavelet transform.

[1598] Step 7:

[1599] Server: The extracted features are input into a machine learning model, such as a trained neural network, to analyze the user's thoughts and emotional state.

[1600] Step 8:

[1601] Server: Inputs the analysis results into the emotion engine to recognize the user's emotional state. The emotion engine uses algorithms to identify emotions from brainwave patterns and other physiological data.

[1602] Step 9:

[1603] Server: Based on the emotional state recognized by the emotion engine, generative AI is used to generate appropriate output such as text, images, audio, and video. Generative AI uses advanced models such as GPT-3 and DALL-E.

[1604] Step 10:

[1605] Server: The generated output data is formatted appropriately, re-encrypted, and sent to the device using a secure protocol.

[1606] Step 11:

[1607] Terminal: Decrypts received data and prepares it for display or notification. It is also stored on the storage device.

[1608] Step 12:

[1609] Terminal: Provides generated feedback to the user, for example by displaying text on a display or playing an audio message.

[1610] Example: Operating assistive devices for physically paralyzed people

[1611] Step 1:

[1612] User: A paralyzed person wears an EEG sensor on their head.

[1613] Step 2:

[1614] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[1615] Step 3:

[1616] Terminal: Encrypts the pre-processed data and sends it to the server.

[1617] Step 4:

[1618] Server: Receives the data and extracts features after decoding.

[1619] Step 5:

[1620] Server: A machine learning model analyzes the user's thoughts and generates operational commands such as "go forward" or "go backward." At the same time, an emotion engine recognizes the user's emotional state.

[1621] Step 6:

[1622] Server: The generation AI generates feedback based on the control command and emotional state.

[1623] Step 7:

[1624] Server: Encrypts the generated commands and sends them to the device using a secure protocol.

[1625] Step 8:

[1626] Terminal: Decodes the received commands and sends them to the wheelchair control system.

[1627] Step 9:

[1628] Terminal: The wheelchair performs the specified action and provides feedback to the user (e.g., displays a success message on the screen).

[1629] Example: Monitoring concentration during a meeting

[1630] Step 1:

[1631] User: Business people participating in a meeting wear brainwave sensors.

[1632] Step 2:

[1633] Terminal: Acquires EEG signals from the EEG sensor and performs preprocessing.

[1634] Step 3:

[1635] Terminal: Encrypts the pre-processed data and sends it to the server.

[1636] Step 4:

[1637] Server: Receives the data and extracts features after decoding.

[1638] Step 5:

[1639] Server: Analyzes and evaluates the user's concentration level using a machine learning model, while simultaneously recognizing their emotional state using an emotion engine.

[1640] Step 6:

[1641] Server: When the threshold is exceeded, the AI ​​generates an appropriate warning message and alert based on the emotion.

[1642] Step 7:

[1643] Server: Encrypts the generated message and sends it to the device using a secure protocol.

[1644] Step 8:

[1645] Device: Decodes the received message and notifies the user, for example by displaying "Your concentration is declining" on the display and providing emotion-based stress reduction advice.

[1646] Example 2

[1647] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1648] Existing EEG analysis systems lack a unified process from acquiring EEG data to recognizing emotions and providing feedback to the user, making it difficult to accurately and quickly reflect the user's emotional state. Furthermore, for specific applications such as assisting physically paralyzed people or monitoring concentration levels during meetings, there was a lack of a system that provided appropriate feedback in real time according to user needs, making it difficult to improve user experience and work efficiency.

[1649] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1650] In this invention, the server includes means for extracting features from the EEG data and analyzing them using a machine learning model, means for recognizing the user's emotional state using an emotion recognition engine based on the analysis results, and means for generating output such as text, images, audio, and video using a generative AI model based on the recognized emotional state. This enables emotion recognition and the generation of appropriate feedback to be performed quickly and accurately based on the user's EEG data, making it possible to improve user experience and business efficiency in specific applications.

[1651] A "non-invasive EEG sensor" is a sensor worn on the head to acquire the user's brainwave signals, but collects data without invading the body.

[1652] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[1653] A "secure protocol" is a communication protocol that uses encryption technology to securely transmit data to another party in order to ensure confidentiality and integrity of the data.

[1654] "Features" are important data points or indicators extracted from EEG data and used for analysis.

[1655] A "machine learning model" is an algorithm or method used to analyze EEG data, primarily for pattern recognition, classification, and regression of data.

[1656] An "emotion recognition engine" is a model trained to identify and recognize a user's emotional state by analyzing EEG data and other physiological data.

[1657] A "generative AI model" is an AI model that generates output such as text, images, audio, or video based on a recognized emotional state. Representative examples include GPT-3 and DALL-E.

[1658] "Storage Device" means a device used to provide generated output back to the user, for storing and displaying data.

[1659] This invention is a system that analyzes EEG signals acquired by an EEG sensor and recognizes a user's emotions using an emotion recognition engine. This system consists of a non-invasive EEG sensor, a terminal that processes data and provides feedback, a server that runs data analysis and generation AI, the emotion recognition engine, and a storage device that provides feedback on the output results to the user.

[1660] Hardware and software used

[1661] User: The user wears a non-invasive EEG sensor on their head, which converts electrical activity in the brain into digital signals and transmits them to a device in real time.

[1662] Terminal: Receives EEG signals sent from the EEG sensor. The terminal is equipped with software to remove noise and filter the received signals, and to perform preprocessing. This mainly involves band-pass filtering and detrending.

[1663] Server: The server receives the preprocessed EEG data via an encrypted protocol (e.g., HTTPS). The server extracts features using FFT (Fast Fourier Transform) or wavelet transform, and analyzes the data using a machine learning model. Machine learning libraries such as Python and TensorFlow are used here.

[1664] Emotion Recognition Engine: Recognizes the user's emotions based on the analysis of EEG data. The emotion recognition engine uses algorithms such as neural networks and support vector machines to identify emotions based on pre-trained models and analyzing EEG patterns and physiological data.

[1665] Generative AI model: Based on the emotional state recognized by the emotion recognition engine, the generative AI generates output such as text, images, audio, and video. For example, advanced generative AI models such as GPT-3 and DALL-E are used, which generate highly accurate output.

[1666] Storage device: Used to feed back the generated output data to the user. The data is encrypted again and sent to the terminal, where it is decrypted and stored in the storage device. The generated text or voice message is notified to the user through devices such as a display or speaker.

[1667] Specific examples

[1668] Example of assistive device operation for physically paralyzed people

[1669] User: A paralyzed person thinks about operating a wheelchair, for example, "Go forward."

[1670] Terminal: The EEG sensor acquires the user's EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1671] Server: Receives and analyzes the data. From the analysis results, it determines that a "move forward" command has been issued. At the same time, it uses an emotion recognition engine to recognize the user's emotional state. It then uses a generative AI to generate feedback based on the operation command and emotion. A message such as "The wheelchair will move forward. Please relax" is generated.

[1672] Storage device: Sends the generated commands to the wheelchair control system, causing the wheelchair to perform the specified actions, while simultaneously displaying feedback messages to the user.

[1673] Example of concentration monitoring during a meeting

[1674] Users: Business people attending the meeting.

[1675] Terminal: EEG sensor acquires EEG data, performs preprocessing, and sends the preprocessed data to the server.

[1676] Server: Receives and analyzes the data. The level of concentration is evaluated based on the analysis results, and if it falls below a certain threshold, the emotion recognition engine recognizes the user's emotions. An alert message such as "Your concentration is declining. Take a deep breath and refresh yourself" is generated.

[1677] Storage device: Sends the generated alert message to the device, which notifies the user.

[1678] Prompt Sentence Examples

[1679] "Recommend music that users listen to when they are in a relaxed state."

[1680] "Generate an alert message when someone loses focus during a meeting."

[1681] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1682] Step 1:

[1683] User: Wears a non-invasive EEG sensor on the head, which measures the user's brainwave signals in real time and transmits the digital signal to the device.

[1684] Input: EEG signal

[1685] Output: Digital signal

[1686] Step 2:

[1687] Terminal: Pre-processing the received digital signal, specifically applying a band-pass filter to remove high frequency noise, and detrending to remove baseline drift.

[1688] Input: Digital signal

[1689] Output: Preprocessed EEG data

[1690] Step 3:

[1691] Terminal: Encrypt the pre-processed EEG data. Send the encrypted data to the server using a secure protocol (e.g., HTTPS).

[1692] Input: Preprocessed EEG data

[1693] Output: Encrypted EEG data

[1694] Step 4:

[1695] Server: Receives and decrypts the encrypted data sent from the device. Then, it extracts frequency components using FFT (Fast Fourier Transform) and obtains features with high time-frequency resolution using wavelet transform.

[1696] Input: Encrypted EEG data

[1697] Output: Feature data

[1698] Step 5:

[1699] Server: The feature data is input into a machine learning model for analysis. Here, analysis is performed using support vector machines and neural networks, and the user's EEG data is analyzed.

[1700] Input: Feature data

[1701] Output: Analysis results of EEG data

[1702] Step 6:

[1703] Server: The analysis results are input into the emotion recognition engine to recognize the user's emotional state. The emotion recognition engine analyzes the user's brainwave patterns based on a pre-trained dataset to identify emotions.

[1704] Input: Analysis results

[1705] Output: User's emotional state

[1706] Step 7:

[1707] Server: Based on the recognized emotional state, a generative AI model (e.g., GPT-3 or DALL-E) is used to generate output data (text, images, audio, video, etc.). For example, a prompt such as "Please recommend music that the user would listen to when in a relaxed state" is input into the generative AI, and output data is generated.

[1708] Input: User's emotional state

[1709] Output: The generated output data

[1710] Step 8:

[1711] Server: The generated output data is re-encrypted and sent to the terminal.

[1712] Input: Generated output data

[1713] Output: The encrypted output data.

[1714] Step 9:

[1715] Terminal: Decrypts the transmitted encrypted data, stores it on a storage device, and notifies the user of generated text and voice messages using devices such as a display and speaker.

[1716] Input: Encrypted output data

[1717] Output: The decoded output data

[1718] These are the specific processing steps of the program. Each step forms a series of steps based on the user's EEG data: data acquisition, preprocessing, analysis, generation, and feedback.

[1719] (Application example 2)

[1720] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1721] Conventional non-invasive EEG sensor systems have the technology to acquire users' EEG signals and provide feedback based on the analysis results, but they have not yet fully developed mechanisms for evaluating fatigue and concentration levels in real time and providing appropriate feedback based on that evaluation. Furthermore, they lack the functionality to link with visual devices worn by workers and manage the data in real time under specific work environments. This has made it difficult to efficiently manage worker health and safety and improve labor productivity.

[1722] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring a user's EEG signal using a non-invasive EEG sensor; means for preprocessing the EEG signal to remove noise and perform filtering; means for transmitting the preprocessed EEG data to the server via a secure protocol; means for extracting features from the EEG data and analyzing them using a machine learning model in the server; means for generating output such as text, images, audio, and video using a generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for providing feedback of the output to the user in the storage device; and means for evaluating a worker's fatigue level and concentration level in real time in cooperation with a visual device worn by the worker and providing appropriate feedback. This makes it possible to evaluate a worker's fatigue level and concentration level in real time, thereby improving work efficiency while managing their health and safety.

[1723] A "non-invasive brainwave sensor" is a device that acquires a user's brainwave signals without invading the skin.

[1724] "User" refers to a person who operates or uses the system.

[1725] "Electroencephalogram signals" refer to changes in electrical potential generated by electrical activity in the brain.

[1726] "Preprocessing" refers to processing that removes noise and filters the acquired electroencephalogram signals to improve the quality of the signals.

[1727] "Noise removal" is a process for removing unnecessary noise contained in the electroencephalogram signal.

[1728] "Filtering" is the process of conditioning a signal by passing only specific frequency bands.

[1729] A "secure protocol" is a secure communication protocol that ensures confidentiality and integrity of communication data.

[1730] A "server" is a central processing unit for analyzing and processing data.

[1731] "Features" refer to important data characteristics extracted from EEG signals.

[1732] A "machine learning model" is an algorithm that learns patterns based on data and makes predictions and classifications.

[1733] "Analysis" is a process for obtaining detailed information based on the acquired data.

[1734] "Generative AI" refers to the ability to generate output such as text, images, audio, and video using artificial intelligence technology.

[1735] "Storage device" refers to a device for storing data.

[1736] "Feedback" refers to returning analysis results or generated output to the user.

[1737] "Visual device" refers to a device worn by a user that provides information visually.

[1738] "Fatigue level" is an index for evaluating the degree of fatigue felt by the user.

[1739] "Level of concentration" is an index for evaluating the level of concentration of a user.

[1740] The present invention provides a system for acquiring a user's electroencephalogram (EEG) signal using a non-invasive electroencephalogram sensor, analyzing the signal, and providing appropriate feedback in real time. Specific embodiments of the present invention will be described in detail below.

[1741] System program description

[1742] EEG data collection and preprocessing

[1743] The user wears a non-invasive EEG sensor on their head, which captures EEG signals in real time and performs noise reduction and filtering. After pre-processing, the data is encrypted and sent to a server via a secure protocol. This process ensures data quality and confidentiality.

[1744] Sending data to the server and analyzing it

[1745] The server decodes the received preprocessed EEG data and uses algorithms such as fast Fourier transform (FFT) and wavelet transform to extract features, which are then input into a machine learning model to analyze the EEG data.

[1746] Emotion recognition by emotion engine

[1747] Based on the analysis results, the server recognizes the user's emotional state using an emotion engine, which uses a trained model to identify the user's emotional state from numerous brainwave patterns and physiological data.

[1748] Feedback generation by generative AI

[1749] Based on the emotional state recognized by the emotion engine, the server uses generative AI to create appropriate feedback, using, for example, advanced text and image generation models to generate highly accurate output.

[1750] Feedback of results

[1751] The generated feedback data is then encrypted again and sent via a secure protocol to the device, which decrypts the data and presents it to the user via a visual device (such as smart glasses).

[1752] Hardware and software used

[1753] Brainwave sensors: Non-invasive devices such as Emotiv and Muse.

[1754] Server: A central processing unit for running data analysis and generative AI models, using Python and libraries such as SciPy and Sklearn.

[1755] Visual devices: smart glasses (e.g., Google Glass, Microsoft HoloLens).

[1756] Communication protocol: Encrypted communication (e.g. HTTPS).

[1757] Specific examples

[1758] Application example: Factory worker monitoring

[1759] Factory workers wear smart glasses, and their brainwaves are monitored in real time using an EEG sensor. If a worker continues working for an extended period of time, data is collected to assess their level of fatigue and concentration. Based on the analysis results, the server uses a generative AI to provide appropriate feedback. For example, if the worker's level of fatigue is high, the smart glasses will display a message saying, "We recommend taking a break."

[1760] Example prompt sentence:

[1761] "Workers working long hours on an assembly line were monitored for brainwaves to assess their fatigue levels. Appropriate feedback was then generated for the workers."

[1762] As such, the present invention is a system that analyzes the user's brainwave data in real time and provides appropriate feedback using an emotion engine and generative AI, thereby improving work efficiency while managing the health and safety of workers.

[1763] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1764] Step 1:

[1765] The user wears a non-invasive EEG sensor on their head. The sensor captures the user's EEG signals in real time. The captured EEG signals are sent to a terminal for noise reduction and filtering. The input is the raw EEG signal, and the output is pre-processed clean EEG data.

[1766] Step 2:

[1767] The device performs noise reduction and bandpass filtering on the received EEG signal. Specifically, it uses a Butterworth filter to pass only the frequency band between 0.5 Hz and 30 Hz. This processing improves the signal quality and removes unnecessary noise. The input is the raw EEG signal, and the output is preprocessed EEG data.

[1768] Step 3:

[1769] The preprocessed EEG data is encrypted and sent from the device to the server using the HTTPS protocol. The server decrypts the received data and prepares it for analysis. The input is the preprocessed EEG data, and the output is the decrypted EEG data.

[1770] Step 4:

[1771] The server extracts features from the decoded EEG data. Specifically, it uses FFT (Fast Fourier Transform) and wavelet transform to separate different frequency components of the EEG signal. The extracted features are input into a machine learning model for further analysis. The input is the decoded EEG data, and the output is the extracted features.

[1772] Step 5:

[1773] The server inputs the extracted features into a machine learning model and analyzes the EEG data. The analysis results are processed by an emotion engine to identify the appropriate emotional state based on information obtained from the user's EEG patterns. The input is feature data, and the output is an evaluation of the emotional state.

[1774] Step 6:

[1775] The server uses generative AI to generate optimal feedback based on the emotional state recognized by the emotion engine. Specifically, it uses GPT-3 or a similar generative AI model to generate appropriate output such as text, images, audio, and video. The input is the evaluation result of the emotional state, and the output is the generated feedback.

[1776] Step 7:

[1777] The generated feedback data is encrypted and sent back to the terminal from the server. The terminal decrypts the data and stores it in a storage device. The input is the generated feedback data, and the output is the decrypted feedback data.

[1778] Step 8:

[1779] The device then notifies the user of the decoded feedback via a visual device such as smart glasses. For example, if the fatigue level is high, a message such as "We recommend you take a break" is displayed. The input is the decoded feedback data, and the output is a notification message displayed to the user.

[1780] This makes it possible to analyze the user's brain wave data in real time and provide appropriate feedback.

[1781] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1782] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1783] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1784] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1785] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1786] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1787] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1788] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1789] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1790] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1791] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1792] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1793] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1794] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1795] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1796] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1797] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1798] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1799] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1800] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1801] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1802] The following is further disclosed regarding the above embodiment.

[1803] (Claim 1)

[1804] means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor;

[1805] means for pre-processing, denoising and filtering said EEG signals;

[1806] means for transmitting the preprocessed EEG data to a server via a secure protocol;

[1807] A server extracts features from the electroencephalogram data and analyzes them using a machine learning model;

[1808] A means of generating output such as text, images, audio, and video using generative AI based on the analysis results;

[1809] means for transmitting the generated output from the server to a storage device;

[1810] means for feeding back said output to a user in said storage device;

[1811] A system including:

[1812] (Claim 2)

[1813] 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

[1814] (Claim 3)

[1815] 10. The system of claim 1, further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold.

[1816] "Example 1"

[1817] (Claim 1)

[1818] means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor;

[1819] means for pre-processing, denoising and filtering said EEG signals;

[1820] means for encrypting the preprocessed EEG data and transmitting it to a server via a secure protocol;

[1821] a means for performing a fast Fourier transform in a server to decode the electroencephalogram data and extract features;

[1822] a means for inputting the feature amount into a machine learning model and analyzing the model;

[1823] A means of generating output such as text, images, audio, and video using generative AI based on the analysis results;

[1824] means for formatting the generated output data into an appropriate format, encrypting it, and transmitting it from the server to the terminal;

[1825] means for decrypting the received data in the terminal and storing it in a storage device;

[1826] means for notifying or displaying said output to a user in said storage device;

[1827] A system including:

[1828] (Claim 2)

[1829] 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

[1830] (Claim 3)

[1831] 10. The system of claim 1, further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold.

[1832] "Application Example 1"

[1833] (Claim 1)

[1834] means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor;

[1835] means for pre-processing, denoising and filtering said EEG signals;

[1836] means for transmitting the preprocessed EEG data to a server via a secure protocol;

[1837] A server extracts features from the electroencephalogram data and analyzes them using a machine learning model;

[1838] A means of generating output such as text, images, audio, and video using generative AI based on the analysis results;

[1839] means for transmitting the generated output from the server to a storage device;

[1840] means for feeding back said output to a user in said storage device;

[1841] A means of identifying whether a user is interested in a product in a physical store or has a desire to purchase it from brain wave data;

[1842] A means of providing users with relevant product information and advice based on their identified interests and purchasing intent;

[1843] A system including:

[1844] (Claim 2)

[1845] 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

[1846] (Claim 3)

[1847] 10. The system of claim 1, further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold.

[1848] "Example 2: Combining Emotion Engines"

[1849] (Claim 1)

[1850] means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor;

[1851] means for pre-processing, denoising and filtering said EEG signals;

[1852] means for transmitting the preprocessed EEG data to a server via a secure protocol;

[1853] A server extracts features from the electroencephalogram data and analyzes them using a machine learning model;

[1854] A means for recognizing the emotional state of the user using an emotion recognition engine based on the analysis results;

[1855] means for generating output, such as text, images, audio, or video, using a generative AI model based on the recognized emotional state;

[1856] means for transmitting the generated output from the server to a storage device;

[1857] means for feeding back said output to a user in said storage device;

[1858] A system including:

[1859] (Claim 2)

[1860] 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

[1861] (Claim 3)

[1862] 10. The system of claim 1, further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold.

[1863] "Application example 2 when combining emotion engines"

[1864] (Claim 1)

[1865] means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor;

[1866] means for pre-processing, denoising and filtering said EEG signals;

[1867] means for transmitting the preprocessed EEG data to a server via a secure protocol;

[1868] A server extracts features from the electroencephalogram data and analyzes them using a machine learning model;

[1869] A means of generating output such as text, images, audio, and video using generative AI based on the analysis results;

[1870] means for transmitting the generated output from the server to a storage device;

[1871] means for feeding back said output to a user in said storage device;

[1872] A means for assessing fatigue and concentration levels in real time in conjunction with visual devices worn by workers and providing appropriate feedback;

[1873] A system including:

[1874] (Claim 2)

[1875] 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

[1876] (Claim 3)

[1877] 10. The system of claim 1, further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold. [Explanation of symbols]

[1878] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring an electroencephalogram signal from the user using a non-invasive electroencephalogram sensor; means for pre-processing, denoising and filtering said EEG signals; means for transmitting the preprocessed EEG data to a server via a secure protocol; A server extracts features from the electroencephalogram data and analyzes them using a machine learning model; A means of generating output such as text, images, audio, and video using generative AI based on the analysis results; means for transmitting the generated output from the server to a storage device; means for feeding back said output to a user in said storage device; A system including:

2. 10. The system of claim 1, further comprising means for enabling a paralyzed person to operate the assistive device by thought.

3. The system of claim 1 , further comprising means for monitoring a user's concentration and generating an alert if the concentration falls below a threshold.

Images