System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring brain wave data in real time, a means for pre-processing the acquired brain wave data, a means for analyzing the pre-processed brain wave data and extracting a feature amount, a means for summarizing an idea on the basis of the extracted feature amount, a means for converting the summarized content into a natural language, and a means for outputting a sentence of the converted natural language.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] We will propose the ``problem that the invention aims to solve'' and ``means for solving the problem.''

[0005] With conventional technologies, when users try to verbalize their ideas, it is difficult to effectively summarize the ideas and convert them into clear sentences. Furthermore, communication with users who are physically disabled and cannot move their mouths or speak is restricted. This can lead to problems such as users being unable to express their thoughts properly, resulting in a decline in the quality of communication. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention provides a system including means for acquiring electroencephalogram data in real time, means for preprocessing the acquired electroencephalogram data, means for analyzing the preprocessed electroencephalogram data to extract features, means for summarizing thoughts based on the extracted features, means for converting the summarized content into natural language, and means for outputting sentences in the converted natural language. This system enables a user to clearly and effectively put their ideas into sentences, and is also useful as a means of communication with users with physical limitations.

[0007] Understood. Below are definitions of important terms contained in the claims.

[0008] "Electroencephalogram data" is a recording of the electrical activity of a user's brain obtained by an electroencephalogram sensor device.

[0009] "Real-time" means that data is acquired and processed with very little delay from the moment it is generated.

[0010] "Preprocessing" is a process for preparing data in a form suitable for analysis, and is a procedure that includes noise removal, filtering, standardization, etc.

[0011] A "feature" is an important indicator used in the context of data analysis and machine learning, and is a specific pattern or attribute extracted from EEG data.

[0012] A "summary" is a concise summary of the essence of the data or information obtained.

[0013] A "natural language" is a language that humans use on a daily basis, and is particularly a language that is constructed according to grammar and context.

[0014] A "deep neural network" is a machine learning model with multiple layers of artificial neurons, and is an algorithm used for complex pattern recognition.

[0015] "Noise reduction" refers to the removal of unwanted signals and interference from data.

[0016] "Filtering" is the process of enhancing or attenuating signals in a particular frequency band.

[0017] "Standardization" means converting data according to certain standards and preparing it in a format suitable for analysis.

[0018] A "sensor device" is a device that measures a physical quantity and outputs it as an electrical signal. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] Understood. Below is the "Mode for Carrying Out the Invention" from the patent specification.

[0041] This invention is a system that uses electroencephalogram (EEG) data to clearly summarize and verbalize a user's thoughts. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via a terminal.

[0042] System Role

[0043] 1. Users

[0044] Users provide brainwave data to the system by wearing a non-invasive EEG sensor device that does not interfere with the user's daily activities.

[0045] 2. Terminal

[0046] The terminal collects EEG data from the sensor device and transmits it to the server. The terminal maintains a secure connection necessary for data transfer and can also temporarily store EEG data, preventing data loss.

[0047] 3. Server

[0048] The server receives the EEG data sent from the device and preprocesses it. Preprocessing includes noise removal, filtering, and standardization. The server then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. Based on these features, the server summarizes the user's thoughts and converts the summary into natural language. The final text is then sent to the device and displayed to the user.

[0049] Program processing explanation

[0050] First, the server preprocesses the EEG data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is fed into an AI model, such as a deep neural network, to analyze EEG patterns and extract features. Based on the extracted features, the server applies a generative model to summarize the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques. Finally, the server sends the generated sentences to the device and displays them to the user.

[0051] Specific examples

[0052] Example 1: Summary of user ideas

[0053] When a user is thinking of an idea for a new application, a sensor device captures their EEG data, which the device sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, the server generates a summary: "I would like a new app with a feature to manage the user's diet." The generated sentence is sent to the device and displayed to the user.

[0054] Example 2: Communicating with users with physical disabilities

[0055] When a physically disabled user is thinking about their health, the sensor device captures EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0056] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to obtain the user's thoughts in real time and clearly verbalize them.

[0057] The processing flow will be explained below.

[0058] Understood. Below is a step-by-step explanation of how the program works.

[0059] Step 1:

[0060] The user wears a specially designed EEG sensor device, which detects electrical activity in the brain and generates EEG data.

[0061] Step 2:

[0062] The device collects EEG data in real time from the sensor device, performs initial temporary storage, and then transmits the collected data to a server via a secure connection.

[0063] Step 3:

[0064] The server preprocesses the received EEG data. First, it applies a noise reduction algorithm to remove unwanted noise from the data. Then it applies filtering to emphasize specific frequency bands. Finally, it standardizes the data and converts it into a format suitable for analysis.

[0065] Step 4:

[0066] The server analyzes the preprocessed EEG data, using AI models such as deep neural networks to analyze EEG patterns and extract features.

[0067] Step 5:

[0068] The server then runs a generative model for summarizing thoughts based on the extracted features. This model generates candidate sentences that summarize thoughts by referencing previously collected data and common patterns.

[0069] Step 6:

[0070] The server converts the generated summary into natural language, using natural language processing (NLP) techniques to format it into grammatically and contextually appropriate sentences that are easy for users to understand.

[0071] Step 7:

[0072] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[0073] Step 8:

[0074] The terminal displays the received sentence to the user, allowing the user to visually check the generated sentence and provide feedback as needed.

[0075] The above are the specific processing steps of the program in the system.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] Current technology has difficulty capturing a user's thoughts in real time and clearly verbalizing them. Furthermore, there is a lack of effective means for preprocessing EEG data and interpreting it using deep learning models. Therefore, there is a need to accurately analyze a user's EEG data and express it in natural language.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes means for preprocessing the transferred EEG data, means for inputting the preprocessed EEG data into a deep learning model to extract features, and means for summarizing thoughts using a generative AI model based on the extracted features. This makes it possible to preprocess the user's EEG data in real time, analyze it with the deep learning model, and further summarize thoughts using the generative AI model and convert them into natural language.

[0081] "User" refers to an individual who wears an EEG sensor device and provides EEG data.

[0082] "EEG sensor device" refers to equipment for acquiring a user's brain waves in real time in a non-invasive manner.

[0083] "Terminal" refers to a device that temporarily stores data obtained from an EEG sensor device and transfers it to a server.

[0084] "Server" refers to a computing device that receives the electroencephalogram data transmitted from the terminal and performs preprocessing and analysis.

[0085] "Preprocessing" refers to the process of removing noise from EEG data, filtering, and standardizing the data.

[0086] A "deep learning model" refers to a multi-layer neural network model for analyzing EEG data and extracting features.

[0087] A "generative AI model" refers to an artificial intelligence model that summarizes a user's thoughts based on extracted features.

[0088] "Natural language processing technology" refers to technology that converts summarized content into grammatically and contextually appropriate natural language.

[0089] "Features" refer to the characteristics and patterns of data extracted from EEG data.

[0090] A "summary" refers to information that succinctly summarizes the user's thoughts based on extracted features.

[0091] "Real-time" means that the processing from data acquisition to final output is carried out with almost no delay while the user is wearing the EEG sensor device.

[0092] The present invention is a system that acquires a user's brainwave data in real time, preprocesses it, analyzes it using a deep learning model, and then summarizes their thoughts using a generative AI model and converts them into natural language. This system is composed of the following means.

[0093] 1. Acquisition of EEG data

[0094] The user wears a dedicated brainwave sensor device to acquire brainwave data in real time. This brainwave sensor device is non-invasive and does not interfere with the user's daily life. For example, a commercially available brainwave sensor device called an "electroencephalograph" can be used. This device detects the user's brainwaves with a sensor, converts them into signals, and transmits them to a terminal.

[0095] 2. Data transfer and temporary storage

[0096] The device temporarily stores the data acquired from the EEG sensor device and then transmits it to the server. A database (e.g., SQLite) is used for storage, and a secure connection (e.g., HTTPS) is used for communication with the server. This prevents data loss and information leakage.

[0097] 3. Data Preprocessing

[0098] The server receives the EEG data transferred from the device and first performs preprocessing. This includes noise removal, filtering, and data standardization. The preprocessing uses libraries such as SciPy for noise removal, and NumPy for filtering and standardization. This eliminates signal distortion and external interference and scales the data uniformly.

[0099] 4. Feature extraction using deep learning models

[0100] The preprocessed data is then fed into a deep learning model such as TensorFlow to analyze the EEG patterns and extract features that accurately capture the thoughts occurring in the user's brain.

[0101] 5. Summary Generation Using Generative AI Models

[0102] The server uses a generative AI model (e.g., GPT-3) to summarize the user's thoughts based on the extracted features. This summary is a concise representation of the user's thoughts and plays an important role as the final output of the entire system.

[0103] 6. Natural Language Translation

[0104] The summarized content is then converted into grammatically and contextually appropriate natural language using natural language processing (NLP) techniques, for example, using an NLP library such as spaCy.

[0105] 7. Displaying the results

[0106] The final generated natural language sentence is sent from the server to the device and displayed to the user, which can be a mobile app or a web browser.

[0107] Examples of concrete examples and prompts

[0108] Example 1: Summary of user ideas

[0109] When a user is thinking of an idea for a new application, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, a generative AI model is used to generate a summary such as, "I would like a new app with a feature to manage the user's diet." This summary is then sent to the device and displayed to the user.

[0110] plaintext

[0111] If a user has an idea for a new app:

[0112] [EEG data]

[0113] Summarize what users think.

[0114] Example 2: Communicating with users with physical disabilities

[0115] When a physically disabled user is thinking about their health, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural language and sent to the device for display to the user.

[0116] plaintext

[0117] If you are a physically challenged user and are thinking about your health:

[0118] [EEG data]

[0119] Summarize your thoughts about the user's health.

[0120] The above is an embodiment of the present invention, and this system makes it possible to acquire the user's thoughts in real time and clearly verbalize them.

[0121] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0122] Step 1: The user wears the EEG sensor device

[0123] The user wears the EEG sensor device and begins collecting EEG data. The device is non-invasive and does not interfere with the user's daily activities. The input is the user's EEG, and the output is EEG data converted into a digital signal. Specifically, the sensor device measures EEG in real time, converts the signal into digital data, and transmits it to a terminal.

[0124] Step 2: The device temporarily stores the EEG data.

[0125] The device receives the acquired EEG data and temporarily stores it. This storage is done using an SQLite database on the edge device. The input is the EEG data sent from the sensor device, and the output is the temporarily stored data. Specifically, the device receives the data in real time and writes it to the database.

[0126] Step 3: The device sends the EEG data to the server

[0127] The device sends the stored EEG data to the server. A secure connection (e.g., HTTPS) is used for communication with the server. The input is the temporarily stored EEG data, and the output is the data sent to the server. Specifically, the device reads the data, encrypts it, and sends it to the server via an HTTP request.

[0128] Step 4: The server receives and preprocesses the data

[0129] The server receives the EEG data sent from the device and begins preprocessing. Preprocessing includes noise removal, filtering, and data standardization. The input is the received raw EEG data, and the output is the preprocessed data. Specifically, the server removes noise using the SciPy library and filters and standardizes the data using the NumPy library.

[0130] Step 5: The server extracts features using a deep learning model

[0131] Using the preprocessed data, the server extracts features using a deep learning model (for example, a model using TensorFlow). The input is the preprocessed data, and the output is the extracted features. Specifically, the server inputs the data into the model, analyzes the EEG patterns through a neural network, and extracts important features.

[0132] Step 6: The server summarizes the thoughts using a generative AI model

[0133] The server inputs the extracted features into a generative AI model (e.g., GPT-3) to summarize the thoughts. The input is the extracted features, and the output is a summary of the thoughts. Specifically, the server inputs the prompt sentence and features into the generative AI model, and the model generates a summary.

[0134] Step 7: The server converts the summary into natural language

[0135] The server converts the generated summary into natural language using natural language processing technology (e.g., spaCy). The input is the summarized content, and the output is a natural language sentence. Specifically, the server inputs the summary into an NLP library and converts it into a grammatically and contextually natural sentence.

[0136] Step 8: The server sends the text to the device

[0137] The server sends the generated natural language text to the terminal. The input is the natural language text, and the output is the text sent to the terminal. In concrete terms, the server sends the generated text to the terminal as an HTTP response.

[0138] Step 9: The terminal displays the text to the user

[0139] The terminal displays the received natural language text to the user. The input is the natural language text sent from the server, and the output is the text displayed to the user. In concrete terms, the terminal displays the text on the screen so that the user can immediately check it.

[0140] (Application example 1)

[0141] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] Existing systems that use EEG data can extract and verbalize a user's thoughts, but they lack the functionality to further apply this information and transmit it to a robot as work instructions to automatically control its movements. Therefore, there is a need, particularly in industrial sites and factories, to directly reflect the consciousness of workers in work instructions to improve efficiency and safety. Currently, workers' instructions must be entered manually, which reduces productivity and leads to operational errors.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0144] In this invention, the server includes a means for acquiring EEG data in real time, a means for preprocessing the acquired EEG data, and a means for analyzing the preprocessed EEG data and extracting features. This allows the server to transmit summarized thoughts to the robot as work instructions and control the robot's operation based on these instructions. This makes it possible to provide an efficient and safe work environment that reflects the worker's awareness in real time.

[0145] "Electroencephalogram data" refers to data obtained by measuring the user's brain wave activity, and is a signal that reflects the user's state of mind and emotions.

[0146] "Real-time" refers to immediate processing or response with little to no delay.

[0147] "Preprocessing" is the process of performing initial processing such as noise removal, filtering, and standardization on the acquired data.

[0148] "Features" are important patterns and characteristics extracted from EEG data, and are the information that forms the basis of data analysis.

[0149] A "summary" is a concise summary of the main points from the information obtained.

[0150] "Natural language" refers to words and sentences that humans use on a daily basis.

[0151] A "sentence" is a string of meaningful sentences.

[0152] "Work instructions" are specific instructions or instructions for performing a specific task.

[0153] A "robot" is a mechanical device that operates autonomously based on programmed instructions.

[0154] "Operation control" refers to giving instructions and making adjustments so that devices or systems perform predetermined operations.

[0155] A "server" is a high-performance computer used for data processing and communication.

[0156] The present invention is a system that uses electroencephalogram data to summarize a user's thoughts and issues instructions to a robot based on the summaries. An embodiment of this system will be described in detail below.

[0157] System Configuration

[0158] 1. User Device

[0159] The user wears an EEG sensor device that collects real-time EEG data. The EEG sensor device is non-invasive and does not interfere with the user's daily activities.

[0160] 2. Terminal

[0161] The terminal collects EEG data from the EEG sensor device and transmits it to a server via a secure connection. The terminal can also temporarily store data, minimizing data loss.

[0162] 3. Server

[0163] The server receives the electroencephalogram data transmitted from the terminal and performs the following processing.

[0164] Pretreatment

[0165] Denoise, filter, and standardize the data.

[0166] analysis

[0167] The preprocessed data is analyzed using an AI model such as a deep neural network (specifically, Keras) to extract features.

[0168] Summary Generation

[0169] Summarize thoughts based on extracted features, then use a generative AI model to translate the summary into natural language.

[0170] Work order generation

[0171] The summarized thoughts are generated as work instructions and transmitted to the robot.

[0172] Program processing explanation

[0173] The server receives data from the EEG sensor device via a terminal. It then performs noise removal and filtering to generate standardized data. Next, it uses a deep neural network model to analyze the EEG patterns and extract features. Based on these features, it summarizes the user's thoughts via a generative AI model. The summarized content is converted into natural language and then compiled as work instructions. Finally, these work instructions are sent to the robot and executed.

[0174] Hardware and software used

[0175] Hardware

[0176] Brainwave sensor device

[0177] User device (smartphone or PC)

[0178] server

[0179] Factory Robots

[0180] software

[0181] Python Program

[0182] Deep Neural Network Framework (Keras)

[0183] Data preprocessing library (Scikit-learn)

[0184] A library for HTTP communication between servers (Requests)

[0185] Specific examples

[0186] As a concrete example, when a user is thinking about how to install a new part, the following process takes place. When the user wears an EEG sensor device and begins thinking about how to install the new part, the EEG data is sent to the server via the terminal. The server preprocesses the data and, after analysis, generates a summary such as "Try installing the new part." This summary is sent to the robot as a work instruction, and the settings are automatically changed.

[0187] Prompt Sentence Examples

[0188] Prompt: Generate work instructions based on the following EEG data. The data is in the form of data obtained from a sensor device, which has been denoised and filtered. The resulting data is as follows:

[0189] data:

[0190] [0.35, 0.22, 0.45, 0.18, 0.09, 0.11, 0.34, 0.29, 0.55, 0.44]

[0191] Expected result: A work order to try out the new part installation method.

[0192] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide an efficient and safe working environment that reflects the user's awareness in real time.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] Acquisition of EEG data

[0196] The user wears an EEG sensor device, which captures EEG data in real time. The captured EEG data is called raw data and is sent to a terminal connected to the sensor device.

[0197] Input: User's brainwaves

[0198] Output: Raw EEG data (unprocessed EEG data)

[0199] Step 2:

[0200] EEG data transfer

[0201] The terminal receives raw wave data acquired from the sensor device and transmits it to the server via a secure protocol. During this time, the data is temporarily stored on the terminal, reducing the risk of data loss even in the event of a network failure.

[0202] Input: Raw wave data

[0203] Output: Raw wave data temporarily saved on the device

[0204] Step 3:

[0205] Data Preprocessing

[0206] The server preprocesses the raw wave data received from the terminal. Preprocessing includes noise reduction, filtering, and standardization. Noise reduction removes unwanted signals and external influences from the raw wave data. Filtering extracts signals in the desired frequency band. Finally, the data is standardized and adjusted to a uniform scale.

[0207] Input: Raw wave data

[0208] Output: Preprocessed EEG data

[0209] Step 4:

[0210] Feature extraction

[0211] The server analyzes the preprocessed EEG data using a deep neural network (using the Keras framework) to extract important features, which are related to the user's thoughts and emotional state from the EEG patterns.

[0212] Input: Preprocessed EEG data

[0213] Output: Feature data

[0214] Step 5:

[0215] Summary of thoughts

[0216] The server uses a generative AI model to summarize the user's thoughts based on the extracted feature data. At this stage, abstract features are converted into concrete language.

[0217] Input: Feature data

[0218] Output: Summary data

[0219] Step 6:

[0220] Natural language translation

[0221] The server converts the summary data into human-understandable natural language sentences using natural language processing (NLP) techniques, a process that generates sentences that are grammatically and contextually appropriate.

[0222] Input: Summary data

[0223] Output: Natural language sentence

[0224] Step 7:

[0225] Generate work orders

[0226] The server converts the generated natural language sentences into work instructions, which generate instructions that describe specific operations. For example, the generated work instruction might be "try installing a new part."

[0227] Input: Natural language sentence

[0228] Output: Work instructions

[0229] Step 8:

[0230] Sending instructions and controlling the robot's movements

[0231] The server sends work instructions to the robot, which then automatically starts working based on these instructions, and the robot performs the work according to the programmed instructions.

[0232] Input: Work Order

[0233] Output: Robot movement

[0234] These are the specific processing steps of the system that realizes this application example. This flow generates work instructions that reflect the user's thoughts in real time, enabling the robot to automatically perform appropriate operations.

[0235] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0236] Understood. Below is a description of the "Mode for carrying out the invention" based on the invention combined with the emotion engine.

[0237] This invention is a system that clearly summarizes and verbalizes a user's thoughts using electroencephalogram (EEG) data and the user's emotions. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via the terminal. Furthermore, an emotion engine analyzes the user's emotions and reflects them in summarizing the thoughts and generating sentences.

[0238] System Role

[0239] 1. Users

[0240] The user wears an EEG sensor device and provides EEG and emotional data to the system. The sensor device is non-invasive and does not interfere with the user's daily activities.

[0241] 2. Terminal

[0242] The device collects real-time brainwave and emotion data from the sensor device, stores it temporarily, and then transmits the collected data to a server via a secure connection.

[0243] 3. Server

[0244] The server receives the EEG and emotion data sent from the device and performs preprocessing. This preprocessing includes noise removal, filtering, and standardization. It then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. The emotion engine combines these features with the user's emotion data and runs a generative model that summarizes the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques.

[0245] Program processing explanation

[0246] First, the server preprocesses the EEG and emotion data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is input into an AI model such as a deep neural network to analyze EEG patterns and extract features. The emotion engine then analyzes the user's emotions based on these features and generates a summary that combines the emotion data. The generated summary is then formatted into a grammatically and contextually appropriate sentence using natural language processing (NLP) techniques. Finally, the server sends the generated sentence to the device, where it is displayed to the user.

[0247] Specific examples

[0248] Example 1: Summary of user ideas

[0249] When a user is thinking of an idea for a new application, a sensor device captures their brainwave and emotion data, which the device then sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the brainwave patterns and extract features. The emotion engine analyzes emotion data such as "excitement" and "anticipation" and, taking these into account, generates a summary such as "I would like a new app with a feature to manage the user's diet." The generated sentence is then sent to the device and displayed to the user.

[0250] Example 2: Communicating with users with physical disabilities

[0251] When a physically disabled user is thinking about their health, a sensor device captures EEG and emotion data, which the device sends to a server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. The emotion engine analyzes emotion data such as "fatigue" and "anxiety" and generates a summary such as "I feel a bit tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0252] This concludes the description of the "Mode for Carrying Out the Invention." This system allows the user's thoughts to be captured in real time and clearly expressed in words, taking into account their emotions.

[0253] The processing flow will be explained below.

[0254] Understood. Below, I will explain the process of the system combined with the emotion engine step by step, including specific operations.

[0255] Step 1:

[0256] The user wears a dedicated brainwave sensor device that detects brainwave and emotional data in real time.

[0257] Step 2:

[0258] The device collects real-time brainwave and emotional data from the sensor device, which is then temporarily stored and sent to a server via a secure connection.

[0259] Step 3:

[0260] The server receives the EEG data sent from the device and applies a noise reduction algorithm to the received data to remove unwanted signals.

[0261] Step 4:

[0262] The server uses filtering algorithms to emphasize signals in specific frequency bands, and also normalizes the data and converts it into a format suitable for analysis.

[0263] Step 5:

[0264] The server uses a deep neural network to analyze the pre-processed EEG data, analyzing EEG patterns and extracting features.

[0265] Step 6:

[0266] The server uses an emotion engine to analyze the emotion data sent in parallel from the terminals. The emotion engine analyzes the emotion data and extracts emotion features such as "excitement," "anxiety," and "expectation."

[0267] Step 7:

[0268] The server combines the extracted EEG features and emotion features to generate a feature set that takes into account the user's current emotional state.

[0269] Step 8:

[0270] The server uses a generative model to summarize the user's thoughts based on the feature set, and the summary is generated by incorporating sentiment features.

[0271] Step 9:

[0272] The server uses natural language processing (NLP) techniques to convert the summary text into natural language, formatting it into grammatically and contextually appropriate sentences.

[0273] Step 10:

[0274] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[0275] Step 11:

[0276] The terminal displays the received text to the user. The displayed text reflects the user's current emotional state and is therefore easy for the user to understand.

[0277] These are the specific processing steps of a system that combines an emotion engine. This process enables users to clearly verbalize and understand their own thoughts while taking into account their emotional state.

[0278] Example 2

[0279] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] Conventional EEG measurement systems have difficulty accurately summarizing a user's thoughts in real time and expressing them in natural language. Furthermore, these systems do not take emotional data into account, resulting in a lack of contextual interpretation based on the user's emotions. Furthermore, data preprocessing and analysis are insufficient, making it difficult to extract accurate features from noisy data. This leads to the problem of being unable to accurately verbalize the user's true intentions.

[0281] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for preprocessing EEG data and emotion data, means for analyzing the preprocessed EEG data and extracting features, means including an emotion engine for analyzing the emotion data, means including a generative AI model for summarizing thoughts based on the analyzed features and emotion data, and means using natural language processing technology for converting the summarized content into natural language. This makes it possible to integrate the user's EEG data and emotion data in real time, generate a highly accurate summary, and express it appropriately in natural language.

[0282] "Electroencephalograms" are weak electrical signals emitted by nerve cells in the brain.

[0283] A "secure connection" is a means of ensuring the security of communications by encrypting data before sending and receiving it.

[0284] "Preprocessing" refers to the initial processing to improve the accuracy of data analysis, mainly by performing noise removal, filtering, and data standardization.

[0285] A "feature" is useful information extracted from data that can be used for analysis and classification.

[0286] An "emotion engine" is an algorithm or software that analyzes emotional data and estimates a user's emotional state.

[0287] A "generative AI model" refers to an artificial intelligence model that generates text, images, etc. from specific input data.

[0288] "Natural language processing technology" is a general term for technology used to process human language using computers, and involves processes such as sentence generation, grammar analysis, and translation.

[0289] "Device" means hardware and software that collects, stores, and transmits data.

[0290] "Server" means a computer or networked system that receives, processes, analyzes, and outputs data.

[0291] "Real-time" refers to highly immediate processing, in which the entire process from data acquisition to processing is carried out almost simultaneously.

[0292] The present invention is a system that collects and analyzes electroencephalogram (EEG) data and user emotional data to summarize the user's thoughts and express them in natural language. This system begins operation when the user wears a dedicated EEG sensor device.

[0293] User Roles

[0294] The user wears an EEG sensor device on their head. This device is non-invasive and measures brain waves in real time. It is designed not to interfere with the user's daily life, and can steadily collect EEG data even when the user is thinking.

[0295] Device Role

[0296] The device acquires EEG and emotion data from the sensor device in real time, temporarily stores the data in its internal memory, and then encrypts the data and transmits it to the server via a secure connection (e.g., HTTPS protocol).

[0297] Server Roles

[0298] The server receives the data sent from the device and first performs preprocessing such as noise removal, filtering, and data standardization. The received EEG and emotion data is then input into a deep neural network (DNN) to analyze EEG patterns and extract features. An emotion engine then analyzes the emotion data and generates a summary that takes this into account. The generated summary is then converted into grammatically and contextually appropriate text using natural language processing (NLP) technology. The final generated text is then sent to the device and displayed to the user.

[0299] Specific examples

[0300] Example 1: New application idea summary

[0301] When a user is thinking of an idea for a new application, the process is as follows: When the user puts on the device and begins thinking, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "excitement" and "anticipation," and based on that, generates a summary such as "I would like a new app with a feature to manage the user's diet." This sentence is then sent from the server to the device and displayed to the user.

[0302] Example prompt:

[0303] "You input EEG data and emotion data while a user is thinking about a new app idea, and generate a summary."

[0304] Example 2: Communicating with users with physical disabilities

[0305] When a physically disabled user is thinking about their health, the process is as follows: When the user puts on the device and begins to think about their health, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "fatigue" and "anxiety," and generates a summary based on that, such as "I feel a little tired today. I might need to rest." This sentence is then sent from the server to the device and displayed to the user.

[0306] Example prompt:

[0307] "Please input EEG data and emotion data of physically disabled users while they are thinking about their physical condition and generate a summary."

[0308] This invention makes it possible to acquire a user's thoughts in real time and verbalize them after taking into account emotional data, thereby enabling the user to clearly and quickly express their thoughts and ideas in their daily lives.

[0309] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0310] Understood. Below, I will explain the process flow of the system program in detail, divided into steps.

[0311] Step 1:

[0312] The user wears a dedicated brainwave sensor device on their head. This device is non-invasive and measures brainwaves in real time. Specifically, the device uses an antenna to capture electrical signals emitted from the brain and converts them into digital signals using a built-in sensor. The input is the user's brainwave signal, and the output is digitized brainwave data.

[0313] Step 2:

[0314] The terminal acquires EEG data and emotion data from the sensor device in real time and temporarily stores it in the terminal's memory. Here, it receives the data sent from the sensor device and stores it in a designated memory area. The input is the digitized EEG data and emotion data from the sensor device, and the output is the data stored in the terminal's memory.

[0315] Step 3:

[0316] The device encrypts the stored data and sends it over a secure connection to a server, where an encryption algorithm is applied and the data is sent using the HTTPS protocol. The input is digital data stored in the device's memory, and the output is encrypted data.

[0317] Step 4:

[0318] The server decrypts and preprocesses the encrypted data received from the device. The received data is first decrypted, then denoised, filtered and normalised. A filtering algorithm is applied to remove noise, and the filtered data is normalised to a reference value. The input is the encrypted data, and the output is the preprocessed data.

[0319] Step 5:

[0320] The server inputs the preprocessed data into a deep neural network (DNN) to analyze the EEG patterns and extract features. Here, the input data is fed forward to the DNN model to extract features. The input is the preprocessed data, and the output is feature data.

[0321] Step 6:

[0322] The emotion engine analyzes the user's emotion data based on the extracted features. Specifically, the emotion engine algorithm evaluates the features and estimates the user's emotional state. The inputs are feature data and emotion data, and the output is the analyzed emotion data.

[0323] Step 7:

[0324] The server runs a generative AI model based on the analyzed features and emotion data to summarize the user's thoughts. The generative AI model is an algorithm that generates a text summary from the features. The input is the analyzed features and emotion data, and the output is the summary text.

[0325] Step 8:

[0326] NLP technology is used to convert the summary text into grammatically and contextually appropriate natural language. The NLP model analyzes the summary text and optimizes it for natural sentences. The input is the summary text, and the output is well-formatted natural language sentences.

[0327] Step 9:

[0328] Finally, the server sends the generated natural language text to the terminal and displays it to the user. Here, the server encrypts the text and sends it securely to the terminal. The terminal decrypts the received text and displays it on the screen. The input is a formatted natural language text, and the output is the text displayed on the terminal.

[0329] (Application example 2)

[0330] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] In today's brick-and-mortar stores, it is difficult for customers to quickly find products that match their interests and emotions. Furthermore, communication with store staff can be difficult, making the shopping experience stressful. To address this issue, there is a need for a system that uses a customer's brainwave and emotional data to summarize what the customer is thinking and suggest appropriate products.

[0332] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0333] In this invention, the server includes means for acquiring electroencephalogram data in real time, means for preprocessing the acquired electroencephalogram data, means for analyzing the preprocessed electroencephalogram data and emotion data to extract features, means for summarizing thoughts based on the extracted features and emotion data, means for converting the summarized content into natural language, means for outputting the converted natural language sentence, and means for providing the summarized content to the user via the smart device, thereby enabling product recommendations based on the user's interests and emotions or summarizing current thoughts in real time to improve the shopping experience.

[0334] "Electroencephalogram data" is data that records the electrical activity of the brain and is used to analyze specific mental states and emotions.

[0335] "Real-time" refers to acquiring and processing current conditions and data without delay.

[0336] "Emotion data" is data that quantifies the user's psychological state and emotions, and is information that indicates the user's excitement, interest, anxiety, etc.

[0337] "Features" are important patterns or characteristics that the AI ​​model extracts from EEG and emotion data, and are the information used to generate a summary of thoughts.

[0338] A "summary" is information that succinctly and concisely expresses the user's thoughts and feelings, and is a concise version of a long piece of text.

[0339] A "natural language" is a language that humans use on a daily basis, and is a sentence that is constructed in a meaningful way based on grammar and vocabulary.

[0340] A "smart device" is an electronic device equipped with Internet connectivity and applications, and capable of two-way communication with users.

[0341] "Preprocessing" refers to the process performed to prepare raw data for analysis, and includes noise removal and data standardization.

[0342] "Conversion" refers to the conversion of data from one form to another, and in this case refers to the conversion of summarized content into natural language.

[0343] "Output" refers to the system providing the final processing results to the user, and refers to the act of displaying information.

[0344] This invention is a system that uses EEG and emotion data to summarize a user's thoughts and convert them into natural language. The system is designed to provide interest- and emotion-based product recommendations to customers in brick-and-mortar stores.

[0345] The system is configured as follows:

[0346] 1. User:

[0347] The user wears smart glasses and an EEG sensor device, which collects the user's brainwave and emotional data in real time. The EEG sensor device is non-invasive and does not interfere with the user's daily activities. The smart glasses display information in the user's field of vision.

[0348] 2. Terminal:

[0349] The device collects and temporarily stores the EEG and emotional data received from the user, then transmits the collected data to a server over a secure connection. The device also performs pre-processing and filtering of the data.

[0350] 3. Server:

[0351] The server receives the data sent from the device and performs preprocessing, which includes noise removal, filtering, and data standardization. The server implements deep neural network analysis and natural language processing techniques, such as:

[0352] Data Analysis:

[0353] The server inputs the preprocessed EEG data into a deep neural network to extract features, and analyzes the emotion data to identify the user's psychological state.

[0354] Summarize and generate:

[0355] Based on the extracted features and emotion data, the server uses a generative AI model to summarize the user's thoughts and converts the summary into natural language. This sentence generation is achieved using natural language processing (NLP) techniques.

[0356] output:

[0357] The server sends the generated text to the terminal and displays it to the user through the smart glasses.

[0358] The hardware used includes an EEG sensor device, smart glasses, devices (such as smartphones and tablets), and a server, while the software includes deep neural networks (such as TensorFlow and Keras), natural language processing techniques, and data preprocessing libraries (such as SciPy and NumPy).

[0359] Examples:

[0360] scenario:

[0361] A user looking for electronic devices at a shopping mall.

[0362] Example prompt sentence:

[0363] Currently, the user's brainwave and emotional data are as follows: The brainwave data indicates high excitement and anticipation. The emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[0364] Produces a summary:

[0365] "I see you're interested in a new smartphone. Here's the latest model."

[0366] This system improves the user's shopping experience and enables efficient product recommendations.

[0367] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0368] Step 1:

[0369] The user wears smart glasses and a brainwave sensor device.

[0370] The user wears the smart glasses and an EEG sensor device to begin capturing EEG and emotional data. The EEG sensor device records the brain's electrical activity in real time, and the smart glasses are used to display additional information.

[0371] Step 2:

[0372] The device collects and temporarily stores brainwave and emotional data.

[0373] The terminal collects and temporarily stores the EEG and emotional data sent from the EEG sensor device. The input is raw EEG and emotional data, and the output is the temporarily stored data. Specifically, the terminal collects data using wireless communication such as Bluetooth or Wi-Fi.

[0374] Step 3:

[0375] The terminal preprocesses the collected data and sends it to the server.

[0376] The device filters the collected EEG data, removes noise, and standardizes the data. The input is the temporarily stored data, and the output is the preprocessed data. Specifically, data preprocessing is performed using libraries such as SciPy and NumPy. The preprocessed data is then sent to the server via a secure connection.

[0377] Step 4:

[0378] The server inputs the received data into a deep neural network and extracts features.

[0379] The server receives preprocessed EEG data and emotion data sent from the device. The input is the preprocessed data, and the output is features. Specifically, it uses TensorFlow and Keras to analyze EEG patterns and extract features.

[0380] Step 5:

[0381] The server summarizes thoughts based on feature and emotional data and converts them into natural language.

[0382] The server runs a generative AI model that summarizes thoughts based on the extracted features and emotion data. It then uses natural language processing techniques to convert the summarized content into grammatically and contextually appropriate sentences. The input is feature and emotion data, and the output is natural language sentences. Specific operations use NLP libraries (e.g., SpaCy and Hugging Face Transformers).

[0383] Step 6:

[0384] The server sends the generated text to the terminal and displays it through the smart glasses.

[0385] The server sends the generated natural language sentence to the device. The input is the generated sentence, which the device receives and displays to the user through the smart glasses. The output is the information displayed on the smart glasses. Specific operations include sending data using WebSocket or REST API.

[0386] Example prompt sentence:

[0387] Currently, the user's EEG data indicates high excitement and anticipation, while the emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[0388] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0389] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0390] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0391] [Second embodiment]

[0392] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0393] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0394] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0395] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0396] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0398] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0399] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0400] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0401] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0402] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0403] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0404] Understood. Below is the "Mode for Carrying Out the Invention" from the patent specification.

[0405] This invention is a system that uses electroencephalogram (EEG) data to clearly summarize and verbalize a user's thoughts. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via a terminal.

[0406] System Role

[0407] 1. Users

[0408] Users provide brainwave data to the system by wearing a non-invasive EEG sensor device that does not interfere with the user's daily activities.

[0409] 2. Terminal

[0410] The terminal collects EEG data from the sensor device and transmits it to the server. The terminal maintains a secure connection necessary for data transfer and can also temporarily store EEG data, preventing data loss.

[0411] 3. Server

[0412] The server receives the EEG data sent from the device and preprocesses it. Preprocessing includes noise removal, filtering, and standardization. The server then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. Based on these features, the server summarizes the user's thoughts and converts the summary into natural language. The final text is then sent to the device and displayed to the user.

[0413] Program processing explanation

[0414] First, the server preprocesses the EEG data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is fed into an AI model, such as a deep neural network, to analyze EEG patterns and extract features. Based on the extracted features, the server applies a generative model to summarize the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques. Finally, the server sends the generated sentences to the device and displays them to the user.

[0415] Specific examples

[0416] Example 1: Summary of user ideas

[0417] When a user is thinking of an idea for a new application, a sensor device captures their EEG data, which the device sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, the server generates a summary: "I would like a new app with a feature to manage the user's diet." The generated sentence is sent to the device and displayed to the user.

[0418] Example 2: Communicating with users with physical disabilities

[0419] When a physically disabled user is thinking about their health, the sensor device captures EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0420] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to obtain the user's thoughts in real time and clearly verbalize them.

[0421] The processing flow will be explained below.

[0422] Understood. Below is a step-by-step explanation of how the program works.

[0423] Step 1:

[0424] The user wears a specially designed EEG sensor device, which detects electrical activity in the brain and generates EEG data.

[0425] Step 2:

[0426] The device collects EEG data in real time from the sensor device, performs initial temporary storage, and then transmits the collected data to a server via a secure connection.

[0427] Step 3:

[0428] The server preprocesses the received EEG data. First, it applies a noise reduction algorithm to remove unwanted noise from the data. Then it applies filtering to emphasize specific frequency bands. Finally, it standardizes the data and converts it into a format suitable for analysis.

[0429] Step 4:

[0430] The server analyzes the preprocessed EEG data, using AI models such as deep neural networks to analyze EEG patterns and extract features.

[0431] Step 5:

[0432] The server then runs a generative model for summarizing thoughts based on the extracted features. This model generates candidate sentences that summarize thoughts by referencing previously collected data and common patterns.

[0433] Step 6:

[0434] The server converts the generated summary into natural language, using natural language processing (NLP) techniques to format it into grammatically and contextually appropriate sentences that are easy for users to understand.

[0435] Step 7:

[0436] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[0437] Step 8:

[0438] The terminal displays the received sentence to the user, allowing the user to visually check the generated sentence and provide feedback as needed.

[0439] The above are the specific processing steps of the program in the system.

[0440] Example 1

[0441] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0442] Current technology has difficulty capturing a user's thoughts in real time and clearly verbalizing them. Furthermore, there is a lack of effective means for preprocessing EEG data and interpreting it using deep learning models. Therefore, there is a need to accurately analyze a user's EEG data and express it in natural language.

[0443] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0444] In this invention, the server includes means for preprocessing the transferred EEG data, means for inputting the preprocessed EEG data into a deep learning model to extract features, and means for summarizing thoughts using a generative AI model based on the extracted features. This makes it possible to preprocess the user's EEG data in real time, analyze it with the deep learning model, and further summarize thoughts using the generative AI model and convert them into natural language.

[0445] "User" refers to an individual who wears an EEG sensor device and provides EEG data.

[0446] "EEG sensor device" refers to equipment for acquiring a user's brain waves in real time in a non-invasive manner.

[0447] "Terminal" refers to a device that temporarily stores data obtained from an EEG sensor device and transfers it to a server.

[0448] "Server" refers to a computing device that receives the electroencephalogram data transmitted from the terminal and performs preprocessing and analysis.

[0449] "Preprocessing" refers to the process of removing noise from EEG data, filtering, and standardizing the data.

[0450] A "deep learning model" refers to a multi-layer neural network model for analyzing EEG data and extracting features.

[0451] A "generative AI model" refers to an artificial intelligence model that summarizes a user's thoughts based on extracted features.

[0452] "Natural language processing technology" refers to technology that converts summarized content into grammatically and contextually appropriate natural language.

[0453] "Features" refer to the characteristics and patterns of data extracted from EEG data.

[0454] A "summary" refers to information that succinctly summarizes the user's thoughts based on extracted features.

[0455] "Real-time" means that the processing from data acquisition to final output is carried out with almost no delay while the user is wearing the EEG sensor device.

[0456] The present invention is a system that acquires a user's brainwave data in real time, preprocesses it, analyzes it using a deep learning model, and then summarizes their thoughts using a generative AI model and converts them into natural language. This system is composed of the following means.

[0457] 1. Acquisition of EEG data

[0458] The user wears a dedicated brainwave sensor device to acquire brainwave data in real time. This brainwave sensor device is non-invasive and does not interfere with the user's daily life. For example, a commercially available brainwave sensor device called an "electroencephalograph" can be used. This device detects the user's brainwaves with a sensor, converts them into signals, and transmits them to a terminal.

[0459] 2. Data transfer and temporary storage

[0460] The device temporarily stores the data acquired from the EEG sensor device and then transmits it to the server. A database (e.g., SQLite) is used for storage, and a secure connection (e.g., HTTPS) is used for communication with the server. This prevents data loss and information leakage.

[0461] 3. Data Preprocessing

[0462] The server receives the EEG data transferred from the device and first performs preprocessing. This includes noise removal, filtering, and data standardization. The preprocessing uses libraries such as SciPy for noise removal, and NumPy for filtering and standardization. This eliminates signal distortion and external interference and scales the data uniformly.

[0463] 4. Feature extraction using deep learning models

[0464] The preprocessed data is then fed into a deep learning model such as TensorFlow to analyze the EEG patterns and extract features that accurately capture the thoughts occurring in the user's brain.

[0465] 5. Summary Generation Using Generative AI Models

[0466] The server uses a generative AI model (e.g., GPT-3) to summarize the user's thoughts based on the extracted features. This summary is a concise representation of the user's thoughts and plays an important role as the final output of the entire system.

[0467] 6. Natural Language Translation

[0468] The summarized content is then converted into grammatically and contextually appropriate natural language using natural language processing (NLP) techniques, for example, using an NLP library such as spaCy.

[0469] 7. Displaying the results

[0470] The final generated natural language sentence is sent from the server to the device and displayed to the user, which can be a mobile app or a web browser.

[0471] Examples of concrete examples and prompts

[0472] Example 1: Summary of user ideas

[0473] When a user is thinking of an idea for a new application, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, a generative AI model is used to generate a summary such as, "I would like a new app with a feature to manage the user's diet." This summary is then sent to the device and displayed to the user.

[0474] plaintext

[0475] If a user has an idea for a new app:

[0476] [EEG data]

[0477] Summarize what users think.

[0478] Example 2: Communicating with users with physical disabilities

[0479] When a physically disabled user is thinking about their health, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural language and sent to the device for display to the user.

[0480] plaintext

[0481] If you are a physically challenged user and are thinking about your health:

[0482] [EEG data]

[0483] Summarize your thoughts about the user's health.

[0484] The above is an embodiment of the present invention, and this system makes it possible to acquire the user's thoughts in real time and clearly verbalize them.

[0485] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0486] Step 1: The user wears the EEG sensor device

[0487] The user wears the EEG sensor device and begins collecting EEG data. The device is non-invasive and does not interfere with the user's daily activities. The input is the user's EEG, and the output is EEG data converted into a digital signal. Specifically, the sensor device measures EEG in real time, converts the signal into digital data, and transmits it to a terminal.

[0488] Step 2: The device temporarily stores the EEG data.

[0489] The device receives the acquired EEG data and temporarily stores it. This storage is done using an SQLite database on the edge device. The input is the EEG data sent from the sensor device, and the output is the temporarily stored data. Specifically, the device receives the data in real time and writes it to the database.

[0490] Step 3: The device sends the EEG data to the server

[0491] The device sends the stored EEG data to the server. A secure connection (e.g., HTTPS) is used for communication with the server. The input is the temporarily stored EEG data, and the output is the data sent to the server. Specifically, the device reads the data, encrypts it, and sends it to the server via an HTTP request.

[0492] Step 4: The server receives and preprocesses the data

[0493] The server receives the EEG data sent from the device and begins preprocessing. Preprocessing includes noise removal, filtering, and data standardization. The input is the received raw EEG data, and the output is the preprocessed data. Specifically, the server removes noise using the SciPy library and filters and standardizes the data using the NumPy library.

[0494] Step 5: The server extracts features using a deep learning model

[0495] Using the preprocessed data, the server extracts features using a deep learning model (for example, a model using TensorFlow). The input is the preprocessed data, and the output is the extracted features. Specifically, the server inputs the data into the model, analyzes the EEG patterns through a neural network, and extracts important features.

[0496] Step 6: The server summarizes the thoughts using a generative AI model

[0497] The server inputs the extracted features into a generative AI model (e.g., GPT-3) to summarize the thoughts. The input is the extracted features, and the output is a summary of the thoughts. Specifically, the server inputs the prompt sentence and features into the generative AI model, and the model generates a summary.

[0498] Step 7: The server converts the summary into natural language

[0499] The server converts the generated summary into natural language using natural language processing technology (e.g., spaCy). The input is the summarized content, and the output is a natural language sentence. Specifically, the server inputs the summary into an NLP library and converts it into a grammatically and contextually natural sentence.

[0500] Step 8: The server sends the text to the device

[0501] The server sends the generated natural language text to the terminal. The input is the natural language text, and the output is the text sent to the terminal. In concrete terms, the server sends the generated text to the terminal as an HTTP response.

[0502] Step 9: The terminal displays the text to the user

[0503] The terminal displays the received natural language text to the user. The input is the natural language text sent from the server, and the output is the text displayed to the user. In concrete terms, the terminal displays the text on the screen so that the user can immediately check it.

[0504] (Application example 1)

[0505] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0506] Existing systems that use EEG data can extract and verbalize a user's thoughts, but they lack the functionality to further apply this information and transmit it to a robot as work instructions to automatically control its movements. Therefore, there is a need, particularly in industrial sites and factories, to directly reflect the consciousness of workers in work instructions to improve efficiency and safety. Currently, workers' instructions must be entered manually, which reduces productivity and leads to operational errors.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0508] In this invention, the server includes a means for acquiring EEG data in real time, a means for preprocessing the acquired EEG data, and a means for analyzing the preprocessed EEG data and extracting features. This allows the server to transmit summarized thoughts to the robot as work instructions and control the robot's operation based on these instructions. This makes it possible to provide an efficient and safe work environment that reflects the worker's awareness in real time.

[0509] "Electroencephalogram data" refers to data obtained by measuring the user's brain wave activity, and is a signal that reflects the user's state of mind and emotions.

[0510] "Real-time" refers to immediate processing or response with little to no delay.

[0511] "Preprocessing" is the process of performing initial processing such as noise removal, filtering, and standardization on the acquired data.

[0512] "Features" are important patterns and characteristics extracted from EEG data, and are the information that forms the basis of data analysis.

[0513] A "summary" is a concise summary of the main points from the information obtained.

[0514] "Natural language" refers to words and sentences that humans use on a daily basis.

[0515] A "sentence" is a string of meaningful sentences.

[0516] "Work instructions" are specific instructions or instructions for performing a specific task.

[0517] A "robot" is a mechanical device that operates autonomously based on programmed instructions.

[0518] "Operation control" refers to giving instructions and making adjustments so that devices or systems perform predetermined operations.

[0519] A "server" is a high-performance computer used for data processing and communication.

[0520] The present invention is a system that uses electroencephalogram data to summarize a user's thoughts and issues instructions to a robot based on the summaries. An embodiment of this system will be described in detail below.

[0521] System Configuration

[0522] 1. User Device

[0523] The user wears an EEG sensor device that collects real-time EEG data. The EEG sensor device is non-invasive and does not interfere with the user's daily activities.

[0524] 2. Terminal

[0525] The terminal collects EEG data from the EEG sensor device and transmits it to a server via a secure connection. The terminal can also temporarily store data, minimizing data loss.

[0526] 3. Server

[0527] The server receives the electroencephalogram data transmitted from the terminal and performs the following processing.

[0528] Pretreatment

[0529] Denoise, filter, and standardize the data.

[0530] analysis

[0531] The preprocessed data is analyzed using an AI model such as a deep neural network (specifically, Keras) to extract features.

[0532] Summary Generation

[0533] Summarize thoughts based on extracted features, then use a generative AI model to translate the summary into natural language.

[0534] Work order generation

[0535] The summarized thoughts are generated as work instructions and transmitted to the robot.

[0536] Program processing explanation

[0537] The server receives data from the EEG sensor device via a terminal. It then performs noise removal and filtering to generate standardized data. Next, it uses a deep neural network model to analyze the EEG patterns and extract features. Based on these features, it summarizes the user's thoughts via a generative AI model. The summarized content is converted into natural language and then compiled as work instructions. Finally, these work instructions are sent to the robot and executed.

[0538] Hardware and software used

[0539] Hardware

[0540] Brainwave sensor device

[0541] User device (smartphone or PC)

[0542] server

[0543] Factory Robots

[0544] software

[0545] Python Program

[0546] Deep Neural Network Framework (Keras)

[0547] Data preprocessing library (Scikit-learn)

[0548] A library for HTTP communication between servers (Requests)

[0549] Specific examples

[0550] As a concrete example, when a user is thinking about how to install a new part, the following process takes place. When the user wears an EEG sensor device and begins thinking about how to install the new part, the EEG data is sent to the server via the terminal. The server preprocesses the data and, after analysis, generates a summary such as "Try installing the new part." This summary is sent to the robot as a work instruction, and the settings are automatically changed.

[0551] Prompt Sentence Examples

[0552] Prompt: Generate work instructions based on the following EEG data. The data is in the form of data obtained from a sensor device, which has been denoised and filtered. The resulting data is as follows:

[0553] data:

[0554] [0.35, 0.22, 0.45, 0.18, 0.09, 0.11, 0.34, 0.29, 0.55, 0.44]

[0555] Expected result: A work order to try out the new part installation method.

[0556] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide an efficient and safe working environment that reflects the user's awareness in real time.

[0557] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0558] Step 1:

[0559] Acquisition of EEG data

[0560] The user wears an EEG sensor device, which captures EEG data in real time. The captured EEG data is called raw data and is sent to a terminal connected to the sensor device.

[0561] Input: User's brainwaves

[0562] Output: Raw EEG data (unprocessed EEG data)

[0563] Step 2:

[0564] EEG data transfer

[0565] The terminal receives raw wave data acquired from the sensor device and transmits it to the server via a secure protocol. During this time, the data is temporarily stored on the terminal, reducing the risk of data loss even in the event of a network failure.

[0566] Input: Raw wave data

[0567] Output: Raw wave data temporarily saved on the device

[0568] Step 3:

[0569] Data Preprocessing

[0570] The server preprocesses the raw wave data received from the terminal. Preprocessing includes noise reduction, filtering, and standardization. Noise reduction removes unwanted signals and external influences from the raw wave data. Filtering extracts signals in the desired frequency band. Finally, the data is standardized and adjusted to a uniform scale.

[0571] Input: Raw wave data

[0572] Output: Preprocessed EEG data

[0573] Step 4:

[0574] Feature extraction

[0575] The server analyzes the preprocessed EEG data using a deep neural network (using the Keras framework) to extract important features, which are related to the user's thoughts and emotional state from the EEG patterns.

[0576] Input: Preprocessed EEG data

[0577] Output: Feature data

[0578] Step 5:

[0579] Summary of thoughts

[0580] The server uses a generative AI model to summarize the user's thoughts based on the extracted feature data. At this stage, abstract features are converted into concrete language.

[0581] Input: Feature data

[0582] Output: Summary data

[0583] Step 6:

[0584] Natural language translation

[0585] The server converts the summary data into human-understandable natural language sentences using natural language processing (NLP) techniques, a process that generates sentences that are grammatically and contextually appropriate.

[0586] Input: Summary data

[0587] Output: Natural language sentence

[0588] Step 7:

[0589] Generate work orders

[0590] The server converts the generated natural language sentences into work instructions, which generate instructions that describe specific operations. For example, the generated work instruction might be "try installing a new part."

[0591] Input: Natural language sentence

[0592] Output: Work instructions

[0593] Step 8:

[0594] Sending instructions and controlling the robot's movements

[0595] The server sends work instructions to the robot, which then automatically starts working based on these instructions, and the robot performs the work according to the programmed instructions.

[0596] Input: Work Order

[0597] Output: Robot movement

[0598] These are the specific processing steps of the system that realizes this application example. This flow generates work instructions that reflect the user's thoughts in real time, enabling the robot to automatically perform appropriate operations.

[0599] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0600] Understood. Below is a description of the "Mode for carrying out the invention" based on the invention combined with the emotion engine.

[0601] This invention is a system that clearly summarizes and verbalizes a user's thoughts using electroencephalogram (EEG) data and the user's emotions. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via the terminal. Furthermore, an emotion engine analyzes the user's emotions and reflects them in summarizing the thoughts and generating sentences.

[0602] System Role

[0603] 1. Users

[0604] The user wears an EEG sensor device and provides EEG and emotional data to the system. The sensor device is non-invasive and does not interfere with the user's daily activities.

[0605] 2. Terminal

[0606] The device collects real-time brainwave and emotion data from the sensor device, stores it temporarily, and then transmits the collected data to a server via a secure connection.

[0607] 3. Server

[0608] The server receives the EEG and emotion data sent from the device and performs preprocessing. This preprocessing includes noise removal, filtering, and standardization. It then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. The emotion engine combines these features with the user's emotion data and runs a generative model that summarizes the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques.

[0609] Program processing explanation

[0610] First, the server preprocesses the EEG and emotion data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is input into an AI model such as a deep neural network to analyze EEG patterns and extract features. The emotion engine then analyzes the user's emotions based on these features and generates a summary that combines the emotion data. The generated summary is then formatted into a grammatically and contextually appropriate sentence using natural language processing (NLP) techniques. Finally, the server sends the generated sentence to the device, where it is displayed to the user.

[0611] Specific examples

[0612] Example 1: Summary of user ideas

[0613] When a user is thinking of an idea for a new application, a sensor device captures their brainwave and emotion data, which the device then sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the brainwave patterns and extract features. The emotion engine analyzes emotion data such as "excitement" and "anticipation" and, taking these into account, generates a summary such as "I would like a new app with a feature to manage the user's diet." The generated sentence is then sent to the device and displayed to the user.

[0614] Example 2: Communicating with users with physical disabilities

[0615] When a physically disabled user is thinking about their health, a sensor device captures EEG and emotion data, which the device sends to a server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. The emotion engine analyzes emotion data such as "fatigue" and "anxiety" and generates a summary such as "I feel a bit tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0616] This concludes the description of the "Mode for Carrying Out the Invention." This system allows the user's thoughts to be captured in real time and clearly expressed in words, taking into account their emotions.

[0617] The processing flow will be explained below.

[0618] Understood. Below, I will explain the process of the system combined with the emotion engine step by step, including specific operations.

[0619] Step 1:

[0620] The user wears a dedicated brainwave sensor device that detects brainwave and emotional data in real time.

[0621] Step 2:

[0622] The device collects real-time brainwave and emotional data from the sensor device, which is then temporarily stored and sent to a server via a secure connection.

[0623] Step 3:

[0624] The server receives the EEG data sent from the device and applies a noise reduction algorithm to the received data to remove unwanted signals.

[0625] Step 4:

[0626] The server uses filtering algorithms to emphasize signals in specific frequency bands, and also normalizes the data and converts it into a format suitable for analysis.

[0627] Step 5:

[0628] The server uses a deep neural network to analyze the pre-processed EEG data, analyzing EEG patterns and extracting features.

[0629] Step 6:

[0630] The server uses an emotion engine to analyze the emotion data sent in parallel from the terminals. The emotion engine analyzes the emotion data and extracts emotion features such as "excitement," "anxiety," and "expectation."

[0631] Step 7:

[0632] The server combines the extracted EEG features and emotion features to generate a feature set that takes into account the user's current emotional state.

[0633] Step 8:

[0634] The server uses a generative model to summarize the user's thoughts based on the feature set, and the summary is generated by incorporating sentiment features.

[0635] Step 9:

[0636] The server uses natural language processing (NLP) techniques to convert the summary text into natural language, formatting it into grammatically and contextually appropriate sentences.

[0637] Step 10:

[0638] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[0639] Step 11:

[0640] The terminal displays the received text to the user. The displayed text reflects the user's current emotional state and is therefore easy for the user to understand.

[0641] These are the specific processing steps of a system that combines an emotion engine. This process enables users to clearly verbalize and understand their own thoughts while taking into account their emotional state.

[0642] Example 2

[0643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0644] Conventional EEG measurement systems have difficulty accurately summarizing a user's thoughts in real time and expressing them in natural language. Furthermore, these systems do not take emotional data into account, resulting in a lack of contextual interpretation based on the user's emotions. Furthermore, data preprocessing and analysis are insufficient, making it difficult to extract accurate features from noisy data. This leads to the problem of being unable to accurately verbalize the user's true intentions.

[0645] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for preprocessing EEG data and emotion data, means for analyzing the preprocessed EEG data and extracting features, means including an emotion engine for analyzing the emotion data, means including a generative AI model for summarizing thoughts based on the analyzed features and emotion data, and means using natural language processing technology for converting the summarized content into natural language. This makes it possible to integrate the user's EEG data and emotion data in real time, generate a highly accurate summary, and express it appropriately in natural language.

[0646] "Electroencephalograms" are weak electrical signals emitted by nerve cells in the brain.

[0647] A "secure connection" is a means of ensuring the security of communications by encrypting data before sending and receiving it.

[0648] "Preprocessing" refers to the initial processing to improve the accuracy of data analysis, mainly by performing noise removal, filtering, and data standardization.

[0649] A "feature" is useful information extracted from data that can be used for analysis and classification.

[0650] An "emotion engine" is an algorithm or software that analyzes emotional data and estimates a user's emotional state.

[0651] A "generative AI model" refers to an artificial intelligence model that generates text, images, etc. from specific input data.

[0652] "Natural language processing technology" is a general term for technology used to process human language using computers, and involves processes such as sentence generation, grammar analysis, and translation.

[0653] "Device" means hardware and software that collects, stores, and transmits data.

[0654] "Server" means a computer or networked system that receives, processes, analyzes, and outputs data.

[0655] "Real-time" refers to highly immediate processing, in which the entire process from data acquisition to processing is carried out almost simultaneously.

[0656] The present invention is a system that collects and analyzes electroencephalogram (EEG) data and user emotional data to summarize the user's thoughts and express them in natural language. This system begins operation when the user wears a dedicated EEG sensor device.

[0657] User Roles

[0658] The user wears an EEG sensor device on their head. This device is non-invasive and measures brain waves in real time. It is designed not to interfere with the user's daily life, and can steadily collect EEG data even when the user is thinking.

[0659] Device Role

[0660] The device acquires EEG and emotion data from the sensor device in real time, temporarily stores the data in its internal memory, and then encrypts the data and transmits it to the server via a secure connection (e.g., HTTPS protocol).

[0661] Server Roles

[0662] The server receives the data sent from the device and first performs preprocessing such as noise removal, filtering, and data standardization. The received EEG and emotion data is then input into a deep neural network (DNN) to analyze EEG patterns and extract features. An emotion engine then analyzes the emotion data and generates a summary that takes this into account. The generated summary is then converted into grammatically and contextually appropriate text using natural language processing (NLP) technology. The final generated text is then sent to the device and displayed to the user.

[0663] Specific examples

[0664] Example 1: New application idea summary

[0665] When a user is thinking of an idea for a new application, the process is as follows: When the user puts on the device and begins thinking, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "excitement" and "anticipation," and based on that, generates a summary such as "I would like a new app with a feature to manage the user's diet." This sentence is then sent from the server to the device and displayed to the user.

[0666] Example prompt:

[0667] "You input EEG data and emotion data while a user is thinking about a new app idea, and generate a summary."

[0668] Example 2: Communicating with users with physical disabilities

[0669] When a physically disabled user is thinking about their health, the process is as follows: When the user puts on the device and begins to think about their health, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "fatigue" and "anxiety," and generates a summary based on that, such as "I feel a little tired today. I might need to rest." This sentence is then sent from the server to the device and displayed to the user.

[0670] Example prompt:

[0671] "Please input EEG data and emotion data of physically disabled users while they are thinking about their physical condition and generate a summary."

[0672] This invention makes it possible to acquire a user's thoughts in real time and verbalize them after taking into account emotional data, thereby enabling the user to clearly and quickly express their thoughts and ideas in their daily lives.

[0673] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0674] Understood. Below, I will explain the process flow of the system program in detail, divided into steps.

[0675] Step 1:

[0676] The user wears a dedicated brainwave sensor device on their head. This device is non-invasive and measures brainwaves in real time. Specifically, the device uses an antenna to capture electrical signals emitted from the brain and converts them into digital signals using a built-in sensor. The input is the user's brainwave signal, and the output is digitized brainwave data.

[0677] Step 2:

[0678] The terminal acquires EEG data and emotion data from the sensor device in real time and temporarily stores it in the terminal's memory. Here, it receives the data sent from the sensor device and stores it in a designated memory area. The input is the digitized EEG data and emotion data from the sensor device, and the output is the data stored in the terminal's memory.

[0679] Step 3:

[0680] The device encrypts the stored data and sends it over a secure connection to a server, where an encryption algorithm is applied and the data is sent using the HTTPS protocol. The input is digital data stored in the device's memory, and the output is encrypted data.

[0681] Step 4:

[0682] The server decrypts and preprocesses the encrypted data received from the device. The received data is first decrypted, then denoised, filtered and normalised. A filtering algorithm is applied to remove noise, and the filtered data is normalised to a reference value. The input is the encrypted data, and the output is the preprocessed data.

[0683] Step 5:

[0684] The server inputs the preprocessed data into a deep neural network (DNN) to analyze the EEG patterns and extract features. Here, the input data is fed forward to the DNN model to extract features. The input is the preprocessed data, and the output is feature data.

[0685] Step 6:

[0686] The emotion engine analyzes the user's emotion data based on the extracted features. Specifically, the emotion engine algorithm evaluates the features and estimates the user's emotional state. The inputs are feature data and emotion data, and the output is the analyzed emotion data.

[0687] Step 7:

[0688] The server runs a generative AI model based on the analyzed features and emotion data to summarize the user's thoughts. The generative AI model is an algorithm that generates a text summary from the features. The input is the analyzed features and emotion data, and the output is the summary text.

[0689] Step 8:

[0690] NLP technology is used to convert the summary text into grammatically and contextually appropriate natural language. The NLP model analyzes the summary text and optimizes it for natural sentences. The input is the summary text, and the output is well-formatted natural language sentences.

[0691] Step 9:

[0692] Finally, the server sends the generated natural language text to the terminal and displays it to the user. Here, the server encrypts the text and sends it securely to the terminal. The terminal decrypts the received text and displays it on the screen. The input is a formatted natural language text, and the output is the text displayed on the terminal.

[0693] (Application example 2)

[0694] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0695] In today's brick-and-mortar stores, it is difficult for customers to quickly find products that match their interests and emotions. Furthermore, communication with store staff can be difficult, making the shopping experience stressful. To address this issue, there is a need for a system that uses a customer's brainwave and emotional data to summarize what the customer is thinking and suggest appropriate products.

[0696] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0697] In this invention, the server includes means for acquiring electroencephalogram data in real time, means for preprocessing the acquired electroencephalogram data, means for analyzing the preprocessed electroencephalogram data and emotion data to extract features, means for summarizing thoughts based on the extracted features and emotion data, means for converting the summarized content into natural language, means for outputting the converted natural language sentence, and means for providing the summarized content to the user via the smart device, thereby enabling product recommendations based on the user's interests and emotions or summarizing current thoughts in real time to improve the shopping experience.

[0698] "Electroencephalogram data" is data that records the electrical activity of the brain and is used to analyze specific mental states and emotions.

[0699] "Real-time" refers to acquiring and processing current conditions and data without delay.

[0700] "Emotion data" is data that quantifies the user's psychological state and emotions, and is information that indicates the user's excitement, interest, anxiety, etc.

[0701] "Features" are important patterns or characteristics that the AI ​​model extracts from EEG and emotion data, and are the information used to generate a summary of thoughts.

[0702] A "summary" is information that succinctly and concisely expresses the user's thoughts and feelings, and is a concise version of a long piece of text.

[0703] A "natural language" is a language that humans use on a daily basis, and is a sentence that is constructed in a meaningful way based on grammar and vocabulary.

[0704] A "smart device" is an electronic device equipped with Internet connectivity and applications, and capable of two-way communication with users.

[0705] "Preprocessing" refers to the process performed to prepare raw data for analysis, and includes noise removal and data standardization.

[0706] "Conversion" refers to the conversion of data from one form to another, and in this case refers to the conversion of summarized content into natural language.

[0707] "Output" refers to the system providing the final processing results to the user, and refers to the act of displaying information.

[0708] This invention is a system that uses EEG and emotion data to summarize a user's thoughts and convert them into natural language. The system is designed to provide interest- and emotion-based product recommendations to customers in brick-and-mortar stores.

[0709] The system is configured as follows:

[0710] 1. User:

[0711] The user wears smart glasses and an EEG sensor device, which collects the user's brainwave and emotional data in real time. The EEG sensor device is non-invasive and does not interfere with the user's daily activities. The smart glasses display information in the user's field of vision.

[0712] 2. Terminal:

[0713] The device collects and temporarily stores the EEG and emotional data received from the user, then transmits the collected data to a server over a secure connection. The device also performs pre-processing and filtering of the data.

[0714] 3. Server:

[0715] The server receives the data sent from the device and performs preprocessing, which includes noise removal, filtering, and data standardization. The server implements deep neural network analysis and natural language processing techniques, such as:

[0716] Data Analysis:

[0717] The server inputs the preprocessed EEG data into a deep neural network to extract features, and analyzes the emotion data to identify the user's psychological state.

[0718] Summarize and generate:

[0719] Based on the extracted features and emotion data, the server uses a generative AI model to summarize the user's thoughts and converts the summary into natural language. This sentence generation is achieved using natural language processing (NLP) techniques.

[0720] output:

[0721] The server sends the generated text to the terminal and displays it to the user through the smart glasses.

[0722] The hardware used includes an EEG sensor device, smart glasses, devices (such as smartphones and tablets), and a server, while the software includes deep neural networks (such as TensorFlow and Keras), natural language processing techniques, and data preprocessing libraries (such as SciPy and NumPy).

[0723] Examples:

[0724] scenario:

[0725] A user looking for electronic devices at a shopping mall.

[0726] Example prompt sentence:

[0727] Currently, the user's brainwave and emotional data are as follows: The brainwave data indicates high excitement and anticipation. The emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[0728] Produces a summary:

[0729] "I see you're interested in a new smartphone. Here's the latest model."

[0730] This system improves the user's shopping experience and enables efficient product recommendations.

[0731] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0732] Step 1:

[0733] The user wears smart glasses and a brainwave sensor device.

[0734] The user wears the smart glasses and an EEG sensor device to begin capturing EEG and emotional data. The EEG sensor device records the brain's electrical activity in real time, and the smart glasses are used to display additional information.

[0735] Step 2:

[0736] The device collects and temporarily stores brainwave and emotional data.

[0737] The terminal collects and temporarily stores the EEG and emotional data sent from the EEG sensor device. The input is raw EEG and emotional data, and the output is the temporarily stored data. Specifically, the terminal collects data using wireless communication such as Bluetooth or Wi-Fi.

[0738] Step 3:

[0739] The terminal preprocesses the collected data and sends it to the server.

[0740] The device filters the collected EEG data, removes noise, and standardizes the data. The input is the temporarily stored data, and the output is the preprocessed data. Specifically, data preprocessing is performed using libraries such as SciPy and NumPy. The preprocessed data is then sent to the server via a secure connection.

[0741] Step 4:

[0742] The server inputs the received data into a deep neural network and extracts features.

[0743] The server receives preprocessed EEG data and emotion data sent from the device. The input is the preprocessed data, and the output is features. Specifically, it uses TensorFlow and Keras to analyze EEG patterns and extract features.

[0744] Step 5:

[0745] The server summarizes thoughts based on feature and emotional data and converts them into natural language.

[0746] The server runs a generative AI model that summarizes thoughts based on the extracted features and emotion data. It then uses natural language processing techniques to convert the summarized content into grammatically and contextually appropriate sentences. The input is feature and emotion data, and the output is natural language sentences. Specific operations use NLP libraries (e.g., SpaCy and Hugging Face Transformers).

[0747] Step 6:

[0748] The server sends the generated text to the terminal and displays it through the smart glasses.

[0749] The server sends the generated natural language sentence to the device. The input is the generated sentence, which the device receives and displays to the user through the smart glasses. The output is the information displayed on the smart glasses. Specific operations include sending data using WebSocket or REST API.

[0750] Example prompt sentence:

[0751] Currently, the user's EEG data indicates high excitement and anticipation, while the emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[0752] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0753] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0754] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0755] [Third embodiment]

[0756] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0757] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0758] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0759] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0760] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0761] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0762] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0763] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0764] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0765] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0766] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0767] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0768] Understood. Below is the "Mode for Carrying Out the Invention" from the patent specification.

[0769] This invention is a system that uses electroencephalogram (EEG) data to clearly summarize and verbalize a user's thoughts. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via a terminal.

[0770] System Role

[0771] 1. Users

[0772] Users provide brainwave data to the system by wearing a non-invasive EEG sensor device that does not interfere with the user's daily activities.

[0773] 2. Terminal

[0774] The terminal collects EEG data from the sensor device and transmits it to the server. The terminal maintains a secure connection necessary for data transfer and can also temporarily store EEG data, preventing data loss.

[0775] 3. Server

[0776] The server receives the EEG data sent from the device and preprocesses it. Preprocessing includes noise removal, filtering, and standardization. The server then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. Based on these features, the server summarizes the user's thoughts and converts the summary into natural language. The final text is then sent to the device and displayed to the user.

[0777] Program processing explanation

[0778] First, the server preprocesses the EEG data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is fed into an AI model, such as a deep neural network, to analyze EEG patterns and extract features. Based on the extracted features, the server applies a generative model to summarize the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques. Finally, the server sends the generated sentences to the device and displays them to the user.

[0779] Specific examples

[0780] Example 1: Summary of user ideas

[0781] When a user is thinking of an idea for a new application, a sensor device captures their EEG data, which the device sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, the server generates a summary: "I would like a new app with a feature to manage the user's diet." The generated sentence is sent to the device and displayed to the user.

[0782] Example 2: Communicating with users with physical disabilities

[0783] When a physically disabled user is thinking about their health, the sensor device captures EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0784] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to obtain the user's thoughts in real time and clearly verbalize them.

[0785] The processing flow will be explained below.

[0786] Understood. Below is a step-by-step explanation of how the program works.

[0787] Step 1:

[0788] The user wears a specially designed EEG sensor device, which detects electrical activity in the brain and generates EEG data.

[0789] Step 2:

[0790] The device collects EEG data in real time from the sensor device, performs initial temporary storage, and then transmits the collected data to a server via a secure connection.

[0791] Step 3:

[0792] The server preprocesses the received EEG data. First, it applies a noise reduction algorithm to remove unwanted noise from the data. Then it applies filtering to emphasize specific frequency bands. Finally, it standardizes the data and converts it into a format suitable for analysis.

[0793] Step 4:

[0794] The server analyzes the preprocessed EEG data, using AI models such as deep neural networks to analyze EEG patterns and extract features.

[0795] Step 5:

[0796] The server then runs a generative model for summarizing thoughts based on the extracted features. This model generates candidate sentences that summarize thoughts by referencing previously collected data and common patterns.

[0797] Step 6:

[0798] The server converts the generated summary into natural language, using natural language processing (NLP) techniques to format it into grammatically and contextually appropriate sentences that are easy for users to understand.

[0799] Step 7:

[0800] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[0801] Step 8:

[0802] The terminal displays the received sentence to the user, allowing the user to visually check the generated sentence and provide feedback as needed.

[0803] The above are the specific processing steps of the program in the system.

[0804] Example 1

[0805] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0806] Current technology has difficulty capturing a user's thoughts in real time and clearly verbalizing them. Furthermore, there is a lack of effective means for preprocessing EEG data and interpreting it using deep learning models. Therefore, there is a need to accurately analyze a user's EEG data and express it in natural language.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0808] In this invention, the server includes means for preprocessing the transferred EEG data, means for inputting the preprocessed EEG data into a deep learning model to extract features, and means for summarizing thoughts using a generative AI model based on the extracted features. This makes it possible to preprocess the user's EEG data in real time, analyze it with the deep learning model, and further summarize thoughts using the generative AI model and convert them into natural language.

[0809] "User" refers to an individual who wears an EEG sensor device and provides EEG data.

[0810] "EEG sensor device" refers to equipment for acquiring a user's brain waves in real time in a non-invasive manner.

[0811] "Terminal" refers to a device that temporarily stores data obtained from an EEG sensor device and transfers it to a server.

[0812] "Server" refers to a computing device that receives the electroencephalogram data transmitted from the terminal and performs preprocessing and analysis.

[0813] "Preprocessing" refers to the process of removing noise from EEG data, filtering, and standardizing the data.

[0814] A "deep learning model" refers to a multi-layer neural network model for analyzing EEG data and extracting features.

[0815] A "generative AI model" refers to an artificial intelligence model that summarizes a user's thoughts based on extracted features.

[0816] "Natural language processing technology" refers to technology that converts summarized content into grammatically and contextually appropriate natural language.

[0817] "Features" refer to the characteristics and patterns of data extracted from EEG data.

[0818] A "summary" refers to information that succinctly summarizes the user's thoughts based on extracted features.

[0819] "Real-time" means that the processing from data acquisition to final output is carried out with almost no delay while the user is wearing the EEG sensor device.

[0820] The present invention is a system that acquires a user's brainwave data in real time, preprocesses it, analyzes it using a deep learning model, and then summarizes their thoughts using a generative AI model and converts them into natural language. This system is composed of the following means.

[0821] 1. Acquisition of EEG data

[0822] The user wears a dedicated brainwave sensor device to acquire brainwave data in real time. This brainwave sensor device is non-invasive and does not interfere with the user's daily life. For example, a commercially available brainwave sensor device called an "electroencephalograph" can be used. This device detects the user's brainwaves with a sensor, converts them into signals, and transmits them to a terminal.

[0823] 2. Data transfer and temporary storage

[0824] The device temporarily stores the data acquired from the EEG sensor device and then transmits it to the server. A database (e.g., SQLite) is used for storage, and a secure connection (e.g., HTTPS) is used for communication with the server. This prevents data loss and information leakage.

[0825] 3. Data Preprocessing

[0826] The server receives the EEG data transferred from the device and first performs preprocessing. This includes noise removal, filtering, and data standardization. The preprocessing uses libraries such as SciPy for noise removal, and NumPy for filtering and standardization. This eliminates signal distortion and external interference and scales the data uniformly.

[0827] 4. Feature extraction using deep learning models

[0828] The preprocessed data is then fed into a deep learning model such as TensorFlow to analyze the EEG patterns and extract features that accurately capture the thoughts occurring in the user's brain.

[0829] 5. Summary Generation Using Generative AI Models

[0830] The server uses a generative AI model (e.g., GPT-3) to summarize the user's thoughts based on the extracted features. This summary is a concise representation of the user's thoughts and plays an important role as the final output of the entire system.

[0831] 6. Natural Language Translation

[0832] The summarized content is then converted into grammatically and contextually appropriate natural language using natural language processing (NLP) techniques, for example, using an NLP library such as spaCy.

[0833] 7. Displaying the results

[0834] The final generated natural language sentence is sent from the server to the device and displayed to the user, which can be a mobile app or a web browser.

[0835] Examples of concrete examples and prompts

[0836] Example 1: Summary of user ideas

[0837] When a user is thinking of an idea for a new application, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, a generative AI model is used to generate a summary such as, "I would like a new app with a feature to manage the user's diet." This summary is then sent to the device and displayed to the user.

[0838] plaintext

[0839] If a user has an idea for a new app:

[0840] [EEG data]

[0841] Summarize what users think.

[0842] Example 2: Communicating with users with physical disabilities

[0843] When a physically disabled user is thinking about their health, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural language and sent to the device for display to the user.

[0844] plaintext

[0845] If you are a physically challenged user and are thinking about your health:

[0846] [EEG data]

[0847] Summarize your thoughts about the user's health.

[0848] The above is an embodiment of the present invention, and this system makes it possible to acquire the user's thoughts in real time and clearly verbalize them.

[0849] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0850] Step 1: The user wears the EEG sensor device

[0851] The user wears the EEG sensor device and begins collecting EEG data. The device is non-invasive and does not interfere with the user's daily activities. The input is the user's EEG, and the output is EEG data converted into a digital signal. Specifically, the sensor device measures EEG in real time, converts the signal into digital data, and transmits it to a terminal.

[0852] Step 2: The device temporarily stores the EEG data.

[0853] The device receives the acquired EEG data and temporarily stores it. This storage is done using an SQLite database on the edge device. The input is the EEG data sent from the sensor device, and the output is the temporarily stored data. Specifically, the device receives the data in real time and writes it to the database.

[0854] Step 3: The device sends the EEG data to the server

[0855] The device sends the stored EEG data to the server. A secure connection (e.g., HTTPS) is used for communication with the server. The input is the temporarily stored EEG data, and the output is the data sent to the server. Specifically, the device reads the data, encrypts it, and sends it to the server via an HTTP request.

[0856] Step 4: The server receives and preprocesses the data

[0857] The server receives the EEG data sent from the device and begins preprocessing. Preprocessing includes noise removal, filtering, and data standardization. The input is the received raw EEG data, and the output is the preprocessed data. Specifically, the server removes noise using the SciPy library and filters and standardizes the data using the NumPy library.

[0858] Step 5: The server extracts features using a deep learning model

[0859] Using the preprocessed data, the server extracts features using a deep learning model (for example, a model using TensorFlow). The input is the preprocessed data, and the output is the extracted features. Specifically, the server inputs the data into the model, analyzes the EEG patterns through a neural network, and extracts important features.

[0860] Step 6: The server summarizes the thoughts using a generative AI model

[0861] The server inputs the extracted features into a generative AI model (e.g., GPT-3) to summarize the thoughts. The input is the extracted features, and the output is a summary of the thoughts. Specifically, the server inputs the prompt sentence and features into the generative AI model, and the model generates a summary.

[0862] Step 7: The server converts the summary into natural language

[0863] The server converts the generated summary into natural language using natural language processing technology (e.g., spaCy). The input is the summarized content, and the output is a natural language sentence. Specifically, the server inputs the summary into an NLP library and converts it into a grammatically and contextually natural sentence.

[0864] Step 8: The server sends the text to the device

[0865] The server sends the generated natural language text to the terminal. The input is the natural language text, and the output is the text sent to the terminal. In concrete terms, the server sends the generated text to the terminal as an HTTP response.

[0866] Step 9: The terminal displays the text to the user

[0867] The terminal displays the received natural language text to the user. The input is the natural language text sent from the server, and the output is the text displayed to the user. In concrete terms, the terminal displays the text on the screen so that the user can immediately check it.

[0868] (Application example 1)

[0869] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0870] Existing systems that use EEG data can extract and verbalize a user's thoughts, but they lack the functionality to further apply this information and transmit it to a robot as work instructions to automatically control its movements. Therefore, there is a need, particularly in industrial sites and factories, to directly reflect the consciousness of workers in work instructions to improve efficiency and safety. Currently, workers' instructions must be entered manually, which reduces productivity and leads to operational errors.

[0871] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0872] In this invention, the server includes a means for acquiring EEG data in real time, a means for preprocessing the acquired EEG data, and a means for analyzing the preprocessed EEG data and extracting features. This allows the server to transmit summarized thoughts to the robot as work instructions and control the robot's operation based on these instructions. This makes it possible to provide an efficient and safe work environment that reflects the worker's awareness in real time.

[0873] "Electroencephalogram data" refers to data obtained by measuring the user's brain wave activity, and is a signal that reflects the user's state of mind and emotions.

[0874] "Real-time" refers to immediate processing or response with little to no delay.

[0875] "Preprocessing" is the process of performing initial processing such as noise removal, filtering, and standardization on the acquired data.

[0876] "Features" are important patterns and characteristics extracted from EEG data, and are the information that forms the basis of data analysis.

[0877] A "summary" is a concise summary of the main points from the information obtained.

[0878] "Natural language" refers to words and sentences that humans use on a daily basis.

[0879] A "sentence" is a string of meaningful sentences.

[0880] "Work instructions" are specific instructions or instructions for performing a specific task.

[0881] A "robot" is a mechanical device that operates autonomously based on programmed instructions.

[0882] "Operation control" refers to giving instructions and making adjustments so that devices or systems perform predetermined operations.

[0883] A "server" is a high-performance computer used for data processing and communication.

[0884] The present invention is a system that uses electroencephalogram data to summarize a user's thoughts and issues instructions to a robot based on the summaries. An embodiment of this system will be described in detail below.

[0885] System Configuration

[0886] 1. User Device

[0887] The user wears an EEG sensor device that collects real-time EEG data. The EEG sensor device is non-invasive and does not interfere with the user's daily activities.

[0888] 2. Terminal

[0889] The terminal collects EEG data from the EEG sensor device and transmits it to a server via a secure connection. The terminal can also temporarily store data, minimizing data loss.

[0890] 3. Server

[0891] The server receives the electroencephalogram data transmitted from the terminal and performs the following processing.

[0892] Pretreatment

[0893] Denoise, filter, and standardize the data.

[0894] analysis

[0895] The preprocessed data is analyzed using an AI model such as a deep neural network (specifically, Keras) to extract features.

[0896] Summary Generation

[0897] Summarize thoughts based on extracted features, then use a generative AI model to translate the summary into natural language.

[0898] Work order generation

[0899] The summarized thoughts are generated as work instructions and transmitted to the robot.

[0900] Program processing explanation

[0901] The server receives data from the EEG sensor device via a terminal. It then performs noise removal and filtering to generate standardized data. Next, it uses a deep neural network model to analyze the EEG patterns and extract features. Based on these features, it summarizes the user's thoughts via a generative AI model. The summarized content is converted into natural language and then compiled as work instructions. Finally, these work instructions are sent to the robot and executed.

[0902] Hardware and software used

[0903] Hardware

[0904] Brainwave sensor device

[0905] User device (smartphone or PC)

[0906] server

[0907] Factory Robots

[0908] software

[0909] Python Program

[0910] Deep Neural Network Framework (Keras)

[0911] Data preprocessing library (Scikit-learn)

[0912] A library for HTTP communication between servers (Requests)

[0913] Specific examples

[0914] As a concrete example, when a user is thinking about how to install a new part, the following process takes place. When the user wears an EEG sensor device and begins thinking about how to install the new part, the EEG data is sent to the server via the terminal. The server preprocesses the data and, after analysis, generates a summary such as "Try installing the new part." This summary is sent to the robot as a work instruction, and the settings are automatically changed.

[0915] Prompt Sentence Examples

[0916] Prompt: Generate work instructions based on the following EEG data. The data is in the form of data obtained from a sensor device, which has been denoised and filtered. The resulting data is as follows:

[0917] data:

[0918] [0.35, 0.22, 0.45, 0.18, 0.09, 0.11, 0.34, 0.29, 0.55, 0.44]

[0919] Expected result: A work order to try out the new part installation method.

[0920] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide an efficient and safe working environment that reflects the user's awareness in real time.

[0921] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0922] Step 1:

[0923] Acquisition of EEG data

[0924] The user wears an EEG sensor device, which captures EEG data in real time. The captured EEG data is called raw data and is sent to a terminal connected to the sensor device.

[0925] Input: User's brainwaves

[0926] Output: Raw EEG data (unprocessed EEG data)

[0927] Step 2:

[0928] EEG data transfer

[0929] The terminal receives raw wave data acquired from the sensor device and transmits it to the server via a secure protocol. During this time, the data is temporarily stored on the terminal, reducing the risk of data loss even in the event of a network failure.

[0930] Input: Raw wave data

[0931] Output: Raw wave data temporarily saved on the device

[0932] Step 3:

[0933] Data Preprocessing

[0934] The server preprocesses the raw wave data received from the terminal. Preprocessing includes noise reduction, filtering, and standardization. Noise reduction removes unwanted signals and external influences from the raw wave data. Filtering extracts signals in the desired frequency band. Finally, the data is standardized and adjusted to a uniform scale.

[0935] Input: Raw wave data

[0936] Output: Preprocessed EEG data

[0937] Step 4:

[0938] Feature extraction

[0939] The server analyzes the preprocessed EEG data using a deep neural network (using the Keras framework) to extract important features, which are related to the user's thoughts and emotional state from the EEG patterns.

[0940] Input: Preprocessed EEG data

[0941] Output: Feature data

[0942] Step 5:

[0943] Summary of thoughts

[0944] The server uses a generative AI model to summarize the user's thoughts based on the extracted feature data. At this stage, abstract features are converted into concrete language.

[0945] Input: Feature data

[0946] Output: Summary data

[0947] Step 6:

[0948] Natural language translation

[0949] The server converts the summary data into human-understandable natural language sentences using natural language processing (NLP) techniques, a process that generates sentences that are grammatically and contextually appropriate.

[0950] Input: Summary data

[0951] Output: Natural language sentence

[0952] Step 7:

[0953] Generate work orders

[0954] The server converts the generated natural language sentences into work instructions, which generate instructions that describe specific operations. For example, the generated work instruction might be "try installing a new part."

[0955] Input: Natural language sentence

[0956] Output: Work instructions

[0957] Step 8:

[0958] Sending instructions and controlling the robot's movements

[0959] The server sends work instructions to the robot, which then automatically starts working based on these instructions, and the robot performs the work according to the programmed instructions.

[0960] Input: Work Order

[0961] Output: Robot movement

[0962] These are the specific processing steps of the system that realizes this application example. This flow generates work instructions that reflect the user's thoughts in real time, enabling the robot to automatically perform appropriate operations.

[0963] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0964] Understood. Below is a description of the "Mode for carrying out the invention" based on the invention combined with the emotion engine.

[0965] This invention is a system that clearly summarizes and verbalizes a user's thoughts using electroencephalogram (EEG) data and the user's emotions. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via the terminal. Furthermore, an emotion engine analyzes the user's emotions and reflects them in summarizing the thoughts and generating sentences.

[0966] System Role

[0967] 1. Users

[0968] The user wears an EEG sensor device and provides EEG and emotional data to the system. The sensor device is non-invasive and does not interfere with the user's daily activities.

[0969] 2. Terminal

[0970] The device collects real-time brainwave and emotion data from the sensor device, stores it temporarily, and then transmits the collected data to a server via a secure connection.

[0971] 3. Server

[0972] The server receives the EEG and emotion data sent from the device and performs preprocessing. This preprocessing includes noise removal, filtering, and standardization. It then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. The emotion engine combines these features with the user's emotion data and runs a generative model that summarizes the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques.

[0973] Program processing explanation

[0974] First, the server preprocesses the EEG and emotion data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is input into an AI model such as a deep neural network to analyze EEG patterns and extract features. The emotion engine then analyzes the user's emotions based on these features and generates a summary that combines the emotion data. The generated summary is then formatted into a grammatically and contextually appropriate sentence using natural language processing (NLP) techniques. Finally, the server sends the generated sentence to the device, where it is displayed to the user.

[0975] Specific examples

[0976] Example 1: Summary of user ideas

[0977] When a user is thinking of an idea for a new application, a sensor device captures their brainwave and emotion data, which the device then sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the brainwave patterns and extract features. The emotion engine analyzes emotion data such as "excitement" and "anticipation" and, taking these into account, generates a summary such as "I would like a new app with a feature to manage the user's diet." The generated sentence is then sent to the device and displayed to the user.

[0978] Example 2: Communicating with users with physical disabilities

[0979] When a physically disabled user is thinking about their health, a sensor device captures EEG and emotion data, which the device sends to a server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. The emotion engine analyzes emotion data such as "fatigue" and "anxiety" and generates a summary such as "I feel a bit tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[0980] This concludes the description of the "Mode for Carrying Out the Invention." This system allows the user's thoughts to be captured in real time and clearly expressed in words, taking into account their emotions.

[0981] The processing flow will be explained below.

[0982] Understood. Below, I will explain the process of the system combined with the emotion engine step by step, including specific operations.

[0983] Step 1:

[0984] The user wears a dedicated brainwave sensor device that detects brainwave and emotional data in real time.

[0985] Step 2:

[0986] The device collects real-time brainwave and emotional data from the sensor device, which is then temporarily stored and sent to a server via a secure connection.

[0987] Step 3:

[0988] The server receives the EEG data sent from the device and applies a noise reduction algorithm to the received data to remove unwanted signals.

[0989] Step 4:

[0990] The server uses filtering algorithms to emphasize signals in specific frequency bands, and also normalizes the data and converts it into a format suitable for analysis.

[0991] Step 5:

[0992] The server uses a deep neural network to analyze the pre-processed EEG data, analyzing EEG patterns and extracting features.

[0993] Step 6:

[0994] The server uses an emotion engine to analyze the emotion data sent in parallel from the terminals. The emotion engine analyzes the emotion data and extracts emotion features such as "excitement," "anxiety," and "expectation."

[0995] Step 7:

[0996] The server combines the extracted EEG features and emotion features to generate a feature set that takes into account the user's current emotional state.

[0997] Step 8:

[0998] The server uses a generative model to summarize the user's thoughts based on the feature set, and the summary is generated by incorporating sentiment features.

[0999] Step 9:

[1000] The server uses natural language processing (NLP) techniques to convert the summary text into natural language, formatting it into grammatically and contextually appropriate sentences.

[1001] Step 10:

[1002] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[1003] Step 11:

[1004] The terminal displays the received text to the user. The displayed text reflects the user's current emotional state and is therefore easy for the user to understand.

[1005] These are the specific processing steps of a system that combines an emotion engine. This process enables users to clearly verbalize and understand their own thoughts while taking into account their emotional state.

[1006] Example 2

[1007] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1008] Conventional EEG measurement systems have difficulty accurately summarizing a user's thoughts in real time and expressing them in natural language. Furthermore, these systems do not take emotional data into account, resulting in a lack of contextual interpretation based on the user's emotions. Furthermore, data preprocessing and analysis are insufficient, making it difficult to extract accurate features from noisy data. This leads to the problem of being unable to accurately verbalize the user's true intentions.

[1009] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for preprocessing EEG data and emotion data, means for analyzing the preprocessed EEG data and extracting features, means including an emotion engine for analyzing the emotion data, means including a generative AI model for summarizing thoughts based on the analyzed features and emotion data, and means using natural language processing technology for converting the summarized content into natural language. This makes it possible to integrate the user's EEG data and emotion data in real time, generate a highly accurate summary, and express it appropriately in natural language.

[1010] "Electroencephalograms" are weak electrical signals emitted by nerve cells in the brain.

[1011] A "secure connection" is a means of ensuring the security of communications by encrypting data before sending and receiving it.

[1012] "Preprocessing" refers to the initial processing to improve the accuracy of data analysis, mainly by performing noise removal, filtering, and data standardization.

[1013] A "feature" is useful information extracted from data that can be used for analysis and classification.

[1014] An "emotion engine" is an algorithm or software that analyzes emotional data and estimates a user's emotional state.

[1015] A "generative AI model" refers to an artificial intelligence model that generates text, images, etc. from specific input data.

[1016] "Natural language processing technology" is a general term for technology used to process human language using computers, and involves processes such as sentence generation, grammar analysis, and translation.

[1017] "Device" means hardware and software that collects, stores, and transmits data.

[1018] "Server" means a computer or networked system that receives, processes, analyzes, and outputs data.

[1019] "Real-time" refers to highly immediate processing, in which the entire process from data acquisition to processing is carried out almost simultaneously.

[1020] The present invention is a system that collects and analyzes electroencephalogram (EEG) data and user emotional data to summarize the user's thoughts and express them in natural language. This system begins operation when the user wears a dedicated EEG sensor device.

[1021] User Roles

[1022] The user wears an EEG sensor device on their head. This device is non-invasive and measures brain waves in real time. It is designed not to interfere with the user's daily life, and can steadily collect EEG data even when the user is thinking.

[1023] Device Role

[1024] The device acquires EEG and emotion data from the sensor device in real time, temporarily stores the data in its internal memory, and then encrypts the data and transmits it to the server via a secure connection (e.g., HTTPS protocol).

[1025] Server Roles

[1026] The server receives the data sent from the device and first performs preprocessing such as noise removal, filtering, and data standardization. The received EEG and emotion data is then input into a deep neural network (DNN) to analyze EEG patterns and extract features. An emotion engine then analyzes the emotion data and generates a summary that takes this into account. The generated summary is then converted into grammatically and contextually appropriate text using natural language processing (NLP) technology. The final generated text is then sent to the device and displayed to the user.

[1027] Specific examples

[1028] Example 1: New application idea summary

[1029] When a user is thinking of an idea for a new application, the process is as follows: When the user puts on the device and begins thinking, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "excitement" and "anticipation," and based on that, generates a summary such as "I would like a new app with a feature to manage the user's diet." This sentence is then sent from the server to the device and displayed to the user.

[1030] Example prompt:

[1031] "You input EEG data and emotion data while a user is thinking about a new app idea, and generate a summary."

[1032] Example 2: Communicating with users with physical disabilities

[1033] When a physically disabled user is thinking about their health, the process is as follows: When the user puts on the device and begins to think about their health, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "fatigue" and "anxiety," and generates a summary based on that, such as "I feel a little tired today. I might need to rest." This sentence is then sent from the server to the device and displayed to the user.

[1034] Example prompt:

[1035] "Please input EEG data and emotion data of physically disabled users while they are thinking about their physical condition and generate a summary."

[1036] This invention makes it possible to acquire a user's thoughts in real time and verbalize them after taking into account emotional data, thereby enabling the user to clearly and quickly express their thoughts and ideas in their daily lives.

[1037] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1038] Understood. Below, I will explain the process flow of the system program in detail, divided into steps.

[1039] Step 1:

[1040] The user wears a dedicated brainwave sensor device on their head. This device is non-invasive and measures brainwaves in real time. Specifically, the device uses an antenna to capture electrical signals emitted from the brain and converts them into digital signals using a built-in sensor. The input is the user's brainwave signal, and the output is digitized brainwave data.

[1041] Step 2:

[1042] The terminal acquires EEG data and emotion data from the sensor device in real time and temporarily stores it in the terminal's memory. Here, it receives the data sent from the sensor device and stores it in a designated memory area. The input is the digitized EEG data and emotion data from the sensor device, and the output is the data stored in the terminal's memory.

[1043] Step 3:

[1044] The device encrypts the stored data and sends it over a secure connection to a server, where an encryption algorithm is applied and the data is sent using the HTTPS protocol. The input is digital data stored in the device's memory, and the output is encrypted data.

[1045] Step 4:

[1046] The server decrypts and preprocesses the encrypted data received from the device. The received data is first decrypted, then denoised, filtered and normalised. A filtering algorithm is applied to remove noise, and the filtered data is normalised to a reference value. The input is the encrypted data, and the output is the preprocessed data.

[1047] Step 5:

[1048] The server inputs the preprocessed data into a deep neural network (DNN) to analyze the EEG patterns and extract features. Here, the input data is fed forward to the DNN model to extract features. The input is the preprocessed data, and the output is feature data.

[1049] Step 6:

[1050] The emotion engine analyzes the user's emotion data based on the extracted features. Specifically, the emotion engine algorithm evaluates the features and estimates the user's emotional state. The inputs are feature data and emotion data, and the output is the analyzed emotion data.

[1051] Step 7:

[1052] The server runs a generative AI model based on the analyzed features and emotion data to summarize the user's thoughts. The generative AI model is an algorithm that generates a text summary from the features. The input is the analyzed features and emotion data, and the output is the summary text.

[1053] Step 8:

[1054] NLP technology is used to convert the summary text into grammatically and contextually appropriate natural language. The NLP model analyzes the summary text and optimizes it for natural sentences. The input is the summary text, and the output is well-formatted natural language sentences.

[1055] Step 9:

[1056] Finally, the server sends the generated natural language text to the terminal and displays it to the user. Here, the server encrypts the text and sends it securely to the terminal. The terminal decrypts the received text and displays it on the screen. The input is a formatted natural language text, and the output is the text displayed on the terminal.

[1057] (Application example 2)

[1058] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] In today's brick-and-mortar stores, it is difficult for customers to quickly find products that match their interests and emotions. Furthermore, communication with store staff can be difficult, making the shopping experience stressful. To address this issue, there is a need for a system that uses a customer's brainwave and emotional data to summarize what the customer is thinking and suggest appropriate products.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1061] In this invention, the server includes means for acquiring electroencephalogram data in real time, means for preprocessing the acquired electroencephalogram data, means for analyzing the preprocessed electroencephalogram data and emotion data to extract features, means for summarizing thoughts based on the extracted features and emotion data, means for converting the summarized content into natural language, means for outputting the converted natural language sentence, and means for providing the summarized content to the user via the smart device, thereby enabling product recommendations based on the user's interests and emotions or summarizing current thoughts in real time to improve the shopping experience.

[1062] "Electroencephalogram data" is data that records the electrical activity of the brain and is used to analyze specific mental states and emotions.

[1063] "Real-time" refers to acquiring and processing current conditions and data without delay.

[1064] "Emotion data" is data that quantifies the user's psychological state and emotions, and is information that indicates the user's excitement, interest, anxiety, etc.

[1065] "Features" are important patterns or characteristics that the AI ​​model extracts from EEG and emotion data, and are the information used to generate a summary of thoughts.

[1066] A "summary" is information that succinctly and concisely expresses the user's thoughts and feelings, and is a concise version of a long piece of text.

[1067] A "natural language" is a language that humans use on a daily basis, and is a sentence that is constructed in a meaningful way based on grammar and vocabulary.

[1068] A "smart device" is an electronic device equipped with Internet connectivity and applications, and capable of two-way communication with users.

[1069] "Preprocessing" refers to the process performed to prepare raw data for analysis, and includes noise removal and data standardization.

[1070] "Conversion" refers to the conversion of data from one form to another, and in this case refers to the conversion of summarized content into natural language.

[1071] "Output" refers to the system providing the final processing results to the user, and refers to the act of displaying information.

[1072] This invention is a system that uses EEG and emotion data to summarize a user's thoughts and convert them into natural language. The system is designed to provide interest- and emotion-based product recommendations to customers in brick-and-mortar stores.

[1073] The system is configured as follows:

[1074] 1. User:

[1075] The user wears smart glasses and an EEG sensor device, which collects the user's brainwave and emotional data in real time. The EEG sensor device is non-invasive and does not interfere with the user's daily activities. The smart glasses display information in the user's field of vision.

[1076] 2. Terminal:

[1077] The device collects and temporarily stores the EEG and emotional data received from the user, then transmits the collected data to a server over a secure connection. The device also performs pre-processing and filtering of the data.

[1078] 3. Server:

[1079] The server receives the data sent from the device and performs preprocessing, which includes noise removal, filtering, and data standardization. The server implements deep neural network analysis and natural language processing techniques, such as:

[1080] Data Analysis:

[1081] The server inputs the preprocessed EEG data into a deep neural network to extract features, and analyzes the emotion data to identify the user's psychological state.

[1082] Summarize and generate:

[1083] Based on the extracted features and emotion data, the server uses a generative AI model to summarize the user's thoughts and converts the summary into natural language. This sentence generation is achieved using natural language processing (NLP) techniques.

[1084] output:

[1085] The server sends the generated text to the terminal and displays it to the user through the smart glasses.

[1086] The hardware used includes an EEG sensor device, smart glasses, devices (such as smartphones and tablets), and a server, while the software includes deep neural networks (such as TensorFlow and Keras), natural language processing techniques, and data preprocessing libraries (such as SciPy and NumPy).

[1087] Examples:

[1088] scenario:

[1089] A user looking for electronic devices at a shopping mall.

[1090] Example prompt sentence:

[1091] Currently, the user's brainwave and emotional data are as follows: The brainwave data indicates high excitement and anticipation. The emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[1092] Produces a summary:

[1093] "I see you're interested in a new smartphone. Here's the latest model."

[1094] This system improves the user's shopping experience and enables efficient product recommendations.

[1095] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1096] Step 1:

[1097] The user wears smart glasses and a brainwave sensor device.

[1098] The user wears the smart glasses and an EEG sensor device to begin capturing EEG and emotional data. The EEG sensor device records the brain's electrical activity in real time, and the smart glasses are used to display additional information.

[1099] Step 2:

[1100] The device collects and temporarily stores brainwave and emotional data.

[1101] The terminal collects and temporarily stores the EEG and emotional data sent from the EEG sensor device. The input is raw EEG and emotional data, and the output is the temporarily stored data. Specifically, the terminal collects data using wireless communication such as Bluetooth or Wi-Fi.

[1102] Step 3:

[1103] The terminal preprocesses the collected data and sends it to the server.

[1104] The device filters the collected EEG data, removes noise, and standardizes the data. The input is the temporarily stored data, and the output is the preprocessed data. Specifically, data preprocessing is performed using libraries such as SciPy and NumPy. The preprocessed data is then sent to the server via a secure connection.

[1105] Step 4:

[1106] The server inputs the received data into a deep neural network and extracts features.

[1107] The server receives preprocessed EEG data and emotion data sent from the device. The input is the preprocessed data, and the output is features. Specifically, it uses TensorFlow and Keras to analyze EEG patterns and extract features.

[1108] Step 5:

[1109] The server summarizes thoughts based on feature and emotional data and converts them into natural language.

[1110] The server runs a generative AI model that summarizes thoughts based on the extracted features and emotion data. It then uses natural language processing techniques to convert the summarized content into grammatically and contextually appropriate sentences. The input is feature and emotion data, and the output is natural language sentences. Specific operations use NLP libraries (e.g., SpaCy and Hugging Face Transformers).

[1111] Step 6:

[1112] The server sends the generated text to the terminal and displays it through the smart glasses.

[1113] The server sends the generated natural language sentence to the device. The input is the generated sentence, which the device receives and displays to the user through the smart glasses. The output is the information displayed on the smart glasses. Specific operations include sending data using WebSocket or REST API.

[1114] Example prompt sentence:

[1115] Currently, the user's EEG data indicates high excitement and anticipation, while the emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[1116] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1117] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1118] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1119] [Fourth embodiment]

[1120] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1121] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1122] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1123] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1124] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1125] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1126] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1127] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1128] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1129] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1130] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1131] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1132] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1133] Understood. Below is the "Mode for Carrying Out the Invention" from the patent specification.

[1134] This invention is a system that uses electroencephalogram (EEG) data to clearly summarize and verbalize a user's thoughts. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via a terminal.

[1135] System Role

[1136] 1. Users

[1137] Users provide brainwave data to the system by wearing a non-invasive EEG sensor device that does not interfere with the user's daily activities.

[1138] 2. Terminal

[1139] The terminal collects EEG data from the sensor device and transmits it to the server. The terminal maintains a secure connection necessary for data transfer and can also temporarily store EEG data, preventing data loss.

[1140] 3. Server

[1141] The server receives the EEG data sent from the device and preprocesses it. Preprocessing includes noise removal, filtering, and standardization. The server then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. Based on these features, the server summarizes the user's thoughts and converts the summary into natural language. The final text is then sent to the device and displayed to the user.

[1142] Program processing explanation

[1143] First, the server preprocesses the EEG data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is fed into an AI model, such as a deep neural network, to analyze EEG patterns and extract features. Based on the extracted features, the server applies a generative model to summarize the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques. Finally, the server sends the generated sentences to the device and displays them to the user.

[1144] Specific examples

[1145] Example 1: Summary of user ideas

[1146] When a user is thinking of an idea for a new application, a sensor device captures their EEG data, which the device sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, the server generates a summary: "I would like a new app with a feature to manage the user's diet." The generated sentence is sent to the device and displayed to the user.

[1147] Example 2: Communicating with users with physical disabilities

[1148] When a physically disabled user is thinking about their health, the sensor device captures EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[1149] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to obtain the user's thoughts in real time and clearly verbalize them.

[1150] The processing flow will be explained below.

[1151] Understood. Below is a step-by-step explanation of how the program works.

[1152] Step 1:

[1153] The user wears a specially designed EEG sensor device, which detects electrical activity in the brain and generates EEG data.

[1154] Step 2:

[1155] The device collects EEG data in real time from the sensor device, performs initial temporary storage, and then transmits the collected data to a server via a secure connection.

[1156] Step 3:

[1157] The server preprocesses the received EEG data. First, it applies a noise reduction algorithm to remove unwanted noise from the data. Then it applies filtering to emphasize specific frequency bands. Finally, it standardizes the data and converts it into a format suitable for analysis.

[1158] Step 4:

[1159] The server analyzes the preprocessed EEG data, using AI models such as deep neural networks to analyze EEG patterns and extract features.

[1160] Step 5:

[1161] The server then runs a generative model for summarizing thoughts based on the extracted features. This model generates candidate sentences that summarize thoughts by referencing previously collected data and common patterns.

[1162] Step 6:

[1163] The server converts the generated summary into natural language, using natural language processing (NLP) techniques to format it into grammatically and contextually appropriate sentences that are easy for users to understand.

[1164] Step 7:

[1165] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[1166] Step 8:

[1167] The terminal displays the received sentence to the user, allowing the user to visually check the generated sentence and provide feedback as needed.

[1168] The above are the specific processing steps of the program in the system.

[1169] Example 1

[1170] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1171] Current technology has difficulty capturing a user's thoughts in real time and clearly verbalizing them. Furthermore, there is a lack of effective means for preprocessing EEG data and interpreting it using deep learning models. Therefore, there is a need to accurately analyze a user's EEG data and express it in natural language.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1173] In this invention, the server includes means for preprocessing the transferred EEG data, means for inputting the preprocessed EEG data into a deep learning model to extract features, and means for summarizing thoughts using a generative AI model based on the extracted features. This makes it possible to preprocess the user's EEG data in real time, analyze it with the deep learning model, and further summarize thoughts using the generative AI model and convert them into natural language.

[1174] "User" refers to an individual who wears an EEG sensor device and provides EEG data.

[1175] "EEG sensor device" refers to equipment for acquiring a user's brain waves in real time in a non-invasive manner.

[1176] "Terminal" refers to a device that temporarily stores data obtained from an EEG sensor device and transfers it to a server.

[1177] "Server" refers to a computing device that receives the electroencephalogram data transmitted from the terminal and performs preprocessing and analysis.

[1178] "Preprocessing" refers to the process of removing noise from EEG data, filtering, and standardizing the data.

[1179] A "deep learning model" refers to a multi-layer neural network model for analyzing EEG data and extracting features.

[1180] A "generative AI model" refers to an artificial intelligence model that summarizes a user's thoughts based on extracted features.

[1181] "Natural language processing technology" refers to technology that converts summarized content into grammatically and contextually appropriate natural language.

[1182] "Features" refer to the characteristics and patterns of data extracted from EEG data.

[1183] A "summary" refers to information that succinctly summarizes the user's thoughts based on extracted features.

[1184] "Real-time" means that the processing from data acquisition to final output is carried out with almost no delay while the user is wearing the EEG sensor device.

[1185] The present invention is a system that acquires a user's brainwave data in real time, preprocesses it, analyzes it using a deep learning model, and then summarizes their thoughts using a generative AI model and converts them into natural language. This system is composed of the following means.

[1186] 1. Acquisition of EEG data

[1187] The user wears a dedicated brainwave sensor device to acquire brainwave data in real time. This brainwave sensor device is non-invasive and does not interfere with the user's daily life. For example, a commercially available brainwave sensor device called an "electroencephalograph" can be used. This device detects the user's brainwaves with a sensor, converts them into signals, and transmits them to a terminal.

[1188] 2. Data transfer and temporary storage

[1189] The device temporarily stores the data acquired from the EEG sensor device and then transmits it to the server. A database (e.g., SQLite) is used for storage, and a secure connection (e.g., HTTPS) is used for communication with the server. This prevents data loss and information leakage.

[1190] 3. Data Preprocessing

[1191] The server receives the EEG data transferred from the device and first performs preprocessing. This includes noise removal, filtering, and data standardization. The preprocessing uses libraries such as SciPy for noise removal, and NumPy for filtering and standardization. This eliminates signal distortion and external interference and scales the data uniformly.

[1192] 4. Feature extraction using deep learning models

[1193] The preprocessed data is then fed into a deep learning model such as TensorFlow to analyze the EEG patterns and extract features that accurately capture the thoughts occurring in the user's brain.

[1194] 5. Summary Generation Using Generative AI Models

[1195] The server uses a generative AI model (e.g., GPT-3) to summarize the user's thoughts based on the extracted features. This summary is a concise representation of the user's thoughts and plays an important role as the final output of the entire system.

[1196] 6. Natural Language Translation

[1197] The summarized content is then converted into grammatically and contextually appropriate natural language using natural language processing (NLP) techniques, for example, using an NLP library such as spaCy.

[1198] 7. Displaying the results

[1199] The final generated natural language sentence is sent from the server to the device and displayed to the user, which can be a mobile app or a web browser.

[1200] Examples of concrete examples and prompts

[1201] Example 1: Summary of user ideas

[1202] When a user is thinking of an idea for a new application, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and uses a deep neural network to analyze the EEG patterns and extract features. Based on these features, a generative AI model is used to generate a summary such as, "I would like a new app with a feature to manage the user's diet." This summary is then sent to the device and displayed to the user.

[1203] plaintext

[1204] If a user has an idea for a new app:

[1205] [EEG data]

[1206] Summarize what users think.

[1207] Example 2: Communicating with users with physical disabilities

[1208] When a physically disabled user is thinking about their health, the EEG sensor device captures the EEG data, which the device sends to the server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. As a result of the analysis, the server generates a summary such as "I feel a little tired today. I might need to rest." This summary is converted into natural language and sent to the device for display to the user.

[1209] plaintext

[1210] If you are a physically challenged user and are thinking about your health:

[1211] [EEG data]

[1212] Summarize your thoughts about the user's health.

[1213] The above is an embodiment of the present invention, and this system makes it possible to acquire the user's thoughts in real time and clearly verbalize them.

[1214] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1215] Step 1: The user wears the EEG sensor device

[1216] The user wears the EEG sensor device and begins collecting EEG data. The device is non-invasive and does not interfere with the user's daily activities. The input is the user's EEG, and the output is EEG data converted into a digital signal. Specifically, the sensor device measures EEG in real time, converts the signal into digital data, and transmits it to a terminal.

[1217] Step 2: The device temporarily stores the EEG data.

[1218] The device receives the acquired EEG data and temporarily stores it. This storage is done using an SQLite database on the edge device. The input is the EEG data sent from the sensor device, and the output is the temporarily stored data. Specifically, the device receives the data in real time and writes it to the database.

[1219] Step 3: The device sends the EEG data to the server

[1220] The device sends the stored EEG data to the server. A secure connection (e.g., HTTPS) is used for communication with the server. The input is the temporarily stored EEG data, and the output is the data sent to the server. Specifically, the device reads the data, encrypts it, and sends it to the server via an HTTP request.

[1221] Step 4: The server receives and preprocesses the data

[1222] The server receives the EEG data sent from the device and begins preprocessing. Preprocessing includes noise removal, filtering, and data standardization. The input is the received raw EEG data, and the output is the preprocessed data. Specifically, the server removes noise using the SciPy library and filters and standardizes the data using the NumPy library.

[1223] Step 5: The server extracts features using a deep learning model

[1224] Using the preprocessed data, the server extracts features using a deep learning model (for example, a model using TensorFlow). The input is the preprocessed data, and the output is the extracted features. Specifically, the server inputs the data into the model, analyzes the EEG patterns through a neural network, and extracts important features.

[1225] Step 6: The server summarizes the thoughts using a generative AI model

[1226] The server inputs the extracted features into a generative AI model (e.g., GPT-3) to summarize the thoughts. The input is the extracted features, and the output is a summary of the thoughts. Specifically, the server inputs the prompt sentence and features into the generative AI model, and the model generates a summary.

[1227] Step 7: The server converts the summary into natural language

[1228] The server converts the generated summary into natural language using natural language processing technology (e.g., spaCy). The input is the summarized content, and the output is a natural language sentence. Specifically, the server inputs the summary into an NLP library and converts it into a grammatically and contextually natural sentence.

[1229] Step 8: The server sends the text to the device

[1230] The server sends the generated natural language text to the terminal. The input is the natural language text, and the output is the text sent to the terminal. In concrete terms, the server sends the generated text to the terminal as an HTTP response.

[1231] Step 9: The terminal displays the text to the user

[1232] The terminal displays the received natural language text to the user. The input is the natural language text sent from the server, and the output is the text displayed to the user. In concrete terms, the terminal displays the text on the screen so that the user can immediately check it.

[1233] (Application example 1)

[1234] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1235] Existing systems that use EEG data can extract and verbalize a user's thoughts, but they lack the functionality to further apply this information and transmit it to a robot as work instructions to automatically control its movements. Therefore, there is a need, particularly in industrial sites and factories, to directly reflect the consciousness of workers in work instructions to improve efficiency and safety. Currently, workers' instructions must be entered manually, which reduces productivity and leads to operational errors.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1237] In this invention, the server includes a means for acquiring EEG data in real time, a means for preprocessing the acquired EEG data, and a means for analyzing the preprocessed EEG data and extracting features. This allows the server to transmit summarized thoughts to the robot as work instructions and control the robot's operation based on these instructions. This makes it possible to provide an efficient and safe work environment that reflects the worker's awareness in real time.

[1238] "Electroencephalogram data" refers to data obtained by measuring the user's brain wave activity, and is a signal that reflects the user's state of mind and emotions.

[1239] "Real-time" refers to immediate processing or response with little to no delay.

[1240] "Preprocessing" is the process of performing initial processing such as noise removal, filtering, and standardization on the acquired data.

[1241] "Features" are important patterns and characteristics extracted from EEG data, and are the information that forms the basis of data analysis.

[1242] A "summary" is a concise summary of the main points from the information obtained.

[1243] "Natural language" refers to words and sentences that humans use on a daily basis.

[1244] A "sentence" is a string of meaningful sentences.

[1245] "Work instructions" are specific instructions or instructions for performing a specific task.

[1246] A "robot" is a mechanical device that operates autonomously based on programmed instructions.

[1247] "Operation control" refers to giving instructions and making adjustments so that devices or systems perform predetermined operations.

[1248] A "server" is a high-performance computer used for data processing and communication.

[1249] The present invention is a system that uses electroencephalogram data to summarize a user's thoughts and issues instructions to a robot based on the summaries. An embodiment of this system will be described in detail below.

[1250] System Configuration

[1251] 1. User Device

[1252] The user wears an EEG sensor device that collects real-time EEG data. The EEG sensor device is non-invasive and does not interfere with the user's daily activities.

[1253] 2. Terminal

[1254] The terminal collects EEG data from the EEG sensor device and transmits it to a server via a secure connection. The terminal can also temporarily store data, minimizing data loss.

[1255] 3. Server

[1256] The server receives the electroencephalogram data transmitted from the terminal and performs the following processing.

[1257] Pretreatment

[1258] Denoise, filter, and standardize the data.

[1259] analysis

[1260] The preprocessed data is analyzed using an AI model such as a deep neural network (specifically, Keras) to extract features.

[1261] Summary Generation

[1262] Summarize thoughts based on extracted features, then use a generative AI model to translate the summary into natural language.

[1263] Work order generation

[1264] The summarized thoughts are generated as work instructions and transmitted to the robot.

[1265] Program processing explanation

[1266] The server receives data from the EEG sensor device via a terminal. It then performs noise removal and filtering to generate standardized data. Next, it uses a deep neural network model to analyze the EEG patterns and extract features. Based on these features, it summarizes the user's thoughts via a generative AI model. The summarized content is converted into natural language and then compiled as work instructions. Finally, these work instructions are sent to the robot and executed.

[1267] Hardware and software used

[1268] Hardware

[1269] Brainwave sensor device

[1270] User device (smartphone or PC)

[1271] server

[1272] Factory Robots

[1273] software

[1274] Python Program

[1275] Deep Neural Network Framework (Keras)

[1276] Data preprocessing library (Scikit-learn)

[1277] A library for HTTP communication between servers (Requests)

[1278] Specific examples

[1279] As a concrete example, when a user is thinking about how to install a new part, the following process takes place. When the user wears an EEG sensor device and begins thinking about how to install the new part, the EEG data is sent to the server via the terminal. The server preprocesses the data and, after analysis, generates a summary such as "Try installing the new part." This summary is sent to the robot as a work instruction, and the settings are automatically changed.

[1280] Prompt Sentence Examples

[1281] Prompt: Generate work instructions based on the following EEG data. The data is in the form of data obtained from a sensor device, which has been denoised and filtered. The resulting data is as follows:

[1282] data:

[1283] [0.35, 0.22, 0.45, 0.18, 0.09, 0.11, 0.34, 0.29, 0.55, 0.44]

[1284] Expected result: A work order to try out the new part installation method.

[1285] This concludes the description of the "Mode for Carrying Out the Invention." This system makes it possible to provide an efficient and safe working environment that reflects the user's awareness in real time.

[1286] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1287] Step 1:

[1288] Acquisition of EEG data

[1289] The user wears an EEG sensor device, which captures EEG data in real time. The captured EEG data is called raw data and is sent to a terminal connected to the sensor device.

[1290] Input: User's brainwaves

[1291] Output: Raw EEG data (unprocessed EEG data)

[1292] Step 2:

[1293] EEG data transfer

[1294] The terminal receives raw wave data acquired from the sensor device and transmits it to the server via a secure protocol. During this time, the data is temporarily stored on the terminal, reducing the risk of data loss even in the event of a network failure.

[1295] Input: Raw wave data

[1296] Output: Raw wave data temporarily saved on the device

[1297] Step 3:

[1298] Data Preprocessing

[1299] The server preprocesses the raw wave data received from the terminal. Preprocessing includes noise reduction, filtering, and standardization. Noise reduction removes unwanted signals and external influences from the raw wave data. Filtering extracts signals in the desired frequency band. Finally, the data is standardized and adjusted to a uniform scale.

[1300] Input: Raw wave data

[1301] Output: Preprocessed EEG data

[1302] Step 4:

[1303] Feature extraction

[1304] The server analyzes the preprocessed EEG data using a deep neural network (using the Keras framework) to extract important features, which are related to the user's thoughts and emotional state from the EEG patterns.

[1305] Input: Preprocessed EEG data

[1306] Output: Feature data

[1307] Step 5:

[1308] Summary of thoughts

[1309] The server uses a generative AI model to summarize the user's thoughts based on the extracted feature data. At this stage, abstract features are converted into concrete language.

[1310] Input: Feature data

[1311] Output: Summary data

[1312] Step 6:

[1313] Natural language translation

[1314] The server converts the summary data into human-understandable natural language sentences using natural language processing (NLP) techniques, a process that generates sentences that are grammatically and contextually appropriate.

[1315] Input: Summary data

[1316] Output: Natural language sentence

[1317] Step 7:

[1318] Generate work orders

[1319] The server converts the generated natural language sentences into work instructions, which generate instructions that describe specific operations. For example, the generated work instruction might be "try installing a new part."

[1320] Input: Natural language sentence

[1321] Output: Work instructions

[1322] Step 8:

[1323] Sending instructions and controlling the robot's movements

[1324] The server sends work instructions to the robot, which then automatically starts working based on these instructions, and the robot performs the work according to the programmed instructions.

[1325] Input: Work Order

[1326] Output: Robot movement

[1327] These are the specific processing steps of the system that realizes this application example. This flow generates work instructions that reflect the user's thoughts in real time, enabling the robot to automatically perform appropriate operations.

[1328] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1329] Understood. Below is a description of the "Mode for carrying out the invention" based on the invention combined with the emotion engine.

[1330] This invention is a system that clearly summarizes and verbalizes a user's thoughts using electroencephalogram (EEG) data and the user's emotions. This system begins operation when the user wears a dedicated EEG sensor device. The sensor device acquires the user's EEG data in real time and transmits it to a server via the terminal. Furthermore, an emotion engine analyzes the user's emotions and reflects them in summarizing the thoughts and generating sentences.

[1331] System Role

[1332] 1. Users

[1333] The user wears an EEG sensor device and provides EEG and emotional data to the system. The sensor device is non-invasive and does not interfere with the user's daily activities.

[1334] 2. Terminal

[1335] The device collects real-time brainwave and emotion data from the sensor device, stores it temporarily, and then transmits the collected data to a server via a secure connection.

[1336] 3. Server

[1337] The server receives the EEG and emotion data sent from the device and performs preprocessing. This preprocessing includes noise removal, filtering, and standardization. It then analyzes the preprocessed data and uses AI models such as deep neural networks to extract features. The emotion engine combines these features with the user's emotion data and runs a generative model that summarizes the user's thoughts. The generated summary is then converted into grammatically and contextually appropriate sentences using natural language processing (NLP) techniques.

[1338] Program processing explanation

[1339] First, the server preprocesses the EEG and emotion data received from the device. This involves noise removal, filtering, and data standardization. Next, the preprocessed data is input into an AI model such as a deep neural network to analyze EEG patterns and extract features. The emotion engine then analyzes the user's emotions based on these features and generates a summary that combines the emotion data. The generated summary is then formatted into a grammatically and contextually appropriate sentence using natural language processing (NLP) techniques. Finally, the server sends the generated sentence to the device, where it is displayed to the user.

[1340] Specific examples

[1341] Example 1: Summary of user ideas

[1342] When a user is thinking of an idea for a new application, a sensor device captures their brainwave and emotion data, which the device then sends to the server. The server removes noise, filters, and standardizes the data. It then uses a deep neural network to analyze the brainwave patterns and extract features. The emotion engine analyzes emotion data such as "excitement" and "anticipation" and, taking these into account, generates a summary such as "I would like a new app with a feature to manage the user's diet." The generated sentence is then sent to the device and displayed to the user.

[1343] Example 2: Communicating with users with physical disabilities

[1344] When a physically disabled user is thinking about their health, a sensor device captures EEG and emotion data, which the device sends to a server. The server performs preprocessing and analyzes the EEG patterns using a deep neural network. The emotion engine analyzes emotion data such as "fatigue" and "anxiety" and generates a summary such as "I feel a bit tired today. I might need to rest." This summary is converted into natural-sounding sentences and sent to the device for display to the user.

[1345] This concludes the description of the "Mode for Carrying Out the Invention." This system allows the user's thoughts to be captured in real time and clearly expressed in words, taking into account their emotions.

[1346] The processing flow will be explained below.

[1347] Understood. Below, I will explain the process of the system combined with the emotion engine step by step, including specific operations.

[1348] Step 1:

[1349] The user wears a dedicated brainwave sensor device that detects brainwave and emotional data in real time.

[1350] Step 2:

[1351] The device collects real-time brainwave and emotional data from the sensor device, which is then temporarily stored and sent to a server via a secure connection.

[1352] Step 3:

[1353] The server receives the EEG data sent from the device and applies a noise reduction algorithm to the received data to remove unwanted signals.

[1354] Step 4:

[1355] The server uses filtering algorithms to emphasize signals in specific frequency bands, and also normalizes the data and converts it into a format suitable for analysis.

[1356] Step 5:

[1357] The server uses a deep neural network to analyze the pre-processed EEG data, analyzing EEG patterns and extracting features.

[1358] Step 6:

[1359] The server uses an emotion engine to analyze the emotion data sent in parallel from the terminals. The emotion engine analyzes the emotion data and extracts emotion features such as "excitement," "anxiety," and "expectation."

[1360] Step 7:

[1361] The server combines the extracted EEG features and emotion features to generate a feature set that takes into account the user's current emotional state.

[1362] Step 8:

[1363] The server uses a generative model to summarize the user's thoughts based on the feature set, and the summary is generated by incorporating sentiment features.

[1364] Step 9:

[1365] The server uses natural language processing (NLP) techniques to convert the summary text into natural language, formatting it into grammatically and contextually appropriate sentences.

[1366] Step 10:

[1367] The server then sends the final natural language sentence to the terminal, allowing the user to see how their thoughts have been summarized and verbalized.

[1368] Step 11:

[1369] The terminal displays the received text to the user. The displayed text reflects the user's current emotional state and is therefore easy for the user to understand.

[1370] These are the specific processing steps of a system that combines an emotion engine. This process enables users to clearly verbalize and understand their own thoughts while taking into account their emotional state.

[1371] Example 2

[1372] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1373] Conventional EEG measurement systems have difficulty accurately summarizing a user's thoughts in real time and expressing them in natural language. Furthermore, these systems do not take emotional data into account, resulting in a lack of contextual interpretation based on the user's emotions. Furthermore, data preprocessing and analysis are insufficient, making it difficult to extract accurate features from noisy data. This leads to the problem of being unable to accurately verbalize the user's true intentions.

[1374] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for preprocessing EEG data and emotion data, means for analyzing the preprocessed EEG data and extracting features, means including an emotion engine for analyzing the emotion data, means including a generative AI model for summarizing thoughts based on the analyzed features and emotion data, and means using natural language processing technology for converting the summarized content into natural language. This makes it possible to integrate the user's EEG data and emotion data in real time, generate a highly accurate summary, and express it appropriately in natural language.

[1375] "Electroencephalograms" are weak electrical signals emitted by nerve cells in the brain.

[1376] A "secure connection" is a means of ensuring the security of communications by encrypting data before sending and receiving it.

[1377] "Preprocessing" refers to the initial processing to improve the accuracy of data analysis, mainly by performing noise removal, filtering, and data standardization.

[1378] A "feature" is useful information extracted from data that can be used for analysis and classification.

[1379] An "emotion engine" is an algorithm or software that analyzes emotional data and estimates a user's emotional state.

[1380] A "generative AI model" refers to an artificial intelligence model that generates text, images, etc. from specific input data.

[1381] "Natural language processing technology" is a general term for technology used to process human language using computers, and involves processes such as sentence generation, grammar analysis, and translation.

[1382] "Device" means hardware and software that collects, stores, and transmits data.

[1383] "Server" means a computer or networked system that receives, processes, analyzes, and outputs data.

[1384] "Real-time" refers to highly immediate processing, in which the entire process from data acquisition to processing is carried out almost simultaneously.

[1385] The present invention is a system that collects and analyzes electroencephalogram (EEG) data and user emotional data to summarize the user's thoughts and express them in natural language. This system begins operation when the user wears a dedicated EEG sensor device.

[1386] User Roles

[1387] The user wears an EEG sensor device on their head. This device is non-invasive and measures brain waves in real time. It is designed not to interfere with the user's daily life, and can steadily collect EEG data even when the user is thinking.

[1388] Device Role

[1389] The device acquires EEG and emotion data from the sensor device in real time, temporarily stores the data in its internal memory, and then encrypts the data and transmits it to the server via a secure connection (e.g., HTTPS protocol).

[1390] Server Roles

[1391] The server receives the data sent from the device and first performs preprocessing such as noise removal, filtering, and data standardization. The received EEG and emotion data is then input into a deep neural network (DNN) to analyze EEG patterns and extract features. An emotion engine then analyzes the emotion data and generates a summary that takes this into account. The generated summary is then converted into grammatically and contextually appropriate text using natural language processing (NLP) technology. The final generated text is then sent to the device and displayed to the user.

[1392] Specific examples

[1393] Example 1: New application idea summary

[1394] When a user is thinking of an idea for a new application, the process is as follows: When the user puts on the device and begins thinking, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "excitement" and "anticipation," and based on that, generates a summary such as "I would like a new app with a feature to manage the user's diet." This sentence is then sent from the server to the device and displayed to the user.

[1395] Example prompt:

[1396] "You input EEG data and emotion data while a user is thinking about a new app idea, and generate a summary."

[1397] Example 2: Communicating with users with physical disabilities

[1398] When a physically disabled user is thinking about their health, the process is as follows: When the user puts on the device and begins to think about their health, the device acquires brainwave and emotion data, temporarily stores it, and securely transmits it to the server. The server receives the data, preprocesses it, and analyzes it using a DNN model to extract features. The emotion engine then analyzes emotions such as "fatigue" and "anxiety," and generates a summary based on that, such as "I feel a little tired today. I might need to rest." This sentence is then sent from the server to the device and displayed to the user.

[1399] Example prompt:

[1400] "Please input EEG data and emotion data of physically disabled users while they are thinking about their physical condition and generate a summary."

[1401] This invention makes it possible to acquire a user's thoughts in real time and verbalize them after taking into account emotional data, thereby enabling the user to clearly and quickly express their thoughts and ideas in their daily lives.

[1402] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1403] Understood. Below, I will explain the process flow of the system program in detail, divided into steps.

[1404] Step 1:

[1405] The user wears a dedicated brainwave sensor device on their head. This device is non-invasive and measures brainwaves in real time. Specifically, the device uses an antenna to capture electrical signals emitted from the brain and converts them into digital signals using a built-in sensor. The input is the user's brainwave signal, and the output is digitized brainwave data.

[1406] Step 2:

[1407] The terminal acquires EEG data and emotion data from the sensor device in real time and temporarily stores it in the terminal's memory. Here, it receives the data sent from the sensor device and stores it in a designated memory area. The input is the digitized EEG data and emotion data from the sensor device, and the output is the data stored in the terminal's memory.

[1408] Step 3:

[1409] The device encrypts the stored data and sends it over a secure connection to a server, where an encryption algorithm is applied and the data is sent using the HTTPS protocol. The input is digital data stored in the device's memory, and the output is encrypted data.

[1410] Step 4:

[1411] The server decrypts and preprocesses the encrypted data received from the device. The received data is first decrypted, then denoised, filtered and normalised. A filtering algorithm is applied to remove noise, and the filtered data is normalised to a reference value. The input is the encrypted data, and the output is the preprocessed data.

[1412] Step 5:

[1413] The server inputs the preprocessed data into a deep neural network (DNN) to analyze the EEG patterns and extract features. Here, the input data is fed forward to the DNN model to extract features. The input is the preprocessed data, and the output is feature data.

[1414] Step 6:

[1415] The emotion engine analyzes the user's emotion data based on the extracted features. Specifically, the emotion engine algorithm evaluates the features and estimates the user's emotional state. The inputs are feature data and emotion data, and the output is the analyzed emotion data.

[1416] Step 7:

[1417] The server runs a generative AI model based on the analyzed features and emotion data to summarize the user's thoughts. The generative AI model is an algorithm that generates a text summary from the features. The input is the analyzed features and emotion data, and the output is the summary text.

[1418] Step 8:

[1419] NLP technology is used to convert the summary text into grammatically and contextually appropriate natural language. The NLP model analyzes the summary text and optimizes it for natural sentences. The input is the summary text, and the output is well-formatted natural language sentences.

[1420] Step 9:

[1421] Finally, the server sends the generated natural language text to the terminal and displays it to the user. Here, the server encrypts the text and sends it securely to the terminal. The terminal decrypts the received text and displays it on the screen. The input is a formatted natural language text, and the output is the text displayed on the terminal.

[1422] (Application example 2)

[1423] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1424] In today's brick-and-mortar stores, it is difficult for customers to quickly find products that match their interests and emotions. Furthermore, communication with store staff can be difficult, making the shopping experience stressful. To address this issue, there is a need for a system that uses a customer's brainwave and emotional data to summarize what the customer is thinking and suggest appropriate products.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1426] In this invention, the server includes means for acquiring electroencephalogram data in real time, means for preprocessing the acquired electroencephalogram data, means for analyzing the preprocessed electroencephalogram data and emotion data to extract features, means for summarizing thoughts based on the extracted features and emotion data, means for converting the summarized content into natural language, means for outputting the converted natural language sentence, and means for providing the summarized content to the user via the smart device, thereby enabling product recommendations based on the user's interests and emotions or summarizing current thoughts in real time to improve the shopping experience.

[1427] "Electroencephalogram data" is data that records the electrical activity of the brain and is used to analyze specific mental states and emotions.

[1428] "Real-time" refers to acquiring and processing current conditions and data without delay.

[1429] "Emotion data" is data that quantifies the user's psychological state and emotions, and is information that indicates the user's excitement, interest, anxiety, etc.

[1430] "Features" are important patterns or characteristics that the AI ​​model extracts from EEG and emotion data, and are the information used to generate a summary of thoughts.

[1431] A "summary" is information that succinctly and concisely expresses the user's thoughts and feelings, and is a concise version of a long piece of text.

[1432] A "natural language" is a language that humans use on a daily basis, and is a sentence that is constructed in a meaningful way based on grammar and vocabulary.

[1433] A "smart device" is an electronic device equipped with Internet connectivity and applications, and capable of two-way communication with users.

[1434] "Preprocessing" refers to the process performed to prepare raw data for analysis, and includes noise removal and data standardization.

[1435] "Conversion" refers to the conversion of data from one form to another, and in this case refers to the conversion of summarized content into natural language.

[1436] "Output" refers to the system providing the final processing results to the user, and refers to the act of displaying information.

[1437] This invention is a system that uses EEG and emotion data to summarize a user's thoughts and convert them into natural language. The system is designed to provide interest- and emotion-based product recommendations to customers in brick-and-mortar stores.

[1438] The system is configured as follows:

[1439] 1. User:

[1440] The user wears smart glasses and an EEG sensor device, which collects the user's brainwave and emotional data in real time. The EEG sensor device is non-invasive and does not interfere with the user's daily activities. The smart glasses display information in the user's field of vision.

[1441] 2. Terminal:

[1442] The device collects and temporarily stores the EEG and emotional data received from the user, then transmits the collected data to a server over a secure connection. The device also performs pre-processing and filtering of the data.

[1443] 3. Server:

[1444] The server receives the data sent from the device and performs preprocessing, which includes noise removal, filtering, and data standardization. The server implements deep neural network analysis and natural language processing techniques, such as:

[1445] Data Analysis:

[1446] The server inputs the preprocessed EEG data into a deep neural network to extract features, and analyzes the emotion data to identify the user's psychological state.

[1447] Summarize and generate:

[1448] Based on the extracted features and emotion data, the server uses a generative AI model to summarize the user's thoughts and converts the summary into natural language. This sentence generation is achieved using natural language processing (NLP) techniques.

[1449] output:

[1450] The server sends the generated text to the terminal and displays it to the user through the smart glasses.

[1451] The hardware used includes an EEG sensor device, smart glasses, devices (such as smartphones and tablets), and a server, while the software includes deep neural networks (such as TensorFlow and Keras), natural language processing techniques, and data preprocessing libraries (such as SciPy and NumPy).

[1452] Examples:

[1453] scenario:

[1454] A user looking for electronic devices at a shopping mall.

[1455] Example prompt sentence:

[1456] Currently, the user's brainwave and emotional data are as follows: The brainwave data indicates high excitement and anticipation. The emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[1457] Produces a summary:

[1458] "I see you're interested in a new smartphone. Here's the latest model."

[1459] This system improves the user's shopping experience and enables efficient product recommendations.

[1460] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1461] Step 1:

[1462] The user wears smart glasses and a brainwave sensor device.

[1463] The user wears the smart glasses and an EEG sensor device to begin capturing EEG and emotional data. The EEG sensor device records the brain's electrical activity in real time, and the smart glasses are used to display additional information.

[1464] Step 2:

[1465] The device collects and temporarily stores brainwave and emotional data.

[1466] The terminal collects and temporarily stores the EEG and emotional data sent from the EEG sensor device. The input is raw EEG and emotional data, and the output is the temporarily stored data. Specifically, the terminal collects data using wireless communication such as Bluetooth or Wi-Fi.

[1467] Step 3:

[1468] The terminal preprocesses the collected data and sends it to the server.

[1469] The device filters the collected EEG data, removes noise, and standardizes the data. The input is the temporarily stored data, and the output is the preprocessed data. Specifically, data preprocessing is performed using libraries such as SciPy and NumPy. The preprocessed data is then sent to the server via a secure connection.

[1470] Step 4:

[1471] The server inputs the received data into a deep neural network and extracts features.

[1472] The server receives preprocessed EEG data and emotion data sent from the device. The input is the preprocessed data, and the output is features. Specifically, it uses TensorFlow and Keras to analyze EEG patterns and extract features.

[1473] Step 5:

[1474] The server summarizes thoughts based on feature and emotional data and converts them into natural language.

[1475] The server runs a generative AI model that summarizes thoughts based on the extracted features and emotion data. It then uses natural language processing techniques to convert the summarized content into grammatically and contextually appropriate sentences. The input is feature and emotion data, and the output is natural language sentences. Specific operations use NLP libraries (e.g., SpaCy and Hugging Face Transformers).

[1476] Step 6:

[1477] The server sends the generated text to the terminal and displays it through the smart glasses.

[1478] The server sends the generated natural language sentence to the device. The input is the generated sentence, which the device receives and displays to the user through the smart glasses. The output is the information displayed on the smart glasses. Specific operations include sending data using WebSocket or REST API.

[1479] Example prompt sentence:

[1480] Currently, the user's EEG data indicates high excitement and anticipation, while the emotional data shows an excitement score of 0.8 and a curiosity score of 0.6.

[1481] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1482] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1483] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1484] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1485] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1486] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1487] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1488] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1489] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1490] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1491] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1492] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1493] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1494] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1495] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1496] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1497] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1498] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1499] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1500] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1501] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1502] The following is further disclosed regarding the above embodiment.

[1503] Understood. Below are some proposed draft patent claims for an AI system that reads thoughts from brain waves and verbalizes them.

[1504] (Claim 1)

[1505] a means for acquiring EEG data in real time;

[1506] means for pre-processing the acquired electroencephalogram data;

[1507] means for analyzing the preprocessed electroencephalogram data and extracting features;

[1508] A means for summarizing thoughts based on the extracted features;

[1509] a means for converting the summarized content into natural language;

[1510] The system includes means for outputting the converted natural language sentence.

[1511] (Claim 2)

[1512] a means for removing noise from the EEG data;

[1513] means for filtering the electroencephalogram data;

[1514] 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

[1515] (Claim 3)

[1516] 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network.

[1517] The above is the draft of the patent claim for an AI system that reads thoughts from brain waves and verbalizes them.

[1518] "Example 1"

[1519] (Claim 1)

[1520] A means for a user to wear an EEG sensor device and acquire EEG data in real time;

[1521] means for transmitting the acquired electroencephalogram data to a server via a terminal;

[1522] A means for the server to preprocess the transferred electroencephalogram data;

[1523] A means for inputting the preprocessed EEG data into a deep learning model to extract features;

[1524] A means of summarizing thoughts using a generative AI model based on extracted features, and

[1525] A means for converting the summarized content into natural language using natural language processing technology;

[1526] The system includes a means for transmitting the converted natural language sentence to a terminal for output.

[1527] (Claim 2)

[1528] a means for removing noise from the EEG data;

[1529] means for filtering the electroencephalogram data;

[1530] 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

[1531] (Claim 3)

[1532] 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network.

[1533] "Application Example 1"

[1534] New Claims

[1535] Rewrite it according to the following format:

[1536] (Claim 1)

[1537] a means for acquiring EEG data in real time;

[1538] means for pre-processing the acquired electroencephalogram data;

[1539] means for analyzing the preprocessed electroencephalogram data and extracting features;

[1540] A means for summarizing thoughts based on the extracted features;

[1541] a means for converting the summarized content into natural language;

[1542] means for outputting the converted natural language sentence;

[1543] a means for transmitting the summarized thoughts to the robot as work instructions;

[1544] A system including a means for controlling the operation of a robot based on a work instruction.

[1545] (Claim 2)

[1546] a means for removing noise from the EEG data;

[1547] means for filtering the electroencephalogram data;

[1548] 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

[1549] (Claim 3)

[1550] 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network.

[1551] "Example 2: Combining Emotion Engines"

[1552] Understood. Below are the claims that take into account the new invention.

[1553] (Claim 1)

[1554] a means for acquiring EEG data in real time;

[1555] A means for temporarily storing the acquired brain wave data and emotion data in a terminal;

[1556] means for transmitting data to a server over a secure connection;

[1557] means for pre-processing the received electroencephalogram data and emotion data;

[1558] means for analyzing the preprocessed electroencephalogram data and extracting features;

[1559] means including an emotion engine for analyzing emotion data;

[1560] a means including a generative AI model that summarizes thoughts based on the analyzed features and emotion data;

[1561] a means for using natural language processing techniques to convert the summarized content into natural language;

[1562] The system includes means for outputting the converted natural language sentence to a user.

[1563] (Claim 2)

[1564] a means for removing noise from the EEG data;

[1565] means for filtering the electroencephalogram data;

[1566] 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

[1567] (Claim 3)

[1568] 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network.

[1569] Above are the revised claims that include details of the new system.

[1570] "Application example 2 when combining emotion engines"

[1571] (Claim 1)

[1572] a means for acquiring EEG data in real time;

[1573] means for pre-processing the acquired electroencephalogram data;

[1574] means for analyzing the preprocessed electroencephalogram data and emotion data to extract features;

[1575] A means for summarizing thoughts based on the extracted features and emotion data;

[1576] a means for converting the summarized content into natural language;

[1577] means for outputting the converted natural language sentence;

[1578] A system including means for providing summarized content to a user via a smart device.

[1579] (Claim 2)

[1580] a means for removing noise from the EEG data;

[1581] means for filtering the electroencephalogram data;

[1582] 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

[1583] (Claim 3)

[1584] 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network. [Explanation of symbols]

[1585] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring EEG data in real time; means for pre-processing the acquired electroencephalogram data; means for analyzing the preprocessed electroencephalogram data and extracting features; A means for summarizing thoughts based on the extracted features; a means for converting the summarized content into natural language; The system includes means for outputting the converted natural language sentence.

2. a means for removing noise from the EEG data; means for filtering the electroencephalogram data; 10. The system of claim 1, further comprising means for normalizing the pre-processed electroencephalogram data.

3. 10. The system of claim 1, further comprising means for analyzing electroencephalogram patterns using a deep neural network.

Images