System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring physiological data; means for analyzing the acquired physiological data to estimate an emotional state; means for generating music based on the estimated emotional state; and means for providing the generated music to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As stress and loneliness increase in modern society, individuals, especially remote workers and freelancers, need means of relaxation and energy enhancement tailored to their emotional state. To address this issue, a system is needed that can accurately grasp a user's real-time emotional state and provide appropriate music. However, existing technologies lack a means to accurately estimate a user's emotional state and generate and provide personalized music in real time based on the results. This poses a challenge in supporting users' psychological wellness, improving their quality of life, reducing stress, and increasing productivity. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means. The present invention is a system including means for acquiring biometric data, means for analyzing the acquired biometric data and estimating an emotional state, means for generating music based on the estimated emotional state, and means for providing the generated music to a user. The present invention is also a system including means for acquiring audio data, means for analyzing the acquired audio data and estimating an emotional state, means for generating music based on the estimated emotional state, and means for providing the generated music to a user. The present invention is also a system including means for customizing music based on the estimated emotional state in consideration of the user's preferences and cultural background, and means for generating customized music.

[0006] This makes it possible to effectively provide music that matches the user's real-time emotional state and provide psychological support. By accurately estimating the user's emotional state and generating and providing personalized music based on the results, it is possible to contribute to long-term improvements in psychological state, such as reducing stress, improving quality of life, and increasing productivity.

[0007] "Biometric data" refers to data related to a person's physical condition, such as heart rate, blood pressure, and activity level.

[0008] "Voice data" refers to data acquired as sound waves, such as the content of a user's speech or voice tone.

[0009] An "emotional state" is the state of mind that the user is currently feeling, and includes psychological states such as joy, sadness, anger, and relaxation.

[0010] "Inference" refers to predicting the user's emotional state based on acquired data.

[0011] "Generation" refers to creating music based on an estimated emotional state.

[0012] "Providing" refers to distributing or playing the generated music in a form that is available to users.

[0013] "Real-time" refers to processing that occurs almost instantly, with almost no delay between data acquisition and provision.

[0014] "User preferences" refers to the user's favorite music genres, artists, music styles, etc.

[0015] "Cultural background" refers to the unique musical characteristics and preferences based on the culture or region to which the user belongs.

[0016] "Personalization" refers to providing music and services in a way that is optimized for each individual user. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0039] Program processing

[0040] Data collection

[0041] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0042] Data Transmission and Encryption

[0043] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0044] sentiment analysis

[0045] The server analyzes the received data. It uses natural language processing (NLP) algorithms to infer emotions from the voice data. At the same time, it analyzes biometric data through machine learning models to infer emotional states. These results are then combined to determine the final emotional state.

[0046] Music Generation

[0047] Based on the results of the sentiment analysis, the server uses a generative AI model to generate music. This takes into account the user's preferences and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0048] Music provided by

[0049] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. Even during this process, the music is provided in real time, allowing users to instantly enjoy music that matches their emotional state.

[0050] Long-term analysis and feedback

[0051] The server tracks daily emotional fluctuations and stores them in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user.

[0052] Specific examples

[0053] For example, if a user is analyzed as being in a high stress state while working remotely, the server will generate relaxation music. Since stress can be detected from the voice data, the generated music is a highly relaxing instrumental piece tailored to the user's preferences. The device then plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0054] This system can effectively support users' psychological wellness, contributing to improving quality of life, reducing stress, and increasing productivity.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] User

[0058] The user wears a smartwatch, which measures heart rate, blood pressure, activity, and audio data in real time. Data is collected continuously as the user goes about their daily activities.

[0059] Step 2:

[0060] Smartwatch

[0061] The smartwatch transmits the collected biometric and audio data to a device (e.g., a smartphone) using wireless communication such as Bluetooth.

[0062] Step 3:

[0063] Terminal

[0064] The device temporarily stores the data received from the smartwatch in its memory, encrypts the stored data using an encryption method (e.g., AES-256), and periodically transmits the encrypted data to the server.

[0065] Step 4:

[0066] server

[0067] The server receives the encrypted data sent from the device. After receiving it, it decrypts the data and verifies its consistency and integrity. After verification, the data is ready for analysis.

[0068] Step 5:

[0069] server

[0070] The server analyzes the acquired data. It uses a natural language processing (NLP) algorithm to extract keywords and emotional tones from the voice data and estimate the emotional state. At the same time, it uses a machine learning model to analyze biometric data (heart rate, blood pressure, activity level) to estimate the emotional state. The analysis results of the voice data and biometric data are integrated to determine the final emotional state.

[0071] Step 6:

[0072] server

[0073] The server generates music using a generative AI model based on the estimated emotional state, taking into account the user's past music history, preferences, and cultural background. The generated music data is then re-encrypted and sent to the device.

[0074] Step 7:

[0075] Terminal

[0076] The device decrypts the encrypted music data received from the server, passes the decrypted music data to the media player, and prepares it for playback. The device then connects to a smartphone or Bluetooth speaker for playback.

[0077] Step 8:

[0078] User

[0079] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0080] Step 9:

[0081] server

[0082] The server tracks daily emotional data and stores it in a database. It then performs long-term analysis to extract trends and patterns, and creates reports to help users improve their psychological state.

[0083] Step 10:

[0084] Terminal

[0085] The device receives the report from the server and provides it to the user, who can then view the report in the application to check long-term trends in their psychological state and take steps to improve their daily habits and work environment.

[0086] Through these specific processing steps, the system of the present invention can provide music that corresponds to the user's real-time emotional state and provide psychological support.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional emotional state estimation systems that use biometric and audio data have difficulty accurately analyzing a user's unique emotional state and generating and providing appropriate music. Furthermore, they lack the ability to grasp a user's long-term emotional trends and provide feedback. This makes it difficult to comprehensively support users' psychological wellness.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for acquiring biometric data, means for analyzing the acquired biometric data and transmitting it to the server using an encryption protocol, means for integrating the analyzed biometric data with audio data to estimate the user's emotional state, means for generating music based on the estimated emotional state using a generative AI model, means for transmitting the generated music to the terminal and providing it to the user, and means for analyzing the user's long-term emotional trends and patterns and providing feedback. This allows the system to accurately analyze the user's unique emotional state and provide appropriate music in real time. It also provides feedback to support the user's long-term psychological wellness.

[0092] "Biometric data" refers to data that indicates the user's physical condition in real time, such as heart rate, blood pressure, and activity level.

[0093] "Encryption protocol" refers to a standard method used to protect the confidentiality of data, such as AES-256.

[0094] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0095] "Inference tools" refer to algorithms or machine learning models that analyze collected data and derive specific results.

[0096] A "generative AI model" is an artificial intelligence model that generates a specific output (in this case, music) by taking into account the user's emotional state and preferences.

[0097] A "terminal" is a device used by a user, such as a smartphone or tablet, that transmits, receives, and displays data.

[0098] "Emotional state" is an indicator of the user's psychological and emotional state, and includes states such as stress and relaxation.

[0099] "Analyzing" is the process of examining collected data in detail and extracting specific information or patterns.

[0100] "Integrating" refers to combining information from multiple data sources to gain a holistic understanding and analysis.

[0101] "Feedback" means information or advice provided to users that is based on past data and analysis.

[0102] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0103] Data collection

[0104] The user wears a smartwatch, which collects real-time biometric data such as heart rate, blood pressure, and activity level. The watch also collects the user's voice via the smartphone, and this data is temporarily stored on the smartphone.

[0105] Data Transmission and Encryption

[0106] The smartphone device encrypts the collected biometric and voice data using the AES-256 encryption protocol, and then transmits the encrypted data to a server, which verifies the integrity and completeness of the received data.

[0107] sentiment analysis

[0108] The server analyzes the received biometric data using a machine learning model to estimate the user's emotional state. This model is trained from past data and can recognize stress and relaxation. The server also analyzes the voice data using an NLP (natural language processing) algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to determine the user's final emotional state.

[0109] Music Generation

[0110] The server then uses a generative AI model to generate personalized music based on the results of the sentiment analysis. This model also takes into account the user's musical history and cultural background. The generated music data is then re-encrypted and sent to the device.

[0111] Music provided by

[0112] The device decodes the received music data and passes it to the media player, allowing users to enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0113] Long-term analysis and feedback

[0114] The server tracks daily emotional fluctuations and stores them in a database. The stored data is then analyzed to extract trends and patterns in the user's emotional state. The server then creates a report based on the analysis results and sends it to the device. The device receives this report and provides it to the user, allowing the user to understand the history of their own emotional state.

[0115] Specific examples

[0116] For example, if a user is analyzed as being in a state of high stress while working remotely, the server will generate relaxation music. The generative AI model will select instrumental songs that the user has previously found relaxing and create new music based on these. The device will receive the generated music data and pass it to the user's media player. The user can immediately listen to the relaxation music and reduce stress.

[0117] Prompt Sentence Examples

[0118] "Please explain the system that generates and provides appropriate relaxation music when it is analyzed that the user is in a high stress state while working remotely. Specifically, please describe what data you collect, what technology you use to perform emotion analysis, and how you use the generative AI model."

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1: Data collection

[0121] The user wears a smartwatch, which collects biometric data such as heart rate, blood pressure, and activity level in real time. At the same time, the user's voice is also collected via the smartphone. This data is temporarily stored on the smartphone. The input is the user's biometric data and voice data, and the output is the data stored on the smartphone.

[0122] Step 2: Data transmission and encryption

[0123] The terminal, a smartphone, receives the acquired biometric data and voice data as input. The smartphone encrypts these data using the AES-256 encryption protocol. The encrypted data is then sent to the server. The output is encrypted data, which is sent to the server. The server verifies the integrity and completeness of the received data.

[0124] Step 3: Sentiment analysis

[0125] The server analyzes the received biometric data using a machine learning model. The input is encrypted biometric data and voice data. The server first decrypts it and then analyzes it to estimate the user's emotional state. A model trained on past data is used to analyze the biometric data. At the same time, the voice data is analyzed using an NLP algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to obtain the final emotional state as the output.

[0126] Step 4: Music Generation

[0127] The server generates personalized music using a generative AI model based on the results of the sentiment analysis. The input is the integrated emotional state, the user's music history, and cultural background. The generative AI model takes into account the user's past data and generates new music. The generated music data is re-encrypted and sent to the device. The output is encrypted music data.

[0128] Step 5: Submit your music

[0129] The device decrypts the received music data and passes it to the media player. The input is encrypted music data, and the output is decrypted music data. Users can enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0130] Step 6: Long-term analysis and feedback

[0131] The server tracks daily emotional fluctuations and stores them in a database. The input is daily emotional data, and the output is the stored data. The server analyzes the stored data and extracts trends and patterns in the user's emotional state. The server creates a report based on the analysis results and sends it to the device. The device receives this and provides it to the user. The output is a report of the analysis results.

[0132] Through this processing step, the system can accurately analyze a user's unique emotional state and provide appropriate music in real time, as well as provide feedback to support the user's long-term psychological wellness.

[0133] (Application example 1)

[0134] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Conventional food delivery systems have struggled to provide an individually customized experience that takes into account the user's emotional state. The user experience, particularly during the ordering process and waiting time, is limited, and there are few ways to alleviate the stress and anxiety felt by users. As a result, user satisfaction tends to decline. To solve this issue, it is necessary to provide new added value based on the user's emotional state.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0137] In this invention, the server includes a means for providing music during the ordering process or waiting time according to the emotional state of the user, a means for acquiring voice data, and a means for analyzing the acquired voice data and estimating the emotional state, thereby providing an individually customized music experience and reducing stress during the waiting time or ordering process while caring for the user's emotions.

[0138] "Biometric data" refers to data that indicates physical conditions such as heart rate, blood pressure, and activity level obtained through smartwatches and other wearable devices.

[0139] "Voice data" refers to data that records the user's speech and surrounding sounds acquired via a smartphone or other voice input device.

[0140] "Emotional state" refers to the user's psychological state or mood that is estimated based on the analyzed biometric and audio data.

[0141] "Means for generating music" refers to a method or algorithm that creates music to make the user feel a particular emotion, such as relaxation or concentration, based on the acquired emotional state.

[0142] "Means for providing to a user" refers to a method or system for transmitting the generated music to a user's device so that the user can listen to it.

[0143] "Means for providing music during the ordering process and while waiting" refers to a method or system within a food delivery application that plays appropriate music while a user is placing an order or waiting for food to be delivered.

[0144] "Customization means" refers to methods and algorithms that individually tailor the content and style of music to suit the user's emotional state, taking into account the user's preferences and cultural background.

[0145] The present invention is a system that provides individually customized music based on the user's emotional state, and its application example is to improve the user experience in food delivery applications.

[0146] In particular, the server analyzes the user's emotional state sequentially and generates and provides music that matches that state. To achieve this, the following hardware and software are used in combination:

[0147] Hardware and Software Overview

[0148] 1. Smartwatch:

[0149] Used to obtain biometric data (heart rate, blood pressure, activity level) in real time.

[0150] 2. Smartphone:

[0151] Biometric and audio data is temporarily stored and transmitted to a cloud server using an encryption protocol.

[0152] Used to play provided music to the user.

[0153] 3. Cloud Server:

[0154] The server that receives and analyzes the data encrypts it and verifies its integrity and completeness.

[0155] Sentiment estimation is performed using natural language processing (NLP) algorithms and machine learning models.

[0156] Generate music using generative AI models.

[0157] The specific software used is as follows:

[0158] PyCryptodome (used to encrypt data)

[0159] Google® Cloud Speech-to-Text API (used for converting voice data to text)

[0160] HuggingFace Transformers (used for NLP analysis of voice data)

[0161] scikit-learn (used for machine learning model analysis of biological data)

[0162] VLC (used to play music on smartphones)

[0163] Detailed system processing flow

[0164] 1. Data Collection:

[0165] When a user wears a smartwatch and connects it to a smartphone, biometric and audio data are collected in real time.

[0166] 2. Data Transmission and Encryption:

[0167] The collected data is encrypted on the smartphone using an encryption protocol (AES-256) and sent to a cloud server, which verifies the data's integrity and completeness.

[0168] 3. Sentiment analysis:

[0169] The cloud server converts the voice data into text using the Google Cloud Speech-to-Text API and estimates the emotional state using HuggingFace Transformers.

[0170] The biometric data is fed into a machine learning model using scikit-learn to analyze the emotional state, and the final emotional state is determined by integrating these data.

[0171] 4. Music Generation:

[0172] Based on the analysis of the emotional state, prompt sentences are provided to a generative AI model (e.g., GPT-3 (registered trademark)) to generate music.

[0173] Examples of prompts:

[0174] Emotional state: Stress

[0175] User Preferences: Jazz

[0176] User's cultural background: Japan

[0177] Type of music generated: Relaxing jazz tunes

[0178] 5. Music provided by:

[0179] The generated music is then re-encrypted and sent to a smartphone, where users can listen to it via their smartphone or Bluetooth speaker while waiting or ordering.

[0180] Specific examples

[0181] For example, if a food delivery application determines that a user is in a high stress state, the cloud server generates relaxing jazz music. This music is customized taking into account the user's preferences and cultural background. By listening to this music, users can reduce stress and spend the waiting time more comfortably after placing their order.

[0182] This system allows users to receive appropriate music in real time according to their emotional state, thereby improving satisfaction with food delivery services.

[0183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0184] Step 1:

[0185] The user wears a smartwatch, and biometric data (heart rate, blood pressure, activity level) is collected in real time via a smartphone. This data is temporarily stored on the smartphone. The input is biometric data, and the output is the biometric data stored on the smartphone.

[0186] Step 2:

[0187] Audio data is collected via the smartphone's microphone. This includes the user's speech and surrounding sounds. This audio data is also stored on the smartphone. The input is audio data, and the output is the audio data stored on the smartphone.

[0188] Step 3:

[0189] The smartphone encrypts the collected biometric and audio data using the AES-256 protocol, ensuring data security: the input is unencrypted data and the output is encrypted data.

[0190] Step 4:

[0191] The encrypted data is sent from the smartphone to a cloud server, which verifies the integrity and completeness of the received data. The input is the encrypted data, and the output is the verified data stored on the server.

[0192] Step 5:

[0193] The cloud server uses the Google Cloud Speech-to-Text API to convert the voice data into text data, which is then used for sentiment analysis. The input is the voice data, and the output is the converted text data.

[0194] Step 6:

[0195] The server uses HuggingFace Transformers to perform natural language processing (NLP) on the transformed text data to estimate the emotional state. The input is the text data, and the output is the estimated emotional state.

[0196] Step 7:

[0197] The server inputs the biometric data into a machine learning model based on scikit-learn to analyze the emotional state. This integrates emotion estimation using natural language processing with biometric data analysis to determine the final emotional state. The input is the biometric data, and the output is the integrated emotional state.

[0198] Step 8:

[0199] The server generates music by inputting a prompt sentence into a generative AI model based on the estimated emotional state. GPT-3 or another model is used as the generative AI model. The input is the prompt sentence, and the output is the generated music data. Specific examples of prompt sentences are as follows:

[0200] Emotional state: Stress

[0201] User Preferences: Jazz

[0202] User's cultural background: Japan

[0203] Type of music generated: Relaxing jazz tunes

[0204] Step 9:

[0205] The generated music data is again encrypted using the AES-256 protocol and sent to the smartphone. The input is unencrypted music data, and the output is encrypted music data.

[0206] Step 10:

[0207] The smartphone receives the encrypted music data, decrypts it, and passes it to the media player. Specifically, the music is played via the VLC library. This allows the user to listen to music optimized for their emotional state in real time. The input is encrypted music data, and the output is decrypted music data.

[0208] By going through the above processing steps, a food delivery application can provide a personalized music experience that corresponds to the user's emotional state.

[0209] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0210] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and audio data, uses an emotion engine to accurately recognize the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0211] Program processing

[0212] Data collection

[0213] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0214] Data Transmission and Encryption

[0215] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0216] Emotion recognition with sentiment analysis and emotion engine

[0217] The server analyzes the received data. The emotion engine combines biometric data and voice data to accurately recognize the emotional state. Natural language processing (NLP) algorithms are used to extract keywords and emotional tones from the voice data to estimate the emotional state. At the same time, the biometric data is also analyzed using a machine learning model to estimate the emotional state. Based on the results of these analyses, the emotion engine determines the final emotional state.

[0218] Music Generation

[0219] Based on the emotional state recognized by the emotion engine, the server uses a generative AI model to generate music. This generation also takes into account the user's past music history, preferences, and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0220] Music provided by

[0221] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. This provides music that responds to the user's emotional state in real time, supporting psychological wellness.

[0222] Long-term analysis and feedback

[0223] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user. The user can view the reports within the application to check long-term trends in their psychological state.

[0224] Specific examples

[0225] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0226] This invention makes it possible to effectively provide music that responds to the user's real-time emotional state and provide psychological support, which is designed to contribute to stress reduction, improved quality of life, and increased productivity.

[0227] The processing flow will be explained below.

[0228] Program processing

[0229] Step 1:

[0230] User

[0231] The user puts on the smartwatch and begins their daily activities, and the smartwatch measures heart rate, blood pressure, activity level, and audio data in real time.

[0232] Step 2:

[0233] Smartwatch

[0234] The smartwatch transmits the biometric and audio data it measures to the device (smartphone) via wireless communication (such as Bluetooth).

[0235] Step 3:

[0236] Terminal

[0237] The device temporarily stores the data received from the smartwatch in its memory, and at the same time encrypts the data (using AES-256, etc.) and prepares it for transmission to the server.

[0238] Step 4:

[0239] Terminal

[0240] The encrypted data is sent to the server in real time, ensuring data security.

[0241] Step 5:

[0242] server

[0243] The server decrypts the encrypted data received from the device and verifies the integrity and completeness of the data, after which it is ready for analysis.

[0244] Step 6:

[0245] server

[0246] The server analyzes the received data. The emotion engine comprehensively analyzes the voice data and biometric data to recognize the user's emotional state. Natural language processing (NLP) algorithms are used to extract emotional tones and keywords from the voice data, and machine learning models are used to estimate the user's emotional state from the biometric data.

[0247] Step 7:

[0248] server

[0249] Based on the analysis results of the emotion engine, the server determines the user's emotional state, for example, determining that the user is in a high stress state.

[0250] Step 8:

[0251] server

[0252] The server uses a generative AI model to generate music that best suits the user's emotional state, taking into account the user's preferences and cultural background. The generated music data is then re-encrypted.

[0253] Step 9:

[0254] server

[0255] The server sends the encrypted music data to the terminal, which then transmits it immediately.

[0256] Step 10:

[0257] Terminal

[0258] The device decrypts the encrypted music data received from the server, and the decrypted music data is passed to the media player.

[0259] Step 11:

[0260] Terminal

[0261] The device connects to the user's smartphone or Bluetooth speaker and prepares to play music.

[0262] Step 12:

[0263] User

[0264] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0265] Step 13:

[0266] server

[0267] The server tracks daily emotional data and stores it in a database, which allows for long-term data analysis.

[0268] Step 14:

[0269] server

[0270] Analyze the accumulated data, extract trends and patterns, and create reports for users to support long-term psychological improvement.

[0271] Step 15:

[0272] Terminal

[0273] The device receives the report from the server and provides it to the user, who can then view the report through the application to check trends and areas for improvement in their own psychological state.

[0274] In this way, the system of the present invention can provide optimal music based on the user's real-time emotional state and provide psychological support.

[0275] Example 2

[0276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] In modern society, people's stress and psychological burden are increasing, but a system for effectively alleviating these stresses and providing users with music that suits their needs has not yet been fully established. Conventional systems have difficulty providing personalized music based on each user's real-time emotional state. There is a need to solve this problem and improve users' psychological wellness.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0279] In this invention, the server includes means for acquiring biometric data from a wearable device worn by a user, means for temporarily storing the acquired biometric data in the terminal and transmitting it to the server using an encryption protocol, means for analyzing the encrypted data and estimating an emotional state using an emotion engine, means for generating music using a generative AI model based on the estimated emotional state, means for encrypting the generated music data and transmitting it to the terminal, means for playing the decrypted music on the terminal via a media player and providing it to the user, and means for tracking and analyzing daily emotional data over the long term and providing feedback to the user. This makes it possible to provide personalized music based on each user's real-time emotional state, which is expected to help relieve stress and improve psychological well-being.

[0280] "Biometric data" refers to data related to the user's physical condition, including heart rate, blood pressure, activity level, and the like.

[0281] A "wearable device" is an electronic device worn by a user, including smartwatches and fitness trackers.

[0282] "Encryption protocols" are technologies used to ensure data security, including algorithms such as AES-256.

[0283] An "emotion engine" is a computer system or algorithm that analyzes biometric and audio data to accurately estimate a user's emotional state.

[0284] A "generative AI model" is an artificial intelligence model that uses machine learning to perform specific tasks, and is used for applications such as music generation and text generation.

[0285] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, tablets, etc.

[0286] A "server" is a centralized computer system for processing and storing data over a network.

[0287] "Voice data" refers to data that records the user's speaking voice or sounds, and is used for voice recognition and emotion analysis.

[0288] A "media player" is a software or hardware device for playing media content such as music and videos.

[0289] "Analyzing encrypted data" means performing processing to decrypt the encrypted data and read the contents.

[0290] "Feedback" refers to the response or report that a system provides to a user, providing information based on the user's actions and emotions.

[0291] This invention relates to a system that generates and provides optimal music based on a user's emotional state. This system uses a wearable device to acquire biometric and audio data, analyzes this data to recognize the user's emotional state, and generates music using a generative AI model. The detailed configuration and processing flow are described below.

[0292] Hardware and Software Configuration

[0293] 1. Wearable devices

[0294] Wearable devices such as smartwatches and fitness trackers worn by users collect biometric data (heart rate, blood pressure, activity level) in real time and transmit the data to a device (e.g., a smartphone) via Bluetooth.

[0295] 2. Terminal

[0296] The terminal temporarily stores biometric data transmitted from the wearable device via Bluetooth. The terminal also has the function of collecting the user's voice data via a microphone. The biometric and voice data are encrypted using a secure encryption protocol (e.g., AES-256) and transmitted to the server via the HTTPS protocol.

[0297] 3. Server

[0298] The server receives the encrypted data and verifies its integrity. An emotion engine on the server analyzes the biometric and audio data to estimate the user's emotional state. Specifically, it uses machine learning models (e.g., TENSORFLOW®, PyTorch) for the biometric data and natural language processing (NLP) algorithms for the audio data. Based on the estimated emotional state, the server generates music using a generative AI model (e.g., GPT-4®, DALL-E).

[0299] 4. Generative AI Models

[0300] The generative AI model generates music based on the emotional state recognized by the emotion engine, taking into account the user's past music history, preferences, and cultural background. One example prompt is: "The user's current emotional state is high stress. Please generate instrumental music that has a relaxing effect appropriate for this state. Also, since the user has previously enjoyed listening to classical music, please take that into consideration."

[0301] 5. Providing generated music

[0302] The generated music data is then encrypted again and sent to the device. The device then decrypts the received music data and passes it to a media player. The user can then listen to the generated music via a smartphone or Bluetooth speaker. This provides personalized music that responds to the user's emotional state in real time.

[0303] Specific examples

[0304] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0305] This invention effectively supports psychological wellness by providing personalized music based on a user's real-time emotional state, which contributes to stress reduction, improved quality of life, and increased productivity.

[0306] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0307] Step 1: Data collection

[0308] A user wears a wearable device (such as a smartwatch or fitness tracker). This device collects biometric data such as heart rate, blood pressure, and activity level in real time. The collected biometric data is sent to a device (e.g., a smartphone) via Bluetooth and temporarily stored in its internal storage. Voice data of the user's speech is also collected by the device via a microphone.

[0309] Input: User biometric and voice data

[0310] Output: Biometric data and voice data temporarily stored on the device

[0311] Step 2: Data transmission and encryption

[0312] The device encrypts the temporarily stored biometric and voice data using an encryption protocol (e.g., AES-256). The encrypted data is sent to the server via the HTTPS protocol. The server receives the encrypted data and performs a hash check to verify the integrity and completeness of the data.

[0313] Input: Biometric data and voice data temporarily stored on the device

[0314] Output: Data encrypted and sent to the server

[0315] Step 3: Sentiment analysis and emotion recognition

[0316] The server decrypts the encrypted data and begins analysis. Biometric data is analyzed using machine learning models (e.g., TensorFlow, PyTorch), and audio data is analyzed using natural language processing (NLP) algorithms. The emotion engine combines these analysis results to estimate the user's emotional state.

[0317] Input: Encrypted data sent to the server

[0318] Output: Estimated emotional state

[0319] Step 4: Music Generation

[0320] The server generates music using a generative AI model (e.g., GPT-4, DALL-E) based on the emotional state estimated by the emotion engine. This generation also takes into account the user's past music history, preferences, and cultural background. The generated music data is encrypted and sent to the device.

[0321] Input: estimated emotional state, past music history, user preferences, cultural background

[0322] Output: Generated music data

[0323] Step 5: Submit your music

[0324] The device decrypts the encrypted music data received from the server. The decrypted music data is passed to the device's media player and played back to the user. The user can then listen to the music via their smartphone or Bluetooth speaker.

[0325] Input: Encrypted music data sent to the device

[0326] Output: Music data passed to a media player and played to the user

[0327] Step 6: Long-term analysis and feedback

[0328] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and a report is created to support the user's long-term improvement of their psychological state. The report is sent to the device and can be viewed by the user within the application.

[0329] Input: Daily emotion data

[0330] Output: Report provided to the user

[0331] Through these steps, the system can provide personalized music according to the user's real-time emotional state and support psychological wellness.

[0332] (Application example 2)

[0333] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] In modern brick-and-mortar stores, providing music that best suits the emotional state of customers in real time is an important factor in improving customer satisfaction. However, conventional systems have low accuracy in recognizing emotional states, making it difficult to provide appropriate music. Furthermore, security and privacy issues have made it difficult to safely handle biometric and voice data. There is a need to solve these issues and safely and effectively provide music that matches the emotional state of customers.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric data and voice data, means for encrypting the biometric data and voice data, means for decrypting and analyzing the encrypted data, means for generating music based on the emotional state, and means for encrypting and providing the generated music. This makes it possible to provide music that is optimal for the emotional state of customers in a physical store in real time while safely handling the biometric data and voice data.

[0336] "Biometric data" refers to data that measures a user's physiological indicators, such as heart rate, blood pressure, and activity level, in real time.

[0337] "Voice data" refers to voice information uttered by a user, and is data from which emotional tones and keywords can be extracted by analyzing the voice information.

[0338] "Encryption means" refers to a technology that encrypts acquired data using an encryption protocol (e.g., AES-256) to ensure security.

[0339] The "server" is a computer system that receives encrypted data, decrypts and analyzes it, estimates the user's emotional state, and generates music.

[0340] "Emotional state" refers to the user's psychological and emotional state, and is estimated from the analysis of heart rate, blood pressure, activity level, and voice data.

[0341] A "generative AI model" is an artificial intelligence algorithm that generates appropriate music based on emotional state.

[0342] A "user terminal" is a device used by a user (e.g., a smartphone or tablet) that receives and decodes data sent from the server and provides music.

[0343] A "physical store" is a commercial facility that operates in a physical form, and is a place that aims to analyze the emotions of customers in real time and provide appropriate music.

[0344] "Real-time" means that data is generated, processed, and results are provided with almost no delay.

[0345] This invention is a system for generating and providing optimal music based on the emotional state of customers in a physical store. The system includes a server, encryption means, a user terminal, biometric data acquisition means, voice data acquisition means, emotion estimation means, and a generative AI model.

[0346] The server first receives and decrypts the encrypted data sent from the means for acquiring biometric and voice data at the physical store. The AES-256 protocol is used for this encryption. The decrypted data is then analyzed by the emotion estimation means to estimate the user's emotional state. Natural language processing (NLP) algorithms and machine learning models are used for emotion estimation.

[0347] Based on the estimated emotional state, the generative AI model on the server generates optimal music in real time, taking into account the user's past music history, preferences, and even cultural background. The generated music data is then re-encrypted and sent to the user's device.

[0348] The user terminal decrypts the encrypted music data sent from the server and provides it to customers through the store's speaker system. This process is carried out in real time, and music that matches the customer's emotional state is provided, thereby improving customer satisfaction.

[0349] As a specific example, if a customer in a store is estimated to be feeling highly stressed, the generative AI model will generate music with a relaxing effect. This music is customized based on the customer's past music history and preferences. The user's device receives the music data sent from the server and plays it through speakers in the store. This helps the customer relax and reduce stress.

[0350] The system is realized by combining various hardware and software components, including a smartwatch, microphone device, and speaker system, and software components, including AES-256 encryption protocols, NLP algorithms, machine learning models, and generative AI models.

[0351] An example of a prompt is:

[0352] "Design an AI model that analyzes a user's emotional state based on their heart rate, blood pressure, activity level, and voice data. Then write a program to generate relaxing music based on that specific emotional state. Include any necessary encryption techniques."

[0353] Examples include:

[0354] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0355] Step 1:

[0356] The server collects biometric data (heart rate, blood pressure, activity level) and voice data from robots and smart devices in the physical store. The input is raw data from the sensor devices, which is received and stored in real time. The output is unencrypted biometric data and voice data.

[0357] Step 2:

[0358] The terminal encrypts the acquired biometric and voice data using the AES-256 protocol. The input is raw data, which is encrypted to ensure data security. The output is encrypted data.

[0359] Step 3:

[0360] The device sends encrypted biometric data and voice data to the server. The input is encrypted data that is securely transferred to the server over the network. The output is the data as it was sent to the server.

[0361] Step 4:

[0362] The server decrypts the received encrypted data using the AES-256 protocol. The input is the encrypted data, and the original raw data is obtained by decrypting it. The output is the decrypted biometric data and audio data.

[0363] Step 5:

[0364] The server analyzes the decoded biometric data and voice data to estimate the user's emotional state. Here, the voice data is analyzed using NLP algorithms to extract emotional tones and keywords. The biometric data is then used to estimate the emotional state using machine learning models. The input is the decoded data, and the output is the estimated emotional state.

[0365] Step 6:

[0366] The server generates music using a generative AI model based on the estimated emotional state. The input is emotional state data, and the generative AI model generates music data that takes into account the user's preferences and cultural background. The output is the generated music data.

[0367] Step 7:

[0368] The server then encrypts the generated music data again using the AES-256 protocol. The input is the music data, and the encryption process prepares it for secure transmission to the user's device. The output is the encrypted music data.

[0369] Step 8:

[0370] The device receives encrypted music data sent from the server and decrypts it using the AES-256 protocol. The input is the encrypted music data, and the original music data is obtained by decrypting it. The output is the decrypted music data.

[0371] Step 9:

[0372] The terminal plays the decoded music data through the speaker system in the physical store. The input is music data, and the output is music provided to customers. This makes it possible to provide music that corresponds to the emotional state of customers in real time.

[0373] This series of processes realizes a system that provides music that best suits the emotional state of customers in a physical store.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0377] [Second embodiment]

[0378] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0379] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0380] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0382] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0384] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0385] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0386] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0389] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0390] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0391] Program processing

[0392] Data collection

[0393] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0394] Data Transmission and Encryption

[0395] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0396] sentiment analysis

[0397] The server analyzes the received data. It uses natural language processing (NLP) algorithms to infer emotions from the voice data. At the same time, it analyzes biometric data through machine learning models to infer emotional states. These results are then combined to determine the final emotional state.

[0398] Music Generation

[0399] Based on the results of the sentiment analysis, the server uses a generative AI model to generate music. This takes into account the user's preferences and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0400] Music provided by

[0401] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. Even during this process, the music is provided in real time, allowing users to instantly enjoy music that matches their emotional state.

[0402] Long-term analysis and feedback

[0403] The server tracks daily emotional fluctuations and stores them in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user.

[0404] Specific examples

[0405] For example, if a user is analyzed as being in a high stress state while working remotely, the server will generate relaxation music. Since stress can be detected from the voice data, the generated music is a highly relaxing instrumental piece tailored to the user's preferences. The device then plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0406] This system can effectively support users' psychological wellness, contributing to improving quality of life, reducing stress, and increasing productivity.

[0407] The processing flow will be explained below.

[0408] Step 1:

[0409] User

[0410] The user wears a smartwatch, which measures heart rate, blood pressure, activity, and audio data in real time. Data is collected continuously as the user goes about their daily activities.

[0411] Step 2:

[0412] Smartwatch

[0413] The smartwatch transmits the collected biometric and audio data to a device (e.g., a smartphone) using wireless communication such as Bluetooth.

[0414] Step 3:

[0415] Terminal

[0416] The device temporarily stores the data received from the smartwatch in its memory, encrypts the stored data using an encryption method (e.g., AES-256), and periodically transmits the encrypted data to the server.

[0417] Step 4:

[0418] server

[0419] The server receives the encrypted data sent from the device. After receiving it, it decrypts the data and verifies its consistency and integrity. After verification, the data is ready for analysis.

[0420] Step 5:

[0421] server

[0422] The server analyzes the acquired data. It uses a natural language processing (NLP) algorithm to extract keywords and emotional tones from the voice data and estimate the emotional state. At the same time, it uses a machine learning model to analyze biometric data (heart rate, blood pressure, activity level) to estimate the emotional state. The analysis results of the voice data and biometric data are integrated to determine the final emotional state.

[0423] Step 6:

[0424] server

[0425] The server generates music using a generative AI model based on the estimated emotional state, taking into account the user's past music history, preferences, and cultural background. The generated music data is then re-encrypted and sent to the device.

[0426] Step 7:

[0427] Terminal

[0428] The device decrypts the encrypted music data received from the server, passes the decrypted music data to the media player, and prepares it for playback. The device then connects to a smartphone or Bluetooth speaker for playback.

[0429] Step 8:

[0430] User

[0431] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0432] Step 9:

[0433] server

[0434] The server tracks daily emotional data and stores it in a database. It then performs long-term analysis to extract trends and patterns, and creates reports to help users improve their psychological state.

[0435] Step 10:

[0436] Terminal

[0437] The device receives the report from the server and provides it to the user, who can then view the report in the application to check long-term trends in their psychological state and take steps to improve their daily habits and work environment.

[0438] Through these specific processing steps, the system of the present invention can provide music that corresponds to the user's real-time emotional state and provide psychological support.

[0439] Example 1

[0440] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0441] Conventional emotional state estimation systems that use biometric and audio data have difficulty accurately analyzing a user's unique emotional state and generating and providing appropriate music. Furthermore, they lack the ability to grasp a user's long-term emotional trends and provide feedback. This makes it difficult to comprehensively support users' psychological wellness.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0443] In this invention, the server includes means for acquiring biometric data, means for analyzing the acquired biometric data and transmitting it to the server using an encryption protocol, means for integrating the analyzed biometric data with audio data to estimate the user's emotional state, means for generating music based on the estimated emotional state using a generative AI model, means for transmitting the generated music to the terminal and providing it to the user, and means for analyzing the user's long-term emotional trends and patterns and providing feedback. This allows the system to accurately analyze the user's unique emotional state and provide appropriate music in real time. It also provides feedback to support the user's long-term psychological wellness.

[0444] "Biometric data" refers to data that indicates the user's physical condition in real time, such as heart rate, blood pressure, and activity level.

[0445] "Encryption protocol" refers to a standard method used to protect the confidentiality of data, such as AES-256.

[0446] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0447] "Inference tools" refer to algorithms or machine learning models that analyze collected data and derive specific results.

[0448] A "generative AI model" is an artificial intelligence model that generates a specific output (in this case, music) by taking into account the user's emotional state and preferences.

[0449] A "terminal" is a device used by a user, such as a smartphone or tablet, that transmits, receives, and displays data.

[0450] "Emotional state" is an indicator of the user's psychological and emotional state, and includes states such as stress and relaxation.

[0451] "Analyzing" is the process of examining collected data in detail and extracting specific information or patterns.

[0452] "Integrating" refers to combining information from multiple data sources to gain a holistic understanding and analysis.

[0453] "Feedback" means information or advice provided to users that is based on past data and analysis.

[0454] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0455] Data collection

[0456] The user wears a smartwatch, which collects real-time biometric data such as heart rate, blood pressure, and activity level. The watch also collects the user's voice via the smartphone, and this data is temporarily stored on the smartphone.

[0457] Data Transmission and Encryption

[0458] The smartphone device encrypts the collected biometric and voice data using the AES-256 encryption protocol, and then transmits the encrypted data to a server, which verifies the integrity and completeness of the received data.

[0459] sentiment analysis

[0460] The server analyzes the received biometric data using a machine learning model to estimate the user's emotional state. This model is trained from past data and can recognize stress and relaxation. The server also analyzes the voice data using an NLP (natural language processing) algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to determine the user's final emotional state.

[0461] Music Generation

[0462] The server then uses a generative AI model to generate personalized music based on the results of the sentiment analysis. This model also takes into account the user's musical history and cultural background. The generated music data is then re-encrypted and sent to the device.

[0463] Music provided by

[0464] The device decodes the received music data and passes it to the media player, allowing users to enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0465] Long-term analysis and feedback

[0466] The server tracks daily emotional fluctuations and stores them in a database. The stored data is then analyzed to extract trends and patterns in the user's emotional state. The server then creates a report based on the analysis results and sends it to the device. The device receives this report and provides it to the user, allowing the user to understand the history of their own emotional state.

[0467] Specific examples

[0468] For example, if a user is analyzed as being in a state of high stress while working remotely, the server will generate relaxation music. The generative AI model will select instrumental songs that the user has previously found relaxing and create new music based on these. The device will receive the generated music data and pass it to the user's media player. The user can immediately listen to the relaxation music and reduce stress.

[0469] Prompt Sentence Examples

[0470] "Please explain the system that generates and provides appropriate relaxation music when it is analyzed that the user is in a high stress state while working remotely. Specifically, please describe what data you collect, what technology you use to perform emotion analysis, and how you use the generative AI model."

[0471] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0472] Step 1: Data collection

[0473] The user wears a smartwatch, which collects biometric data such as heart rate, blood pressure, and activity level in real time. At the same time, the user's voice is also collected via the smartphone. This data is temporarily stored on the smartphone. The input is the user's biometric data and voice data, and the output is the data stored on the smartphone.

[0474] Step 2: Data transmission and encryption

[0475] The terminal, a smartphone, receives the acquired biometric data and voice data as input. The smartphone encrypts these data using the AES-256 encryption protocol. The encrypted data is then sent to the server. The output is encrypted data, which is sent to the server. The server verifies the integrity and completeness of the received data.

[0476] Step 3: Sentiment analysis

[0477] The server analyzes the received biometric data using a machine learning model. The input is encrypted biometric data and voice data. The server first decrypts it and then analyzes it to estimate the user's emotional state. A model trained on past data is used to analyze the biometric data. At the same time, the voice data is analyzed using an NLP algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to obtain the final emotional state as the output.

[0478] Step 4: Music Generation

[0479] The server generates personalized music using a generative AI model based on the results of the sentiment analysis. The input is the integrated emotional state, the user's music history, and cultural background. The generative AI model takes into account the user's past data and generates new music. The generated music data is re-encrypted and sent to the device. The output is encrypted music data.

[0480] Step 5: Submit your music

[0481] The device decrypts the received music data and passes it to the media player. The input is encrypted music data, and the output is decrypted music data. Users can enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0482] Step 6: Long-term analysis and feedback

[0483] The server tracks daily emotional fluctuations and stores them in a database. The input is daily emotional data, and the output is the stored data. The server analyzes the stored data and extracts trends and patterns in the user's emotional state. The server creates a report based on the analysis results and sends it to the device. The device receives this and provides it to the user. The output is a report of the analysis results.

[0484] Through this processing step, the system can accurately analyze a user's unique emotional state and provide appropriate music in real time, as well as provide feedback to support the user's long-term psychological wellness.

[0485] (Application example 1)

[0486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] Conventional food delivery systems have struggled to provide an individually customized experience that takes into account the user's emotional state. The user experience, particularly during the ordering process and waiting time, is limited, and there are few ways to alleviate the stress and anxiety felt by users. As a result, user satisfaction tends to decline. To solve this issue, it is necessary to provide new added value based on the user's emotional state.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0489] In this invention, the server includes a means for providing music during the ordering process or waiting time according to the emotional state of the user, a means for acquiring voice data, and a means for analyzing the acquired voice data and estimating the emotional state, thereby providing an individually customized music experience and reducing stress during the waiting time or ordering process while caring for the user's emotions.

[0490] "Biometric data" refers to data that indicates physical conditions such as heart rate, blood pressure, and activity level obtained through smartwatches and other wearable devices.

[0491] "Voice data" refers to data that records the user's speech and surrounding sounds acquired via a smartphone or other voice input device.

[0492] "Emotional state" refers to the user's psychological state or mood that is estimated based on the analyzed biometric and audio data.

[0493] "Means for generating music" refers to a method or algorithm that creates music to make the user feel a particular emotion, such as relaxation or concentration, based on the acquired emotional state.

[0494] "Means for providing to a user" refers to a method or system for transmitting the generated music to a user's device so that the user can listen to it.

[0495] "Means for providing music during the ordering process and while waiting" refers to a method or system within a food delivery application that plays appropriate music while a user is placing an order or waiting for food to be delivered.

[0496] "Customization means" refers to methods and algorithms that individually tailor the content and style of music to suit the user's emotional state, taking into account the user's preferences and cultural background.

[0497] The present invention is a system that provides individually customized music based on the user's emotional state, and its application example is to improve the user experience in food delivery applications.

[0498] In particular, the server analyzes the user's emotional state sequentially and generates and provides music that matches that state. To achieve this, the following hardware and software are used in combination:

[0499] Hardware and Software Overview

[0500] 1. Smartwatch:

[0501] Used to obtain biometric data (heart rate, blood pressure, activity level) in real time.

[0502] 2. Smartphone:

[0503] Biometric and audio data is temporarily stored and transmitted to a cloud server using an encryption protocol.

[0504] Used to play provided music to the user.

[0505] 3. Cloud Server:

[0506] The server that receives and analyzes the data encrypts it and verifies its integrity and completeness.

[0507] Sentiment estimation is performed using natural language processing (NLP) algorithms and machine learning models.

[0508] Generate music using generative AI models.

[0509] The specific software used is as follows:

[0510] PyCryptodome (used to encrypt data)

[0511] Google Cloud Speech-to-Text API (used to convert voice data into text)

[0512] HuggingFace Transformers (used for NLP analysis of voice data)

[0513] scikit-learn (used for machine learning model analysis of biological data)

[0514] VLC (used to play music on smartphones)

[0515] Detailed system processing flow

[0516] 1. Data Collection:

[0517] When a user wears a smartwatch and connects it to a smartphone, biometric and audio data are collected in real time.

[0518] 2. Data Transmission and Encryption:

[0519] The collected data is encrypted on the smartphone using an encryption protocol (AES-256) and sent to a cloud server, which verifies the data's integrity and completeness.

[0520] 3. Sentiment analysis:

[0521] The cloud server converts the voice data into text using the Google Cloud Speech-to-Text API and estimates the emotional state using HuggingFace Transformers.

[0522] The biometric data is fed into a machine learning model using scikit-learn to analyze the emotional state, and the final emotional state is determined by integrating these data.

[0523] 4. Music Generation:

[0524] Based on the analysis of the emotional state, a prompt sentence is provided to a generative AI model (e.g., GPT-3) to generate music.

[0525] Examples of prompts:

[0526] Emotional state: Stress

[0527] User Preferences: Jazz

[0528] User's cultural background: Japan

[0529] Type of music generated: Relaxing jazz tunes

[0530] 5. Music provided by:

[0531] The generated music is then re-encrypted and sent to a smartphone, where users can listen to it via their smartphone or Bluetooth speaker while waiting or ordering.

[0532] Specific examples

[0533] For example, if a food delivery application determines that a user is in a high stress state, the cloud server generates relaxing jazz music. This music is customized taking into account the user's preferences and cultural background. By listening to this music, users can reduce stress and spend the waiting time more comfortably after placing their order.

[0534] This system allows users to receive appropriate music in real time according to their emotional state, thereby improving satisfaction with food delivery services.

[0535] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0536] Step 1:

[0537] The user wears a smartwatch, and biometric data (heart rate, blood pressure, activity level) is collected in real time via a smartphone. This data is temporarily stored on the smartphone. The input is biometric data, and the output is the biometric data stored on the smartphone.

[0538] Step 2:

[0539] Audio data is collected via the smartphone's microphone. This includes the user's speech and surrounding sounds. This audio data is also stored on the smartphone. The input is audio data, and the output is the audio data stored on the smartphone.

[0540] Step 3:

[0541] The smartphone encrypts the collected biometric and audio data using the AES-256 protocol, ensuring data security: the input is unencrypted data and the output is encrypted data.

[0542] Step 4:

[0543] The encrypted data is sent from the smartphone to a cloud server, which verifies the integrity and completeness of the received data. The input is the encrypted data, and the output is the verified data stored on the server.

[0544] Step 5:

[0545] The cloud server uses the Google Cloud Speech-to-Text API to convert the voice data into text data, which is then used for sentiment analysis. The input is the voice data, and the output is the converted text data.

[0546] Step 6:

[0547] The server uses HuggingFace Transformers to perform natural language processing (NLP) on the transformed text data to estimate the emotional state. The input is the text data, and the output is the estimated emotional state.

[0548] Step 7:

[0549] The server inputs the biometric data into a machine learning model based on scikit-learn to analyze the emotional state. This integrates emotion estimation using natural language processing with biometric data analysis to determine the final emotional state. The input is the biometric data, and the output is the integrated emotional state.

[0550] Step 8:

[0551] The server generates music by inputting a prompt sentence into a generative AI model based on the estimated emotional state. GPT-3 or another model is used as the generative AI model. The input is the prompt sentence, and the output is the generated music data. Specific examples of prompt sentences are as follows:

[0552] Emotional state: Stress

[0553] User Preferences: Jazz

[0554] User's cultural background: Japan

[0555] Type of music generated: Relaxing jazz tunes

[0556] Step 9:

[0557] The generated music data is again encrypted using the AES-256 protocol and sent to the smartphone. The input is unencrypted music data, and the output is encrypted music data.

[0558] Step 10:

[0559] The smartphone receives the encrypted music data, decrypts it, and passes it to the media player. Specifically, the music is played via the VLC library. This allows the user to listen to music optimized for their emotional state in real time. The input is encrypted music data, and the output is decrypted music data.

[0560] By going through the above processing steps, a food delivery application can provide a personalized music experience that corresponds to the user's emotional state.

[0561] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0562] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and audio data, uses an emotion engine to accurately recognize the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0563] Program processing

[0564] Data collection

[0565] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0566] Data Transmission and Encryption

[0567] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0568] Emotion recognition with sentiment analysis and emotion engine

[0569] The server analyzes the received data. The emotion engine combines biometric data and voice data to accurately recognize the emotional state. Natural language processing (NLP) algorithms are used to extract keywords and emotional tones from the voice data to estimate the emotional state. At the same time, the biometric data is also analyzed using a machine learning model to estimate the emotional state. Based on the results of these analyses, the emotion engine determines the final emotional state.

[0570] Music Generation

[0571] Based on the emotional state recognized by the emotion engine, the server uses a generative AI model to generate music. This generation also takes into account the user's past music history, preferences, and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0572] Music provided by

[0573] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. This provides music that responds to the user's emotional state in real time, supporting psychological wellness.

[0574] Long-term analysis and feedback

[0575] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user. The user can view the reports within the application to check long-term trends in their psychological state.

[0576] Specific examples

[0577] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0578] This invention makes it possible to effectively provide music that responds to the user's real-time emotional state and provide psychological support, which is designed to contribute to stress reduction, improved quality of life, and increased productivity.

[0579] The processing flow will be explained below.

[0580] Program processing

[0581] Step 1:

[0582] User

[0583] The user puts on the smartwatch and begins their daily activities, and the smartwatch measures heart rate, blood pressure, activity level, and audio data in real time.

[0584] Step 2:

[0585] Smartwatch

[0586] The smartwatch transmits the biometric and audio data it measures to the device (smartphone) via wireless communication (such as Bluetooth).

[0587] Step 3:

[0588] Terminal

[0589] The device temporarily stores the data received from the smartwatch in its memory, and at the same time encrypts the data (using AES-256, etc.) and prepares it for transmission to the server.

[0590] Step 4:

[0591] Terminal

[0592] The encrypted data is sent to the server in real time, ensuring data security.

[0593] Step 5:

[0594] server

[0595] The server decrypts the encrypted data received from the device and verifies the integrity and completeness of the data, after which it is ready for analysis.

[0596] Step 6:

[0597] server

[0598] The server analyzes the received data. The emotion engine comprehensively analyzes the voice data and biometric data to recognize the user's emotional state. Natural language processing (NLP) algorithms are used to extract emotional tones and keywords from the voice data, and machine learning models are used to estimate the user's emotional state from the biometric data.

[0599] Step 7:

[0600] server

[0601] Based on the analysis results of the emotion engine, the server determines the user's emotional state, for example, determining that the user is in a high stress state.

[0602] Step 8:

[0603] server

[0604] The server uses a generative AI model to generate music that best suits the user's emotional state, taking into account the user's preferences and cultural background. The generated music data is then re-encrypted.

[0605] Step 9:

[0606] server

[0607] The server sends the encrypted music data to the terminal, which then transmits it immediately.

[0608] Step 10:

[0609] Terminal

[0610] The device decrypts the encrypted music data received from the server, and the decrypted music data is passed to the media player.

[0611] Step 11:

[0612] Terminal

[0613] The device connects to the user's smartphone or Bluetooth speaker and prepares to play music.

[0614] Step 12:

[0615] User

[0616] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0617] Step 13:

[0618] server

[0619] The server tracks daily emotional data and stores it in a database, which allows for long-term data analysis.

[0620] Step 14:

[0621] server

[0622] Analyze the accumulated data, extract trends and patterns, and create reports for users to support long-term psychological improvement.

[0623] Step 15:

[0624] Terminal

[0625] The device receives the report from the server and provides it to the user, who can then view the report through the application to check trends and areas for improvement in their own psychological state.

[0626] In this way, the system of the present invention can provide optimal music based on the user's real-time emotional state and provide psychological support.

[0627] Example 2

[0628] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0629] In modern society, people's stress and psychological burden are increasing, but a system for effectively alleviating these stresses and providing users with music that suits their needs has not yet been fully established. Conventional systems have difficulty providing personalized music based on each user's real-time emotional state. There is a need to solve this problem and improve users' psychological wellness.

[0630] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0631] In this invention, the server includes means for acquiring biometric data from a wearable device worn by a user, means for temporarily storing the acquired biometric data in the terminal and transmitting it to the server using an encryption protocol, means for analyzing the encrypted data and estimating an emotional state using an emotion engine, means for generating music using a generative AI model based on the estimated emotional state, means for encrypting the generated music data and transmitting it to the terminal, means for playing the decrypted music on the terminal via a media player and providing it to the user, and means for tracking and analyzing daily emotional data over the long term and providing feedback to the user. This makes it possible to provide personalized music based on each user's real-time emotional state, which is expected to help relieve stress and improve psychological well-being.

[0632] "Biometric data" refers to data related to the user's physical condition, including heart rate, blood pressure, activity level, and the like.

[0633] A "wearable device" is an electronic device worn by a user, including smartwatches and fitness trackers.

[0634] "Encryption protocols" are technologies used to ensure data security, including algorithms such as AES-256.

[0635] An "emotion engine" is a computer system or algorithm that analyzes biometric and audio data to accurately estimate a user's emotional state.

[0636] A "generative AI model" is an artificial intelligence model that uses machine learning to perform specific tasks, and is used for applications such as music generation and text generation.

[0637] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, tablets, etc.

[0638] A "server" is a centralized computer system for processing and storing data over a network.

[0639] "Voice data" refers to data that records the user's speaking voice or sounds, and is used for voice recognition and emotion analysis.

[0640] A "media player" is a software or hardware device for playing media content such as music and videos.

[0641] "Analyzing encrypted data" means performing processing to decrypt the encrypted data and read the contents.

[0642] "Feedback" refers to the response or report that a system provides to a user, providing information based on the user's actions and emotions.

[0643] This invention relates to a system that generates and provides optimal music based on a user's emotional state. This system uses a wearable device to acquire biometric and audio data, analyzes this data to recognize the user's emotional state, and generates music using a generative AI model. The detailed configuration and processing flow are described below.

[0644] Hardware and Software Configuration

[0645] 1. Wearable devices

[0646] Wearable devices such as smartwatches and fitness trackers worn by users collect biometric data (heart rate, blood pressure, activity level) in real time and transmit the data to a device (e.g., a smartphone) via Bluetooth.

[0647] 2. Terminal

[0648] The terminal temporarily stores biometric data transmitted from the wearable device via Bluetooth. The terminal also has the function of collecting the user's voice data via a microphone. The biometric and voice data are encrypted using a secure encryption protocol (e.g., AES-256) and transmitted to the server via the HTTPS protocol.

[0649] 3. Server

[0650] The server receives the encrypted data and verifies its integrity. The emotion engine on the server analyzes the biometric and audio data to estimate the user's emotional state. Specifically, it uses machine learning models (e.g., TensorFlow, PyTorch) for the biometric data and natural language processing (NLP) algorithms for the audio data. Based on the estimated emotional state, the server generates music using a generative AI model (e.g., GPT-4, DALL-E).

[0651] 4. Generative AI Models

[0652] The generative AI model generates music based on the emotional state recognized by the emotion engine, taking into account the user's past music history, preferences, and cultural background. One example prompt is: "The user's current emotional state is high stress. Please generate instrumental music that has a relaxing effect appropriate for this state. Also, since the user has previously enjoyed listening to classical music, please take that into consideration."

[0653] 5. Providing generated music

[0654] The generated music data is then encrypted again and sent to the device. The device then decrypts the received music data and passes it to a media player. The user can then listen to the generated music via a smartphone or Bluetooth speaker. This provides personalized music that responds to the user's emotional state in real time.

[0655] Specific examples

[0656] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0657] This invention effectively supports psychological wellness by providing personalized music based on a user's real-time emotional state, which contributes to stress reduction, improved quality of life, and increased productivity.

[0658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0659] Step 1: Data collection

[0660] A user wears a wearable device (such as a smartwatch or fitness tracker). This device collects biometric data such as heart rate, blood pressure, and activity level in real time. The collected biometric data is sent to a device (e.g., a smartphone) via Bluetooth and temporarily stored in its internal storage. Voice data of the user's speech is also collected by the device via a microphone.

[0661] Input: User biometric and voice data

[0662] Output: Biometric data and voice data temporarily stored on the device

[0663] Step 2: Data transmission and encryption

[0664] The device encrypts the temporarily stored biometric and voice data using an encryption protocol (e.g., AES-256). The encrypted data is sent to the server via the HTTPS protocol. The server receives the encrypted data and performs a hash check to verify the integrity and completeness of the data.

[0665] Input: Biometric data and voice data temporarily stored on the device

[0666] Output: Data encrypted and sent to the server

[0667] Step 3: Sentiment analysis and emotion recognition

[0668] The server decrypts the encrypted data and begins analysis. Biometric data is analyzed using machine learning models (e.g., TensorFlow, PyTorch), and audio data is analyzed using natural language processing (NLP) algorithms. The emotion engine combines these analysis results to estimate the user's emotional state.

[0669] Input: Encrypted data sent to the server

[0670] Output: Estimated emotional state

[0671] Step 4: Music Generation

[0672] The server generates music using a generative AI model (e.g., GPT-4, DALL-E) based on the emotional state estimated by the emotion engine. This generation also takes into account the user's past music history, preferences, and cultural background. The generated music data is encrypted and sent to the device.

[0673] Input: estimated emotional state, past music history, user preferences, cultural background

[0674] Output: Generated music data

[0675] Step 5: Submit your music

[0676] The device decrypts the encrypted music data received from the server. The decrypted music data is passed to the device's media player and played back to the user. The user can then listen to the music via their smartphone or Bluetooth speaker.

[0677] Input: Encrypted music data sent to the device

[0678] Output: Music data passed to a media player and played to the user

[0679] Step 6: Long-term analysis and feedback

[0680] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and a report is created to support the user's long-term improvement of their psychological state. The report is sent to the device and can be viewed by the user within the application.

[0681] Input: Daily emotion data

[0682] Output: Report provided to the user

[0683] Through these steps, the system can provide personalized music according to the user's real-time emotional state and support psychological wellness.

[0684] (Application example 2)

[0685] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] In modern brick-and-mortar stores, providing music that best suits the emotional state of customers in real time is an important factor in improving customer satisfaction. However, conventional systems have low accuracy in recognizing emotional states, making it difficult to provide appropriate music. Furthermore, security and privacy issues have made it difficult to safely handle biometric and voice data. There is a need to solve these issues and safely and effectively provide music that matches the emotional state of customers.

[0687] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric data and voice data, means for encrypting the biometric data and voice data, means for decrypting and analyzing the encrypted data, means for generating music based on the emotional state, and means for encrypting and providing the generated music. This makes it possible to provide music that is optimal for the emotional state of customers in a physical store in real time while safely handling the biometric data and voice data.

[0688] "Biometric data" refers to data that measures a user's physiological indicators, such as heart rate, blood pressure, and activity level, in real time.

[0689] "Voice data" refers to voice information uttered by a user, and is data from which emotional tones and keywords can be extracted by analyzing the voice information.

[0690] "Encryption means" refers to a technology that encrypts acquired data using an encryption protocol (e.g., AES-256) to ensure security.

[0691] The "server" is a computer system that receives encrypted data, decrypts and analyzes it, estimates the user's emotional state, and generates music.

[0692] "Emotional state" refers to the user's psychological and emotional state, and is estimated from the analysis of heart rate, blood pressure, activity level, and voice data.

[0693] A "generative AI model" is an artificial intelligence algorithm that generates appropriate music based on emotional state.

[0694] A "user terminal" is a device used by a user (e.g., a smartphone or tablet) that receives and decodes data sent from the server and provides music.

[0695] A "physical store" is a commercial facility that operates in a physical form, and is a place that aims to analyze the emotions of customers in real time and provide appropriate music.

[0696] "Real-time" means that data is generated, processed, and results are provided with almost no delay.

[0697] This invention is a system for generating and providing optimal music based on the emotional state of customers in a physical store. The system includes a server, encryption means, a user terminal, biometric data acquisition means, voice data acquisition means, emotion estimation means, and a generative AI model.

[0698] The server first receives and decrypts the encrypted data sent from the means for acquiring biometric and voice data at the physical store. The AES-256 protocol is used for this encryption. The decrypted data is then analyzed by the emotion estimation means to estimate the user's emotional state. Natural language processing (NLP) algorithms and machine learning models are used for emotion estimation.

[0699] Based on the estimated emotional state, the generative AI model on the server generates optimal music in real time, taking into account the user's past music history, preferences, and even cultural background. The generated music data is then re-encrypted and sent to the user's device.

[0700] The user terminal decrypts the encrypted music data sent from the server and provides it to customers through the store's speaker system. This process is carried out in real time, and music that matches the customer's emotional state is provided, thereby improving customer satisfaction.

[0701] As a specific example, if a customer in a store is estimated to be feeling highly stressed, the generative AI model will generate music with a relaxing effect. This music is customized based on the customer's past music history and preferences. The user's device receives the music data sent from the server and plays it through speakers in the store. This helps the customer relax and reduce stress.

[0702] The system is realized by combining various hardware and software components, including a smartwatch, microphone device, and speaker system, and software components, including AES-256 encryption protocols, NLP algorithms, machine learning models, and generative AI models.

[0703] An example of a prompt is:

[0704] "Design an AI model that analyzes a user's emotional state based on their heart rate, blood pressure, activity level, and voice data. Then write a program to generate relaxing music based on that specific emotional state. Include any necessary encryption techniques."

[0705] Examples include:

[0706] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0707] Step 1:

[0708] The server collects biometric data (heart rate, blood pressure, activity level) and voice data from robots and smart devices in the physical store. The input is raw data from the sensor devices, which is received and stored in real time. The output is unencrypted biometric data and voice data.

[0709] Step 2:

[0710] The terminal encrypts the acquired biometric and voice data using the AES-256 protocol. The input is raw data, which is encrypted to ensure data security. The output is encrypted data.

[0711] Step 3:

[0712] The device sends encrypted biometric data and voice data to the server. The input is encrypted data that is securely transferred to the server over the network. The output is the data as it was sent to the server.

[0713] Step 4:

[0714] The server decrypts the received encrypted data using the AES-256 protocol. The input is the encrypted data, and the original raw data is obtained by decrypting it. The output is the decrypted biometric data and audio data.

[0715] Step 5:

[0716] The server analyzes the decoded biometric data and voice data to estimate the user's emotional state. Here, the voice data is analyzed using NLP algorithms to extract emotional tones and keywords. The biometric data is then used to estimate the emotional state using machine learning models. The input is the decoded data, and the output is the estimated emotional state.

[0717] Step 6:

[0718] The server generates music using a generative AI model based on the estimated emotional state. The input is emotional state data, and the generative AI model generates music data that takes into account the user's preferences and cultural background. The output is the generated music data.

[0719] Step 7:

[0720] The server then encrypts the generated music data again using the AES-256 protocol. The input is the music data, and the encryption process prepares it for secure transmission to the user's device. The output is the encrypted music data.

[0721] Step 8:

[0722] The device receives encrypted music data sent from the server and decrypts it using the AES-256 protocol. The input is the encrypted music data, and the original music data is obtained by decrypting it. The output is the decrypted music data.

[0723] Step 9:

[0724] The terminal plays the decoded music data through the speaker system in the physical store. The input is music data, and the output is music provided to customers. This makes it possible to provide music that corresponds to the emotional state of customers in real time.

[0725] This series of processes realizes a system that provides music that best suits the emotional state of customers in a physical store.

[0726] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0727] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0728] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0729] [Third embodiment]

[0730] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0731] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0732] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0733] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0734] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0735] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0736] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0737] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0738] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0739] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0740] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0741] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0742] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0743] Program processing

[0744] Data collection

[0745] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0746] Data Transmission and Encryption

[0747] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0748] sentiment analysis

[0749] The server analyzes the received data. It uses natural language processing (NLP) algorithms to infer emotions from the voice data. At the same time, it analyzes biometric data through machine learning models to infer emotional states. These results are then combined to determine the final emotional state.

[0750] Music Generation

[0751] Based on the results of the sentiment analysis, the server uses a generative AI model to generate music. This takes into account the user's preferences and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0752] Music provided by

[0753] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. Even during this process, the music is provided in real time, allowing users to instantly enjoy music that matches their emotional state.

[0754] Long-term analysis and feedback

[0755] The server tracks daily emotional fluctuations and stores them in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user.

[0756] Specific examples

[0757] For example, if a user is analyzed as being in a high stress state while working remotely, the server will generate relaxation music. Since stress can be detected from the voice data, the generated music is a highly relaxing instrumental piece tailored to the user's preferences. The device then plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0758] This system can effectively support users' psychological wellness, contributing to improving quality of life, reducing stress, and increasing productivity.

[0759] The processing flow will be explained below.

[0760] Step 1:

[0761] User

[0762] The user wears a smartwatch, which measures heart rate, blood pressure, activity, and audio data in real time. Data is collected continuously as the user goes about their daily activities.

[0763] Step 2:

[0764] Smartwatch

[0765] The smartwatch transmits the collected biometric and audio data to a device (e.g., a smartphone) using wireless communication such as Bluetooth.

[0766] Step 3:

[0767] Terminal

[0768] The device temporarily stores the data received from the smartwatch in its memory, encrypts the stored data using an encryption method (e.g., AES-256), and periodically transmits the encrypted data to the server.

[0769] Step 4:

[0770] server

[0771] The server receives the encrypted data sent from the device. After receiving it, it decrypts the data and verifies its consistency and integrity. After verification, the data is ready for analysis.

[0772] Step 5:

[0773] server

[0774] The server analyzes the acquired data. It uses a natural language processing (NLP) algorithm to extract keywords and emotional tones from the voice data and estimate the emotional state. At the same time, it uses a machine learning model to analyze biometric data (heart rate, blood pressure, activity level) to estimate the emotional state. The analysis results of the voice data and biometric data are integrated to determine the final emotional state.

[0775] Step 6:

[0776] server

[0777] The server generates music using a generative AI model based on the estimated emotional state, taking into account the user's past music history, preferences, and cultural background. The generated music data is then re-encrypted and sent to the device.

[0778] Step 7:

[0779] Terminal

[0780] The device decrypts the encrypted music data received from the server, passes the decrypted music data to the media player, and prepares it for playback. The device then connects to a smartphone or Bluetooth speaker for playback.

[0781] Step 8:

[0782] User

[0783] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0784] Step 9:

[0785] server

[0786] The server tracks daily emotional data and stores it in a database. It then performs long-term analysis to extract trends and patterns, and creates reports to help users improve their psychological state.

[0787] Step 10:

[0788] Terminal

[0789] The device receives the report from the server and provides it to the user, who can then view the report in the application to check long-term trends in their psychological state and take steps to improve their daily habits and work environment.

[0790] Through these specific processing steps, the system of the present invention can provide music that corresponds to the user's real-time emotional state and provide psychological support.

[0791] Example 1

[0792] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0793] Conventional emotional state estimation systems that use biometric and audio data have difficulty accurately analyzing a user's unique emotional state and generating and providing appropriate music. Furthermore, they lack the ability to grasp a user's long-term emotional trends and provide feedback. This makes it difficult to comprehensively support users' psychological wellness.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0795] In this invention, the server includes means for acquiring biometric data, means for analyzing the acquired biometric data and transmitting it to the server using an encryption protocol, means for integrating the analyzed biometric data with audio data to estimate the user's emotional state, means for generating music based on the estimated emotional state using a generative AI model, means for transmitting the generated music to the terminal and providing it to the user, and means for analyzing the user's long-term emotional trends and patterns and providing feedback. This allows the system to accurately analyze the user's unique emotional state and provide appropriate music in real time. It also provides feedback to support the user's long-term psychological wellness.

[0796] "Biometric data" refers to data that indicates the user's physical condition in real time, such as heart rate, blood pressure, and activity level.

[0797] "Encryption protocol" refers to a standard method used to protect the confidentiality of data, such as AES-256.

[0798] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[0799] "Inference tools" refer to algorithms or machine learning models that analyze collected data and derive specific results.

[0800] A "generative AI model" is an artificial intelligence model that generates a specific output (in this case, music) by taking into account the user's emotional state and preferences.

[0801] A "terminal" is a device used by a user, such as a smartphone or tablet, that transmits, receives, and displays data.

[0802] "Emotional state" is an indicator of the user's psychological and emotional state, and includes states such as stress and relaxation.

[0803] "Analyzing" is the process of examining collected data in detail and extracting specific information or patterns.

[0804] "Integrating" refers to combining information from multiple data sources to gain a holistic understanding and analysis.

[0805] "Feedback" means information or advice provided to users that is based on past data and analysis.

[0806] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0807] Data collection

[0808] The user wears a smartwatch, which collects real-time biometric data such as heart rate, blood pressure, and activity level. The watch also collects the user's voice via the smartphone, and this data is temporarily stored on the smartphone.

[0809] Data Transmission and Encryption

[0810] The smartphone device encrypts the collected biometric and voice data using the AES-256 encryption protocol, and then transmits the encrypted data to a server, which verifies the integrity and completeness of the received data.

[0811] sentiment analysis

[0812] The server analyzes the received biometric data using a machine learning model to estimate the user's emotional state. This model is trained from past data and can recognize stress and relaxation. The server also analyzes the voice data using an NLP (natural language processing) algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to determine the user's final emotional state.

[0813] Music Generation

[0814] The server then uses a generative AI model to generate personalized music based on the results of the sentiment analysis. This model also takes into account the user's musical history and cultural background. The generated music data is then re-encrypted and sent to the device.

[0815] Music provided by

[0816] The device decodes the received music data and passes it to the media player, allowing users to enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0817] Long-term analysis and feedback

[0818] The server tracks daily emotional fluctuations and stores them in a database. The stored data is then analyzed to extract trends and patterns in the user's emotional state. The server then creates a report based on the analysis results and sends it to the device. The device receives this report and provides it to the user, allowing the user to understand the history of their own emotional state.

[0819] Specific examples

[0820] For example, if a user is analyzed as being in a state of high stress while working remotely, the server will generate relaxation music. The generative AI model will select instrumental songs that the user has previously found relaxing and create new music based on these. The device will receive the generated music data and pass it to the user's media player. The user can immediately listen to the relaxation music and reduce stress.

[0821] Prompt Sentence Examples

[0822] "Please explain the system that generates and provides appropriate relaxation music when it is analyzed that the user is in a high stress state while working remotely. Specifically, please describe what data you collect, what technology you use to perform emotion analysis, and how you use the generative AI model."

[0823] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0824] Step 1: Data collection

[0825] The user wears a smartwatch, which collects biometric data such as heart rate, blood pressure, and activity level in real time. At the same time, the user's voice is also collected via the smartphone. This data is temporarily stored on the smartphone. The input is the user's biometric data and voice data, and the output is the data stored on the smartphone.

[0826] Step 2: Data transmission and encryption

[0827] The terminal, a smartphone, receives the acquired biometric data and voice data as input. The smartphone encrypts these data using the AES-256 encryption protocol. The encrypted data is then sent to the server. The output is encrypted data, which is sent to the server. The server verifies the integrity and completeness of the received data.

[0828] Step 3: Sentiment analysis

[0829] The server analyzes the received biometric data using a machine learning model. The input is encrypted biometric data and voice data. The server first decrypts it and then analyzes it to estimate the user's emotional state. A model trained on past data is used to analyze the biometric data. At the same time, the voice data is analyzed using an NLP algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to obtain the final emotional state as the output.

[0830] Step 4: Music Generation

[0831] The server generates personalized music using a generative AI model based on the results of the sentiment analysis. The input is the integrated emotional state, the user's music history, and cultural background. The generative AI model takes into account the user's past data and generates new music. The generated music data is re-encrypted and sent to the device. The output is encrypted music data.

[0832] Step 5: Submit your music

[0833] The device decrypts the received music data and passes it to the media player. The input is encrypted music data, and the output is decrypted music data. Users can enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[0834] Step 6: Long-term analysis and feedback

[0835] The server tracks daily emotional fluctuations and stores them in a database. The input is daily emotional data, and the output is the stored data. The server analyzes the stored data and extracts trends and patterns in the user's emotional state. The server creates a report based on the analysis results and sends it to the device. The device receives this and provides it to the user. The output is a report of the analysis results.

[0836] Through this processing step, the system can accurately analyze a user's unique emotional state and provide appropriate music in real time, as well as provide feedback to support the user's long-term psychological wellness.

[0837] (Application example 1)

[0838] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] Conventional food delivery systems have struggled to provide an individually customized experience that takes into account the user's emotional state. The user experience, particularly during the ordering process and waiting time, is limited, and there are few ways to alleviate the stress and anxiety felt by users. As a result, user satisfaction tends to decline. To solve this issue, it is necessary to provide new added value based on the user's emotional state.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0841] In this invention, the server includes a means for providing music during the ordering process or waiting time according to the emotional state of the user, a means for acquiring voice data, and a means for analyzing the acquired voice data and estimating the emotional state, thereby providing an individually customized music experience and reducing stress during the waiting time or ordering process while caring for the user's emotions.

[0842] "Biometric data" refers to data that indicates physical conditions such as heart rate, blood pressure, and activity level obtained through smartwatches and other wearable devices.

[0843] "Voice data" refers to data that records the user's speech and surrounding sounds acquired via a smartphone or other voice input device.

[0844] "Emotional state" refers to the user's psychological state or mood that is estimated based on the analyzed biometric and audio data.

[0845] "Means for generating music" refers to a method or algorithm that creates music to make the user feel a particular emotion, such as relaxation or concentration, based on the acquired emotional state.

[0846] "Means for providing to a user" refers to a method or system for transmitting the generated music to a user's device so that the user can listen to it.

[0847] "Means for providing music during the ordering process and while waiting" refers to a method or system within a food delivery application that plays appropriate music while a user is placing an order or waiting for food to be delivered.

[0848] "Customization means" refers to methods and algorithms that individually tailor the content and style of music to suit the user's emotional state, taking into account the user's preferences and cultural background.

[0849] The present invention is a system that provides individually customized music based on the user's emotional state, and its application example is to improve the user experience in food delivery applications.

[0850] In particular, the server analyzes the user's emotional state sequentially and generates and provides music that matches that state. To achieve this, the following hardware and software are used in combination:

[0851] Hardware and Software Overview

[0852] 1. Smartwatch:

[0853] Used to obtain biometric data (heart rate, blood pressure, activity level) in real time.

[0854] 2. Smartphone:

[0855] Biometric and audio data is temporarily stored and transmitted to a cloud server using an encryption protocol.

[0856] Used to play provided music to the user.

[0857] 3. Cloud Server:

[0858] The server that receives and analyzes the data encrypts it and verifies its integrity and completeness.

[0859] Sentiment estimation is performed using natural language processing (NLP) algorithms and machine learning models.

[0860] Generate music using generative AI models.

[0861] The specific software used is as follows:

[0862] PyCryptodome (used to encrypt data)

[0863] Google Cloud Speech-to-Text API (used to convert voice data into text)

[0864] HuggingFace Transformers (used for NLP analysis of voice data)

[0865] scikit-learn (used for machine learning model analysis of biological data)

[0866] VLC (used to play music on smartphones)

[0867] Detailed system processing flow

[0868] 1. Data Collection:

[0869] When a user wears a smartwatch and connects it to a smartphone, biometric and audio data are collected in real time.

[0870] 2. Data Transmission and Encryption:

[0871] The collected data is encrypted on the smartphone using an encryption protocol (AES-256) and sent to a cloud server, which verifies the data's integrity and completeness.

[0872] 3. Sentiment analysis:

[0873] The cloud server converts the voice data into text using the Google Cloud Speech-to-Text API and estimates the emotional state using HuggingFace Transformers.

[0874] The biometric data is fed into a machine learning model using scikit-learn to analyze the emotional state, and the final emotional state is determined by integrating these data.

[0875] 4. Music Generation:

[0876] Based on the analysis of the emotional state, a prompt sentence is provided to a generative AI model (e.g., GPT-3) to generate music.

[0877] Examples of prompts:

[0878] Emotional state: Stress

[0879] User Preferences: Jazz

[0880] User's cultural background: Japan

[0881] Type of music generated: Relaxing jazz tunes

[0882] 5. Music provided by:

[0883] The generated music is then re-encrypted and sent to a smartphone, where users can listen to it via their smartphone or Bluetooth speaker while waiting or ordering.

[0884] Specific examples

[0885] For example, if a food delivery application determines that a user is in a high stress state, the cloud server generates relaxing jazz music. This music is customized taking into account the user's preferences and cultural background. By listening to this music, users can reduce stress and spend the waiting time more comfortably after placing their order.

[0886] This system allows users to receive appropriate music in real time according to their emotional state, thereby improving satisfaction with food delivery services.

[0887] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0888] Step 1:

[0889] The user wears a smartwatch, and biometric data (heart rate, blood pressure, activity level) is collected in real time via a smartphone. This data is temporarily stored on the smartphone. The input is biometric data, and the output is the biometric data stored on the smartphone.

[0890] Step 2:

[0891] Audio data is collected via the smartphone's microphone. This includes the user's speech and surrounding sounds. This audio data is also stored on the smartphone. The input is audio data, and the output is the audio data stored on the smartphone.

[0892] Step 3:

[0893] The smartphone encrypts the collected biometric and audio data using the AES-256 protocol, ensuring data security: the input is unencrypted data and the output is encrypted data.

[0894] Step 4:

[0895] The encrypted data is sent from the smartphone to a cloud server, which verifies the integrity and completeness of the received data. The input is the encrypted data, and the output is the verified data stored on the server.

[0896] Step 5:

[0897] The cloud server uses the Google Cloud Speech-to-Text API to convert the voice data into text data, which is then used for sentiment analysis. The input is the voice data, and the output is the converted text data.

[0898] Step 6:

[0899] The server uses HuggingFace Transformers to perform natural language processing (NLP) on the transformed text data to estimate the emotional state. The input is the text data, and the output is the estimated emotional state.

[0900] Step 7:

[0901] The server inputs the biometric data into a machine learning model based on scikit-learn to analyze the emotional state. This integrates emotion estimation using natural language processing with biometric data analysis to determine the final emotional state. The input is the biometric data, and the output is the integrated emotional state.

[0902] Step 8:

[0903] The server generates music by inputting a prompt sentence into a generative AI model based on the estimated emotional state. GPT-3 or another model is used as the generative AI model. The input is the prompt sentence, and the output is the generated music data. Specific examples of prompt sentences are as follows:

[0904] Emotional state: Stress

[0905] User Preferences: Jazz

[0906] User's cultural background: Japan

[0907] Type of music generated: Relaxing jazz tunes

[0908] Step 9:

[0909] The generated music data is again encrypted using the AES-256 protocol and sent to the smartphone. The input is unencrypted music data, and the output is encrypted music data.

[0910] Step 10:

[0911] The smartphone receives the encrypted music data, decrypts it, and passes it to the media player. Specifically, the music is played via the VLC library. This allows the user to listen to music optimized for their emotional state in real time. The input is encrypted music data, and the output is decrypted music data.

[0912] By going through the above processing steps, a food delivery application can provide a personalized music experience that corresponds to the user's emotional state.

[0913] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0914] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and audio data, uses an emotion engine to accurately recognize the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[0915] Program processing

[0916] Data collection

[0917] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[0918] Data Transmission and Encryption

[0919] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[0920] Emotion recognition with sentiment analysis and emotion engine

[0921] The server analyzes the received data. The emotion engine combines biometric data and voice data to accurately recognize the emotional state. Natural language processing (NLP) algorithms are used to extract keywords and emotional tones from the voice data to estimate the emotional state. At the same time, the biometric data is also analyzed using a machine learning model to estimate the emotional state. Based on the results of these analyses, the emotion engine determines the final emotional state.

[0922] Music Generation

[0923] Based on the emotional state recognized by the emotion engine, the server uses a generative AI model to generate music. This generation also takes into account the user's past music history, preferences, and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[0924] Music provided by

[0925] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. This provides music that responds to the user's emotional state in real time, supporting psychological wellness.

[0926] Long-term analysis and feedback

[0927] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user. The user can view the reports within the application to check long-term trends in their psychological state.

[0928] Specific examples

[0929] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[0930] This invention makes it possible to effectively provide music that responds to the user's real-time emotional state and provide psychological support, which is designed to contribute to stress reduction, improved quality of life, and increased productivity.

[0931] The processing flow will be explained below.

[0932] Program processing

[0933] Step 1:

[0934] User

[0935] The user puts on the smartwatch and begins their daily activities, and the smartwatch measures heart rate, blood pressure, activity level, and audio data in real time.

[0936] Step 2:

[0937] Smartwatch

[0938] The smartwatch transmits the biometric and audio data it measures to the device (smartphone) via wireless communication (such as Bluetooth).

[0939] Step 3:

[0940] Terminal

[0941] The device temporarily stores the data received from the smartwatch in its memory, and at the same time encrypts the data (using AES-256, etc.) and prepares it for transmission to the server.

[0942] Step 4:

[0943] Terminal

[0944] The encrypted data is sent to the server in real time, ensuring data security.

[0945] Step 5:

[0946] server

[0947] The server decrypts the encrypted data received from the device and verifies the integrity and completeness of the data, after which it is ready for analysis.

[0948] Step 6:

[0949] server

[0950] The server analyzes the received data. The emotion engine comprehensively analyzes the voice data and biometric data to recognize the user's emotional state. Natural language processing (NLP) algorithms are used to extract emotional tones and keywords from the voice data, and machine learning models are used to estimate the user's emotional state from the biometric data.

[0951] Step 7:

[0952] server

[0953] Based on the analysis results of the emotion engine, the server determines the user's emotional state, for example, determining that the user is in a high stress state.

[0954] Step 8:

[0955] server

[0956] The server uses a generative AI model to generate music that best suits the user's emotional state, taking into account the user's preferences and cultural background. The generated music data is then re-encrypted.

[0957] Step 9:

[0958] server

[0959] The server sends the encrypted music data to the terminal, which then transmits it immediately.

[0960] Step 10:

[0961] Terminal

[0962] The device decrypts the encrypted music data received from the server, and the decrypted music data is passed to the media player.

[0963] Step 11:

[0964] Terminal

[0965] The device connects to the user's smartphone or Bluetooth speaker and prepares to play music.

[0966] Step 12:

[0967] User

[0968] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[0969] Step 13:

[0970] server

[0971] The server tracks daily emotional data and stores it in a database, which allows for long-term data analysis.

[0972] Step 14:

[0973] server

[0974] Analyze the accumulated data, extract trends and patterns, and create reports for users to support long-term psychological improvement.

[0975] Step 15:

[0976] Terminal

[0977] The device receives the report from the server and provides it to the user, who can then view the report through the application to check trends and areas for improvement in their own psychological state.

[0978] In this way, the system of the present invention can provide optimal music based on the user's real-time emotional state and provide psychological support.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] In modern society, people's stress and psychological burden are increasing, but a system for effectively alleviating these stresses and providing users with music that suits their needs has not yet been fully established. Conventional systems have difficulty providing personalized music based on each user's real-time emotional state. There is a need to solve this problem and improve users' psychological wellness.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0983] In this invention, the server includes means for acquiring biometric data from a wearable device worn by a user, means for temporarily storing the acquired biometric data in the terminal and transmitting it to the server using an encryption protocol, means for analyzing the encrypted data and estimating an emotional state using an emotion engine, means for generating music using a generative AI model based on the estimated emotional state, means for encrypting the generated music data and transmitting it to the terminal, means for playing the decrypted music on the terminal via a media player and providing it to the user, and means for tracking and analyzing daily emotional data over the long term and providing feedback to the user. This makes it possible to provide personalized music based on each user's real-time emotional state, which is expected to help relieve stress and improve psychological well-being.

[0984] "Biometric data" refers to data related to the user's physical condition, including heart rate, blood pressure, activity level, and the like.

[0985] A "wearable device" is an electronic device worn by a user, including smartwatches and fitness trackers.

[0986] "Encryption protocols" are technologies used to ensure data security, including algorithms such as AES-256.

[0987] An "emotion engine" is a computer system or algorithm that analyzes biometric and audio data to accurately estimate a user's emotional state.

[0988] A "generative AI model" is an artificial intelligence model that uses machine learning to perform specific tasks, and is used for applications such as music generation and text generation.

[0989] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, tablets, etc.

[0990] A "server" is a centralized computer system for processing and storing data over a network.

[0991] "Voice data" refers to data that records the user's speaking voice or sounds, and is used for voice recognition and emotion analysis.

[0992] A "media player" is a software or hardware device for playing media content such as music and videos.

[0993] "Analyzing encrypted data" means performing processing to decrypt the encrypted data and read the contents.

[0994] "Feedback" refers to the response or report that a system provides to a user, providing information based on the user's actions and emotions.

[0995] This invention relates to a system that generates and provides optimal music based on a user's emotional state. This system uses a wearable device to acquire biometric and audio data, analyzes this data to recognize the user's emotional state, and generates music using a generative AI model. The detailed configuration and processing flow are described below.

[0996] Hardware and Software Configuration

[0997] 1. Wearable devices

[0998] Wearable devices such as smartwatches and fitness trackers worn by users collect biometric data (heart rate, blood pressure, activity level) in real time and transmit the data to a device (e.g., a smartphone) via Bluetooth.

[0999] 2. Terminal

[1000] The terminal temporarily stores biometric data transmitted from the wearable device via Bluetooth. The terminal also has the function of collecting the user's voice data via a microphone. The biometric and voice data are encrypted using a secure encryption protocol (e.g., AES-256) and transmitted to the server via the HTTPS protocol.

[1001] 3. Server

[1002] The server receives the encrypted data and verifies its integrity. The emotion engine on the server analyzes the biometric and audio data to estimate the user's emotional state. Specifically, it uses machine learning models (e.g., TensorFlow, PyTorch) for the biometric data and natural language processing (NLP) algorithms for the audio data. Based on the estimated emotional state, the server generates music using a generative AI model (e.g., GPT-4, DALL-E).

[1003] 4. Generative AI Models

[1004] The generative AI model generates music based on the emotional state recognized by the emotion engine, taking into account the user's past music history, preferences, and cultural background. One example prompt is: "The user's current emotional state is high stress. Please generate instrumental music that has a relaxing effect appropriate for this state. Also, since the user has previously enjoyed listening to classical music, please take that into consideration."

[1005] 5. Providing generated music

[1006] The generated music data is then encrypted again and sent to the device. The device then decrypts the received music data and passes it to a media player. The user can then listen to the generated music via a smartphone or Bluetooth speaker. This provides personalized music that responds to the user's emotional state in real time.

[1007] Specific examples

[1008] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[1009] This invention effectively supports psychological wellness by providing personalized music based on a user's real-time emotional state, which contributes to stress reduction, improved quality of life, and increased productivity.

[1010] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1011] Step 1: Data collection

[1012] A user wears a wearable device (such as a smartwatch or fitness tracker). This device collects biometric data such as heart rate, blood pressure, and activity level in real time. The collected biometric data is sent to a device (e.g., a smartphone) via Bluetooth and temporarily stored in its internal storage. Voice data of the user's speech is also collected by the device via a microphone.

[1013] Input: User biometric and voice data

[1014] Output: Biometric data and voice data temporarily stored on the device

[1015] Step 2: Data transmission and encryption

[1016] The device encrypts the temporarily stored biometric and voice data using an encryption protocol (e.g., AES-256). The encrypted data is sent to the server via the HTTPS protocol. The server receives the encrypted data and performs a hash check to verify the integrity and completeness of the data.

[1017] Input: Biometric data and voice data temporarily stored on the device

[1018] Output: Data encrypted and sent to the server

[1019] Step 3: Sentiment analysis and emotion recognition

[1020] The server decrypts the encrypted data and begins analysis. Biometric data is analyzed using machine learning models (e.g., TensorFlow, PyTorch), and audio data is analyzed using natural language processing (NLP) algorithms. The emotion engine combines these analysis results to estimate the user's emotional state.

[1021] Input: Encrypted data sent to the server

[1022] Output: Estimated emotional state

[1023] Step 4: Music Generation

[1024] The server generates music using a generative AI model (e.g., GPT-4, DALL-E) based on the emotional state estimated by the emotion engine. This generation also takes into account the user's past music history, preferences, and cultural background. The generated music data is encrypted and sent to the device.

[1025] Input: estimated emotional state, past music history, user preferences, cultural background

[1026] Output: Generated music data

[1027] Step 5: Submit your music

[1028] The device decrypts the encrypted music data received from the server. The decrypted music data is passed to the device's media player and played back to the user. The user can then listen to the music via their smartphone or Bluetooth speaker.

[1029] Input: Encrypted music data sent to the device

[1030] Output: Music data passed to a media player and played to the user

[1031] Step 6: Long-term analysis and feedback

[1032] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and a report is created to support the user's long-term improvement of their psychological state. The report is sent to the device and can be viewed by the user within the application.

[1033] Input: Daily emotion data

[1034] Output: Report provided to the user

[1035] Through these steps, the system can provide personalized music according to the user's real-time emotional state and support psychological wellness.

[1036] (Application example 2)

[1037] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] In modern brick-and-mortar stores, providing music that best suits the emotional state of customers in real time is an important factor in improving customer satisfaction. However, conventional systems have low accuracy in recognizing emotional states, making it difficult to provide appropriate music. Furthermore, security and privacy issues have made it difficult to safely handle biometric and voice data. There is a need to solve these issues and safely and effectively provide music that matches the emotional state of customers.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric data and voice data, means for encrypting the biometric data and voice data, means for decrypting and analyzing the encrypted data, means for generating music based on the emotional state, and means for encrypting and providing the generated music. This makes it possible to provide music that is optimal for the emotional state of customers in a physical store in real time while safely handling the biometric data and voice data.

[1040] "Biometric data" refers to data that measures a user's physiological indicators, such as heart rate, blood pressure, and activity level, in real time.

[1041] "Voice data" refers to voice information uttered by a user, and is data from which emotional tones and keywords can be extracted by analyzing the voice information.

[1042] "Encryption means" refers to a technology that encrypts acquired data using an encryption protocol (e.g., AES-256) to ensure security.

[1043] The "server" is a computer system that receives encrypted data, decrypts and analyzes it, estimates the user's emotional state, and generates music.

[1044] "Emotional state" refers to the user's psychological and emotional state, and is estimated from the analysis of heart rate, blood pressure, activity level, and voice data.

[1045] A "generative AI model" is an artificial intelligence algorithm that generates appropriate music based on emotional state.

[1046] A "user terminal" is a device used by a user (e.g., a smartphone or tablet) that receives and decodes data sent from the server and provides music.

[1047] A "physical store" is a commercial facility that operates in a physical form, and is a place that aims to analyze the emotions of customers in real time and provide appropriate music.

[1048] "Real-time" means that data is generated, processed, and results are provided with almost no delay.

[1049] This invention is a system for generating and providing optimal music based on the emotional state of customers in a physical store. The system includes a server, encryption means, a user terminal, biometric data acquisition means, voice data acquisition means, emotion estimation means, and a generative AI model.

[1050] The server first receives and decrypts the encrypted data sent from the means for acquiring biometric and voice data at the physical store. The AES-256 protocol is used for this encryption. The decrypted data is then analyzed by the emotion estimation means to estimate the user's emotional state. Natural language processing (NLP) algorithms and machine learning models are used for emotion estimation.

[1051] Based on the estimated emotional state, the generative AI model on the server generates optimal music in real time, taking into account the user's past music history, preferences, and even cultural background. The generated music data is then re-encrypted and sent to the user's device.

[1052] The user terminal decrypts the encrypted music data sent from the server and provides it to customers through the store's speaker system. This process is carried out in real time, and music that matches the customer's emotional state is provided, thereby improving customer satisfaction.

[1053] As a specific example, if a customer in a store is estimated to be feeling highly stressed, the generative AI model will generate music with a relaxing effect. This music is customized based on the customer's past music history and preferences. The user's device receives the music data sent from the server and plays it through speakers in the store. This helps the customer relax and reduce stress.

[1054] The system is realized by combining various hardware and software components, including a smartwatch, microphone device, and speaker system, and software components, including AES-256 encryption protocols, NLP algorithms, machine learning models, and generative AI models.

[1055] An example of a prompt is:

[1056] "Design an AI model that analyzes a user's emotional state based on their heart rate, blood pressure, activity level, and voice data. Then write a program to generate relaxing music based on that specific emotional state. Include any necessary encryption techniques."

[1057] Examples include:

[1058] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1059] Step 1:

[1060] The server collects biometric data (heart rate, blood pressure, activity level) and voice data from robots and smart devices in the physical store. The input is raw data from the sensor devices, which is received and stored in real time. The output is unencrypted biometric data and voice data.

[1061] Step 2:

[1062] The terminal encrypts the acquired biometric and voice data using the AES-256 protocol. The input is raw data, which is encrypted to ensure data security. The output is encrypted data.

[1063] Step 3:

[1064] The device sends encrypted biometric data and voice data to the server. The input is encrypted data that is securely transferred to the server over the network. The output is the data as it was sent to the server.

[1065] Step 4:

[1066] The server decrypts the received encrypted data using the AES-256 protocol. The input is the encrypted data, and the original raw data is obtained by decrypting it. The output is the decrypted biometric data and audio data.

[1067] Step 5:

[1068] The server analyzes the decoded biometric data and voice data to estimate the user's emotional state. Here, the voice data is analyzed using NLP algorithms to extract emotional tones and keywords. The biometric data is then used to estimate the emotional state using machine learning models. The input is the decoded data, and the output is the estimated emotional state.

[1069] Step 6:

[1070] The server generates music using a generative AI model based on the estimated emotional state. The input is emotional state data, and the generative AI model generates music data that takes into account the user's preferences and cultural background. The output is the generated music data.

[1071] Step 7:

[1072] The server then encrypts the generated music data again using the AES-256 protocol. The input is the music data, and the encryption process prepares it for secure transmission to the user's device. The output is the encrypted music data.

[1073] Step 8:

[1074] The device receives encrypted music data sent from the server and decrypts it using the AES-256 protocol. The input is the encrypted music data, and the original music data is obtained by decrypting it. The output is the decrypted music data.

[1075] Step 9:

[1076] The terminal plays the decoded music data through the speaker system in the physical store. The input is music data, and the output is music provided to customers. This makes it possible to provide music that corresponds to the emotional state of customers in real time.

[1077] This series of processes realizes a system that provides music that best suits the emotional state of customers in a physical store.

[1078] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1079] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1080] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1081] [Fourth embodiment]

[1082] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1083] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1084] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1085] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1086] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1087] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1088] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1089] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1090] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1091] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1092] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1093] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1094] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1095] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[1096] Program processing

[1097] Data collection

[1098] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[1099] Data Transmission and Encryption

[1100] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[1101] sentiment analysis

[1102] The server analyzes the received data. It uses natural language processing (NLP) algorithms to infer emotions from the voice data. At the same time, it analyzes biometric data through machine learning models to infer emotional states. These results are then combined to determine the final emotional state.

[1103] Music Generation

[1104] Based on the results of the sentiment analysis, the server uses a generative AI model to generate music. This takes into account the user's preferences and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[1105] Music provided by

[1106] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. Even during this process, the music is provided in real time, allowing users to instantly enjoy music that matches their emotional state.

[1107] Long-term analysis and feedback

[1108] The server tracks daily emotional fluctuations and stores them in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user.

[1109] Specific examples

[1110] For example, if a user is analyzed as being in a high stress state while working remotely, the server will generate relaxation music. Since stress can be detected from the voice data, the generated music is a highly relaxing instrumental piece tailored to the user's preferences. The device then plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[1111] This system can effectively support users' psychological wellness, contributing to improving quality of life, reducing stress, and increasing productivity.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] User

[1115] The user wears a smartwatch, which measures heart rate, blood pressure, activity, and audio data in real time. Data is collected continuously as the user goes about their daily activities.

[1116] Step 2:

[1117] Smartwatch

[1118] The smartwatch transmits the collected biometric and audio data to a device (e.g., a smartphone) using wireless communication such as Bluetooth.

[1119] Step 3:

[1120] Terminal

[1121] The device temporarily stores the data received from the smartwatch in its memory, encrypts the stored data using an encryption method (e.g., AES-256), and periodically transmits the encrypted data to the server.

[1122] Step 4:

[1123] server

[1124] The server receives the encrypted data sent from the device. After receiving it, it decrypts the data and verifies its consistency and integrity. After verification, the data is ready for analysis.

[1125] Step 5:

[1126] server

[1127] The server analyzes the acquired data. It uses a natural language processing (NLP) algorithm to extract keywords and emotional tones from the voice data and estimate the emotional state. At the same time, it uses a machine learning model to analyze biometric data (heart rate, blood pressure, activity level) to estimate the emotional state. The analysis results of the voice data and biometric data are integrated to determine the final emotional state.

[1128] Step 6:

[1129] server

[1130] The server generates music using a generative AI model based on the estimated emotional state, taking into account the user's past music history, preferences, and cultural background. The generated music data is then re-encrypted and sent to the device.

[1131] Step 7:

[1132] Terminal

[1133] The device decrypts the encrypted music data received from the server, passes the decrypted music data to the media player, and prepares it for playback. The device then connects to a smartphone or Bluetooth speaker for playback.

[1134] Step 8:

[1135] User

[1136] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[1137] Step 9:

[1138] server

[1139] The server tracks daily emotional data and stores it in a database. It then performs long-term analysis to extract trends and patterns, and creates reports to help users improve their psychological state.

[1140] Step 10:

[1141] Terminal

[1142] The device receives the report from the server and provides it to the user, who can then view the report in the application to check long-term trends in their psychological state and take steps to improve their daily habits and work environment.

[1143] Through these specific processing steps, the system of the present invention can provide music that corresponds to the user's real-time emotional state and provide psychological support.

[1144] Example 1

[1145] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1146] Conventional emotional state estimation systems that use biometric and audio data have difficulty accurately analyzing a user's unique emotional state and generating and providing appropriate music. Furthermore, they lack the ability to grasp a user's long-term emotional trends and provide feedback. This makes it difficult to comprehensively support users' psychological wellness.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1148] In this invention, the server includes means for acquiring biometric data, means for analyzing the acquired biometric data and transmitting it to the server using an encryption protocol, means for integrating the analyzed biometric data with audio data to estimate the user's emotional state, means for generating music based on the estimated emotional state using a generative AI model, means for transmitting the generated music to the terminal and providing it to the user, and means for analyzing the user's long-term emotional trends and patterns and providing feedback. This allows the system to accurately analyze the user's unique emotional state and provide appropriate music in real time. It also provides feedback to support the user's long-term psychological wellness.

[1149] "Biometric data" refers to data that indicates the user's physical condition in real time, such as heart rate, blood pressure, and activity level.

[1150] "Encryption protocol" refers to a standard method used to protect the confidentiality of data, such as AES-256.

[1151] A "server" is a computer system that receives, analyzes, and transmits data over a network.

[1152] "Inference tools" refer to algorithms or machine learning models that analyze collected data and derive specific results.

[1153] A "generative AI model" is an artificial intelligence model that generates a specific output (in this case, music) by taking into account the user's emotional state and preferences.

[1154] A "terminal" is a device used by a user, such as a smartphone or tablet, that transmits, receives, and displays data.

[1155] "Emotional state" is an indicator of the user's psychological and emotional state, and includes states such as stress and relaxation.

[1156] "Analyzing" is the process of examining collected data in detail and extracting specific information or patterns.

[1157] "Integrating" refers to combining information from multiple data sources to gain a holistic understanding and analysis.

[1158] "Feedback" means information or advice provided to users that is based on past data and analysis.

[1159] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and voice data, estimates the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[1160] Data collection

[1161] The user wears a smartwatch, which collects real-time biometric data such as heart rate, blood pressure, and activity level. The watch also collects the user's voice via the smartphone, and this data is temporarily stored on the smartphone.

[1162] Data Transmission and Encryption

[1163] The smartphone device encrypts the collected biometric and voice data using the AES-256 encryption protocol, and then transmits the encrypted data to a server, which verifies the integrity and completeness of the received data.

[1164] sentiment analysis

[1165] The server analyzes the received biometric data using a machine learning model to estimate the user's emotional state. This model is trained from past data and can recognize stress and relaxation. The server also analyzes the voice data using an NLP (natural language processing) algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to determine the user's final emotional state.

[1166] Music Generation

[1167] The server then uses a generative AI model to generate personalized music based on the results of the sentiment analysis. This model also takes into account the user's musical history and cultural background. The generated music data is then re-encrypted and sent to the device.

[1168] Music provided by

[1169] The device decodes the received music data and passes it to the media player, allowing users to enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[1170] Long-term analysis and feedback

[1171] The server tracks daily emotional fluctuations and stores them in a database. The stored data is then analyzed to extract trends and patterns in the user's emotional state. The server then creates a report based on the analysis results and sends it to the device. The device receives this report and provides it to the user, allowing the user to understand the history of their own emotional state.

[1172] Specific examples

[1173] For example, if a user is analyzed as being in a state of high stress while working remotely, the server will generate relaxation music. The generative AI model will select instrumental songs that the user has previously found relaxing and create new music based on these. The device will receive the generated music data and pass it to the user's media player. The user can immediately listen to the relaxation music and reduce stress.

[1174] Prompt Sentence Examples

[1175] "Please explain the system that generates and provides appropriate relaxation music when it is analyzed that the user is in a high stress state while working remotely. Specifically, please describe what data you collect, what technology you use to perform emotion analysis, and how you use the generative AI model."

[1176] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1177] Step 1: Data collection

[1178] The user wears a smartwatch, which collects biometric data such as heart rate, blood pressure, and activity level in real time. At the same time, the user's voice is also collected via the smartphone. This data is temporarily stored on the smartphone. The input is the user's biometric data and voice data, and the output is the data stored on the smartphone.

[1179] Step 2: Data transmission and encryption

[1180] The terminal, a smartphone, receives the acquired biometric data and voice data as input. The smartphone encrypts these data using the AES-256 encryption protocol. The encrypted data is then sent to the server. The output is encrypted data, which is sent to the server. The server verifies the integrity and completeness of the received data.

[1181] Step 3: Sentiment analysis

[1182] The server analyzes the received biometric data using a machine learning model. The input is encrypted biometric data and voice data. The server first decrypts it and then analyzes it to estimate the user's emotional state. A model trained on past data is used to analyze the biometric data. At the same time, the voice data is analyzed using an NLP algorithm to diagnose the user's emotions based on what they are saying and their tone of voice. These results are combined to obtain the final emotional state as the output.

[1183] Step 4: Music Generation

[1184] The server generates personalized music using a generative AI model based on the results of the sentiment analysis. The input is the integrated emotional state, the user's music history, and cultural background. The generative AI model takes into account the user's past data and generates new music. The generated music data is re-encrypted and sent to the device. The output is encrypted music data.

[1185] Step 5: Submit your music

[1186] The device decrypts the received music data and passes it to the media player. The input is encrypted music data, and the output is decrypted music data. Users can enjoy personalized music in real time through their smartphones or Bluetooth speakers.

[1187] Step 6: Long-term analysis and feedback

[1188] The server tracks daily emotional fluctuations and stores them in a database. The input is daily emotional data, and the output is the stored data. The server analyzes the stored data and extracts trends and patterns in the user's emotional state. The server creates a report based on the analysis results and sends it to the device. The device receives this and provides it to the user. The output is a report of the analysis results.

[1189] Through this processing step, the system can accurately analyze a user's unique emotional state and provide appropriate music in real time, as well as provide feedback to support the user's long-term psychological wellness.

[1190] (Application example 1)

[1191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1192] Conventional food delivery systems have struggled to provide an individually customized experience that takes into account the user's emotional state. The user experience, particularly during the ordering process and waiting time, is limited, and there are few ways to alleviate the stress and anxiety felt by users. As a result, user satisfaction tends to decline. To solve this issue, it is necessary to provide new added value based on the user's emotional state.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1194] In this invention, the server includes a means for providing music during the ordering process or waiting time according to the emotional state of the user, a means for acquiring voice data, and a means for analyzing the acquired voice data and estimating the emotional state, thereby providing an individually customized music experience and reducing stress during the waiting time or ordering process while caring for the user's emotions.

[1195] "Biometric data" refers to data that indicates physical conditions such as heart rate, blood pressure, and activity level obtained through smartwatches and other wearable devices.

[1196] "Voice data" refers to data that records the user's speech and surrounding sounds acquired via a smartphone or other voice input device.

[1197] "Emotional state" refers to the user's psychological state or mood that is estimated based on the analyzed biometric and audio data.

[1198] "Means for generating music" refers to a method or algorithm that creates music to make the user feel a particular emotion, such as relaxation or concentration, based on the acquired emotional state.

[1199] "Means for providing to a user" refers to a method or system for transmitting the generated music to a user's device so that the user can listen to it.

[1200] "Means for providing music during the ordering process and while waiting" refers to a method or system within a food delivery application that plays appropriate music while a user is placing an order or waiting for food to be delivered.

[1201] "Customization means" refers to methods and algorithms that individually tailor the content and style of music to suit the user's emotional state, taking into account the user's preferences and cultural background.

[1202] The present invention is a system that provides individually customized music based on the user's emotional state, and its application example is to improve the user experience in food delivery applications.

[1203] In particular, the server analyzes the user's emotional state sequentially and generates and provides music that matches that state. To achieve this, the following hardware and software are used in combination:

[1204] Hardware and Software Overview

[1205] 1. Smartwatch:

[1206] Used to obtain biometric data (heart rate, blood pressure, activity level) in real time.

[1207] 2. Smartphone:

[1208] Biometric and audio data is temporarily stored and transmitted to a cloud server using an encryption protocol.

[1209] Used to play provided music to the user.

[1210] 3. Cloud Server:

[1211] The server that receives and analyzes the data encrypts it and verifies its integrity and completeness.

[1212] Sentiment estimation is performed using natural language processing (NLP) algorithms and machine learning models.

[1213] Generate music using generative AI models.

[1214] The specific software used is as follows:

[1215] PyCryptodome (used to encrypt data)

[1216] Google Cloud Speech-to-Text API (used to convert voice data into text)

[1217] HuggingFace Transformers (used for NLP analysis of voice data)

[1218] scikit-learn (used for machine learning model analysis of biological data)

[1219] VLC (used to play music on smartphones)

[1220] Detailed system processing flow

[1221] 1. Data Collection:

[1222] When a user wears a smartwatch and connects it to a smartphone, biometric and audio data are collected in real time.

[1223] 2. Data Transmission and Encryption:

[1224] The collected data is encrypted on the smartphone using an encryption protocol (AES-256) and sent to a cloud server, which verifies the data's integrity and completeness.

[1225] 3. Sentiment analysis:

[1226] The cloud server converts the voice data into text using the Google Cloud Speech-to-Text API and estimates the emotional state using HuggingFace Transformers.

[1227] The biometric data is fed into a machine learning model using scikit-learn to analyze the emotional state, and the final emotional state is determined by integrating these data.

[1228] 4. Music Generation:

[1229] Based on the analysis of the emotional state, a prompt sentence is provided to a generative AI model (e.g., GPT-3) to generate music.

[1230] Examples of prompts:

[1231] Emotional state: Stress

[1232] User Preferences: Jazz

[1233] User's cultural background: Japan

[1234] Type of music generated: Relaxing jazz tunes

[1235] 5. Music provided by:

[1236] The generated music is then re-encrypted and sent to a smartphone, where users can listen to it via their smartphone or Bluetooth speaker while waiting or ordering.

[1237] Specific examples

[1238] For example, if a food delivery application determines that a user is in a high stress state, the cloud server generates relaxing jazz music. This music is customized taking into account the user's preferences and cultural background. By listening to this music, users can reduce stress and spend the waiting time more comfortably after placing their order.

[1239] This system allows users to receive appropriate music in real time according to their emotional state, thereby improving satisfaction with food delivery services.

[1240] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1241] Step 1:

[1242] The user wears a smartwatch, and biometric data (heart rate, blood pressure, activity level) is collected in real time via a smartphone. This data is temporarily stored on the smartphone. The input is biometric data, and the output is the biometric data stored on the smartphone.

[1243] Step 2:

[1244] Audio data is collected via the smartphone's microphone. This includes the user's speech and surrounding sounds. This audio data is also stored on the smartphone. The input is audio data, and the output is the audio data stored on the smartphone.

[1245] Step 3:

[1246] The smartphone encrypts the collected biometric and audio data using the AES-256 protocol, ensuring data security: the input is unencrypted data and the output is encrypted data.

[1247] Step 4:

[1248] The encrypted data is sent from the smartphone to a cloud server, which verifies the integrity and completeness of the received data. The input is the encrypted data, and the output is the verified data stored on the server.

[1249] Step 5:

[1250] The cloud server uses the Google Cloud Speech-to-Text API to convert the voice data into text data, which is then used for sentiment analysis. The input is the voice data, and the output is the converted text data.

[1251] Step 6:

[1252] The server uses HuggingFace Transformers to perform natural language processing (NLP) on the transformed text data to estimate the emotional state. The input is the text data, and the output is the estimated emotional state.

[1253] Step 7:

[1254] The server inputs the biometric data into a machine learning model based on scikit-learn to analyze the emotional state. This integrates emotion estimation using natural language processing with biometric data analysis to determine the final emotional state. The input is the biometric data, and the output is the integrated emotional state.

[1255] Step 8:

[1256] The server generates music by inputting a prompt sentence into a generative AI model based on the estimated emotional state. GPT-3 or another model is used as the generative AI model. The input is the prompt sentence, and the output is the generated music data. Specific examples of prompt sentences are as follows:

[1257] Emotional state: Stress

[1258] User Preferences: Jazz

[1259] User's cultural background: Japan

[1260] Type of music generated: Relaxing jazz tunes

[1261] Step 9:

[1262] The generated music data is again encrypted using the AES-256 protocol and sent to the smartphone. The input is unencrypted music data, and the output is encrypted music data.

[1263] Step 10:

[1264] The smartphone receives the encrypted music data, decrypts it, and passes it to the media player. Specifically, the music is played via the VLC library. This allows the user to listen to music optimized for their emotional state in real time. The input is encrypted music data, and the output is decrypted music data.

[1265] By going through the above processing steps, a food delivery application can provide a personalized music experience that corresponds to the user's emotional state.

[1266] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1267] This invention relates to a system that generates and provides optimal music based on the user's emotional state. This system acquires and analyzes biometric and audio data, uses an emotion engine to accurately recognize the user's emotional state, and then generates and provides music taking into account the user's preferences and cultural background.

[1268] Program processing

[1269] Data collection

[1270] When a user wears a smartwatch, biometric data (heart rate, blood pressure, activity level) is collected in real time. Voice data is also collected at the same time. This data is sent to a device (e.g., a smartphone) and temporarily stored.

[1271] Data Transmission and Encryption

[1272] The device encrypts the collected data using an encryption protocol (e.g., AES-256) and sends it to the server, which ensures data security. The server then verifies the integrity and completeness of the received data.

[1273] Emotion recognition with sentiment analysis and emotion engine

[1274] The server analyzes the received data. The emotion engine combines biometric data and voice data to accurately recognize the emotional state. Natural language processing (NLP) algorithms are used to extract keywords and emotional tones from the voice data to estimate the emotional state. At the same time, the biometric data is also analyzed using a machine learning model to estimate the emotional state. Based on the results of these analyses, the emotion engine determines the final emotional state.

[1275] Music Generation

[1276] Based on the emotional state recognized by the emotion engine, the server uses a generative AI model to generate music. This generation also takes into account the user's past music history, preferences, and cultural background, resulting in personalized music. The generated music data is then re-encrypted and sent to the device.

[1277] Music provided by

[1278] The device decodes the music data received from the server and passes it to the media player. The user can then listen to the generated music via their smartphone or Bluetooth speaker. This provides music that responds to the user's emotional state in real time, supporting psychological wellness.

[1279] Long-term analysis and feedback

[1280] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and reports are created to support long-term improvement of the user's psychological state. The device receives these reports and provides them to the user. The user can view the reports within the application to check long-term trends in their psychological state.

[1281] Specific examples

[1282] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[1283] This invention makes it possible to effectively provide music that responds to the user's real-time emotional state and provide psychological support, which is designed to contribute to stress reduction, improved quality of life, and increased productivity.

[1284] The processing flow will be explained below.

[1285] Program processing

[1286] Step 1:

[1287] User

[1288] The user puts on the smartwatch and begins their daily activities, and the smartwatch measures heart rate, blood pressure, activity level, and audio data in real time.

[1289] Step 2:

[1290] Smartwatch

[1291] The smartwatch transmits the biometric and audio data it measures to the device (smartphone) via wireless communication (such as Bluetooth).

[1292] Step 3:

[1293] Terminal

[1294] The device temporarily stores the data received from the smartwatch in its memory, and at the same time encrypts the data (using AES-256, etc.) and prepares it for transmission to the server.

[1295] Step 4:

[1296] Terminal

[1297] The encrypted data is sent to the server in real time, ensuring data security.

[1298] Step 5:

[1299] server

[1300] The server decrypts the encrypted data received from the device and verifies the integrity and completeness of the data, after which it is ready for analysis.

[1301] Step 6:

[1302] server

[1303] The server analyzes the received data. The emotion engine comprehensively analyzes the voice data and biometric data to recognize the user's emotional state. Natural language processing (NLP) algorithms are used to extract emotional tones and keywords from the voice data, and machine learning models are used to estimate the user's emotional state from the biometric data.

[1304] Step 7:

[1305] server

[1306] Based on the analysis results of the emotion engine, the server determines the user's emotional state, for example, determining that the user is in a high stress state.

[1307] Step 8:

[1308] server

[1309] The server uses a generative AI model to generate music that best suits the user's emotional state, taking into account the user's preferences and cultural background. The generated music data is then re-encrypted.

[1310] Step 9:

[1311] server

[1312] The server sends the encrypted music data to the terminal, which then transmits it immediately.

[1313] Step 10:

[1314] Terminal

[1315] The device decrypts the encrypted music data received from the server, and the decrypted music data is passed to the media player.

[1316] Step 11:

[1317] Terminal

[1318] The device connects to the user's smartphone or Bluetooth speaker and prepares to play music.

[1319] Step 12:

[1320] User

[1321] Users listen to the generated music through their smartphone or Bluetooth speaker, which provides music tailored to their emotional state in real time, providing a sense of relaxation and improved concentration.

[1322] Step 13:

[1323] server

[1324] The server tracks daily emotional data and stores it in a database, which allows for long-term data analysis.

[1325] Step 14:

[1326] server

[1327] Analyze the accumulated data, extract trends and patterns, and create reports for users to support long-term psychological improvement.

[1328] Step 15:

[1329] Terminal

[1330] The device receives the report from the server and provides it to the user, who can then view the report through the application to check trends and areas for improvement in their own psychological state.

[1331] In this way, the system of the present invention can provide optimal music based on the user's real-time emotional state and provide psychological support.

[1332] Example 2

[1333] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1334] In modern society, people's stress and psychological burden are increasing, but a system for effectively alleviating these stresses and providing users with music that suits their needs has not yet been fully established. Conventional systems have difficulty providing personalized music based on each user's real-time emotional state. There is a need to solve this problem and improve users' psychological wellness.

[1335] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1336] In this invention, the server includes means for acquiring biometric data from a wearable device worn by a user, means for temporarily storing the acquired biometric data in the terminal and transmitting it to the server using an encryption protocol, means for analyzing the encrypted data and estimating an emotional state using an emotion engine, means for generating music using a generative AI model based on the estimated emotional state, means for encrypting the generated music data and transmitting it to the terminal, means for playing the decrypted music on the terminal via a media player and providing it to the user, and means for tracking and analyzing daily emotional data over the long term and providing feedback to the user. This makes it possible to provide personalized music based on each user's real-time emotional state, which is expected to help relieve stress and improve psychological well-being.

[1337] "Biometric data" refers to data related to the user's physical condition, including heart rate, blood pressure, activity level, and the like.

[1338] A "wearable device" is an electronic device worn by a user, including smartwatches and fitness trackers.

[1339] "Encryption protocols" are technologies used to ensure data security, including algorithms such as AES-256.

[1340] An "emotion engine" is a computer system or algorithm that analyzes biometric and audio data to accurately estimate a user's emotional state.

[1341] A "generative AI model" is an artificial intelligence model that uses machine learning to perform specific tasks, and is used for applications such as music generation and text generation.

[1342] A "terminal" is an electronic device that is directly operated by a user, and includes smartphones, tablets, etc.

[1343] A "server" is a centralized computer system for processing and storing data over a network.

[1344] "Voice data" refers to data that records the user's speaking voice or sounds, and is used for voice recognition and emotion analysis.

[1345] A "media player" is a software or hardware device for playing media content such as music and videos.

[1346] "Analyzing encrypted data" means performing processing to decrypt the encrypted data and read the contents.

[1347] "Feedback" refers to the response or report that a system provides to a user, providing information based on the user's actions and emotions.

[1348] This invention relates to a system that generates and provides optimal music based on a user's emotional state. This system uses a wearable device to acquire biometric and audio data, analyzes this data to recognize the user's emotional state, and generates music using a generative AI model. The detailed configuration and processing flow are described below.

[1349] Hardware and Software Configuration

[1350] 1. Wearable devices

[1351] Wearable devices such as smartwatches and fitness trackers worn by users collect biometric data (heart rate, blood pressure, activity level) in real time and transmit the data to a device (e.g., a smartphone) via Bluetooth.

[1352] 2. Terminal

[1353] The terminal temporarily stores biometric data transmitted from the wearable device via Bluetooth. The terminal also has the function of collecting the user's voice data via a microphone. The biometric and voice data are encrypted using a secure encryption protocol (e.g., AES-256) and transmitted to the server via the HTTPS protocol.

[1354] 3. Server

[1355] The server receives the encrypted data and verifies its integrity. The emotion engine on the server analyzes the biometric and audio data to estimate the user's emotional state. Specifically, it uses machine learning models (e.g., TensorFlow, PyTorch) for the biometric data and natural language processing (NLP) algorithms for the audio data. Based on the estimated emotional state, the server generates music using a generative AI model (e.g., GPT-4, DALL-E).

[1356] 4. Generative AI Models

[1357] The generative AI model generates music based on the emotional state recognized by the emotion engine, taking into account the user's past music history, preferences, and cultural background. One example prompt is: "The user's current emotional state is high stress. Please generate instrumental music that has a relaxing effect appropriate for this state. Also, since the user has previously enjoyed listening to classical music, please take that into consideration."

[1358] 5. Providing generated music

[1359] The generated music data is then encrypted again and sent to the device. The device then decrypts the received music data and passes it to a media player. The user can then listen to the generated music via a smartphone or Bluetooth speaker. This provides personalized music that responds to the user's emotional state in real time.

[1360] Specific examples

[1361] For example, if the emotion engine analyzes that a user is in a state of high stress while working remotely, the server will generate relaxation music. Stress is also detected from the voice data, and the generated music is an instrumental piece with a high relaxing effect. This music is customized to the user's preferences. The device plays the music, allowing the user to instantly listen to relaxation music and reduce stress.

[1362] This invention effectively supports psychological wellness by providing personalized music based on a user's real-time emotional state, which contributes to stress reduction, improved quality of life, and increased productivity.

[1363] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1364] Step 1: Data collection

[1365] A user wears a wearable device (such as a smartwatch or fitness tracker). This device collects biometric data such as heart rate, blood pressure, and activity level in real time. The collected biometric data is sent to a device (e.g., a smartphone) via Bluetooth and temporarily stored in its internal storage. Voice data of the user's speech is also collected by the device via a microphone.

[1366] Input: User biometric and voice data

[1367] Output: Biometric data and voice data temporarily stored on the device

[1368] Step 2: Data transmission and encryption

[1369] The device encrypts the temporarily stored biometric and voice data using an encryption protocol (e.g., AES-256). The encrypted data is sent to the server via the HTTPS protocol. The server receives the encrypted data and performs a hash check to verify the integrity and completeness of the data.

[1370] Input: Biometric data and voice data temporarily stored on the device

[1371] Output: Data encrypted and sent to the server

[1372] Step 3: Sentiment analysis and emotion recognition

[1373] The server decrypts the encrypted data and begins analysis. Biometric data is analyzed using machine learning models (e.g., TensorFlow, PyTorch), and audio data is analyzed using natural language processing (NLP) algorithms. The emotion engine combines these analysis results to estimate the user's emotional state.

[1374] Input: Encrypted data sent to the server

[1375] Output: Estimated emotional state

[1376] Step 4: Music Generation

[1377] The server generates music using a generative AI model (e.g., GPT-4, DALL-E) based on the emotional state estimated by the emotion engine. This generation also takes into account the user's past music history, preferences, and cultural background. The generated music data is encrypted and sent to the device.

[1378] Input: estimated emotional state, past music history, user preferences, cultural background

[1379] Output: Generated music data

[1380] Step 5: Submit your music

[1381] The device decrypts the encrypted music data received from the server. The decrypted music data is passed to the device's media player and played back to the user. The user can then listen to the music via their smartphone or Bluetooth speaker.

[1382] Input: Encrypted music data sent to the device

[1383] Output: Music data passed to a media player and played to the user

[1384] Step 6: Long-term analysis and feedback

[1385] The server tracks daily emotional data and stores it in a database. By analyzing the accumulated data, trends and patterns are extracted and a report is created to support the user's long-term improvement of their psychological state. The report is sent to the device and can be viewed by the user within the application.

[1386] Input: Daily emotion data

[1387] Output: Report provided to the user

[1388] Through these steps, the system can provide personalized music according to the user's real-time emotional state and support psychological wellness.

[1389] (Application example 2)

[1390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1391] In modern brick-and-mortar stores, providing music that best suits the emotional state of customers in real time is an important factor in improving customer satisfaction. However, conventional systems have low accuracy in recognizing emotional states, making it difficult to provide appropriate music. Furthermore, security and privacy issues have made it difficult to safely handle biometric and voice data. There is a need to solve these issues and safely and effectively provide music that matches the emotional state of customers.

[1392] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring biometric data and voice data, means for encrypting the biometric data and voice data, means for decrypting and analyzing the encrypted data, means for generating music based on the emotional state, and means for encrypting and providing the generated music. This makes it possible to provide music that is optimal for the emotional state of customers in a physical store in real time while safely handling the biometric data and voice data.

[1393] "Biometric data" refers to data that measures a user's physiological indicators, such as heart rate, blood pressure, and activity level, in real time.

[1394] "Voice data" refers to voice information uttered by a user, and is data from which emotional tones and keywords can be extracted by analyzing the voice information.

[1395] "Encryption means" refers to a technology that encrypts acquired data using an encryption protocol (e.g., AES-256) to ensure security.

[1396] The "server" is a computer system that receives encrypted data, decrypts and analyzes it, estimates the user's emotional state, and generates music.

[1397] "Emotional state" refers to the user's psychological and emotional state, and is estimated from the analysis of heart rate, blood pressure, activity level, and voice data.

[1398] A "generative AI model" is an artificial intelligence algorithm that generates appropriate music based on emotional state.

[1399] A "user terminal" is a device used by a user (e.g., a smartphone or tablet) that receives and decodes data sent from the server and provides music.

[1400] A "physical store" is a commercial facility that operates in a physical form, and is a place that aims to analyze the emotions of customers in real time and provide appropriate music.

[1401] "Real-time" means that data is generated, processed, and results are provided with almost no delay.

[1402] This invention is a system for generating and providing optimal music based on the emotional state of customers in a physical store. The system includes a server, encryption means, a user terminal, biometric data acquisition means, voice data acquisition means, emotion estimation means, and a generative AI model.

[1403] The server first receives and decrypts the encrypted data sent from the means for acquiring biometric and voice data at the physical store. The AES-256 protocol is used for this encryption. The decrypted data is then analyzed by the emotion estimation means to estimate the user's emotional state. Natural language processing (NLP) algorithms and machine learning models are used for emotion estimation.

[1404] Based on the estimated emotional state, the generative AI model on the server generates optimal music in real time, taking into account the user's past music history, preferences, and even cultural background. The generated music data is then re-encrypted and sent to the user's device.

[1405] The user terminal decrypts the encrypted music data sent from the server and provides it to customers through the store's speaker system. This process is carried out in real time, and music that matches the customer's emotional state is provided, thereby improving customer satisfaction.

[1406] As a specific example, if a customer in a store is estimated to be feeling highly stressed, the generative AI model will generate music with a relaxing effect. This music is customized based on the customer's past music history and preferences. The user's device receives the music data sent from the server and plays it through speakers in the store. This helps the customer relax and reduce stress.

[1407] The system is realized by combining various hardware and software components, including a smartwatch, microphone device, and speaker system, and software components, including AES-256 encryption protocols, NLP algorithms, machine learning models, and generative AI models.

[1408] An example of a prompt is:

[1409] "Design an AI model that analyzes a user's emotional state based on their heart rate, blood pressure, activity level, and voice data. Then write a program to generate relaxing music based on that specific emotional state. Include any necessary encryption techniques."

[1410] Examples include:

[1411] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1412] Step 1:

[1413] The server collects biometric data (heart rate, blood pressure, activity level) and voice data from robots and smart devices in the physical store. The input is raw data from the sensor devices, which is received and stored in real time. The output is unencrypted biometric data and voice data.

[1414] Step 2:

[1415] The terminal encrypts the acquired biometric and voice data using the AES-256 protocol. The input is raw data, which is encrypted to ensure data security. The output is encrypted data.

[1416] Step 3:

[1417] The device sends encrypted biometric data and voice data to the server. The input is encrypted data that is securely transferred to the server over the network. The output is the data as it was sent to the server.

[1418] Step 4:

[1419] The server decrypts the received encrypted data using the AES-256 protocol. The input is the encrypted data, and the original raw data is obtained by decrypting it. The output is the decrypted biometric data and audio data.

[1420] Step 5:

[1421] The server analyzes the decoded biometric data and voice data to estimate the user's emotional state. Here, the voice data is analyzed using NLP algorithms to extract emotional tones and keywords. The biometric data is then used to estimate the emotional state using machine learning models. The input is the decoded data, and the output is the estimated emotional state.

[1422] Step 6:

[1423] The server generates music using a generative AI model based on the estimated emotional state. The input is emotional state data, and the generative AI model generates music data that takes into account the user's preferences and cultural background. The output is the generated music data.

[1424] Step 7:

[1425] The server then encrypts the generated music data again using the AES-256 protocol. The input is the music data, and the encryption process prepares it for secure transmission to the user's device. The output is the encrypted music data.

[1426] Step 8:

[1427] The device receives encrypted music data sent from the server and decrypts it using the AES-256 protocol. The input is the encrypted music data, and the original music data is obtained by decrypting it. The output is the decrypted music data.

[1428] Step 9:

[1429] The terminal plays the decoded music data through the speaker system in the physical store. The input is music data, and the output is music provided to customers. This makes it possible to provide music that corresponds to the emotional state of customers in real time.

[1430] This series of processes realizes a system that provides music that best suits the emotional state of customers in a physical store.

[1431] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1432] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1433] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1434] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1435] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1436] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1437] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1438] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1439] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1440] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1441] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1442] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1443] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1444] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1445] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1446] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1447] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1448] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1449] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1450] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1451] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1452] The following is further disclosed regarding the above embodiment.

[1453] (Claim 1)

[1454] a means for acquiring biometric data;

[1455] means for analyzing the acquired biometric data and estimating an emotional state;

[1456] means for generating music based on the estimated emotional state;

[1457] means for providing the generated music to a user;

[1458] A system including:

[1459] (Claim 2)

[1460] means for acquiring audio data;

[1461] means for analyzing the acquired voice data and estimating an emotional state;

[1462] means for generating music based on the estimated emotional state;

[1463] means for providing the generated music to a user;

[1464] 10. The system of claim 1, comprising:

[1465] (Claim 3)

[1466] means for customizing music based on the estimated emotional state, taking into account the user's preferences and cultural background;

[1467] a means for generating customized music;

[1468] 10. The system of claim 1, comprising:

[1469] "Example 1"

[1470] (Claim 1)

[1471] a means for acquiring biometric data;

[1472] means for analyzing the acquired biometric data and transmitting it to a server using an encryption protocol;

[1473] means for integrating the analyzed biometric data and audio data to estimate the user's emotional state;

[1474] means for generating music based on the estimated emotional state using a generative AI model;

[1475] means for transmitting the generated music to a terminal and providing it to a user;

[1476] a means of analyzing long-term user sentiment trends and patterns and providing feedback;

[1477] A system including:

[1478] (Claim 2)

[1479] means for acquiring voice data, encrypting the acquired voice data, and transmitting the encrypted data to a server;

[1480] means for estimating an emotional state from speech data;

[1481] means for generating music using a generative AI model based on the estimated emotional state;

[1482] means for transmitting and providing the generated music to a terminal;

[1483] 10. The system of claim 1, comprising:

[1484] (Claim 3)

[1485] means for generating personalized music taking into account the user's preferences and cultural background;

[1486] a means for encrypting the generated music and transmitting it to a terminal for provision;

[1487] 10. The system of claim 1, comprising:

[1488] "Application Example 1"

[1489] (Claim 1)

[1490] a means for acquiring biometric data;

[1491] means for analyzing the acquired biometric data and estimating an emotional state;

[1492] means for generating music based on the estimated emotional state;

[1493] means for providing the generated music to a user;

[1494] means for providing music during the ordering process and waiting time according to the user's emotional state;

[1495] A system including:

[1496] (Claim 2)

[1497] means for acquiring audio data;

[1498] means for analyzing the acquired voice data and estimating an emotional state;

[1499] means for generating music based on the estimated emotional state;

[1500] means for providing the generated music to a user;

[1501] 10. The system of claim 1, comprising:

[1502] (Claim 3)

[1503] means for customizing music based on the estimated emotional state, taking into account the user's preferences and cultural background;

[1504] a means for generating customized music;

[1505] 10. The system of claim 1, comprising:

[1506] "Example 2: Combining Emotion Engines"

[1507] (Claim 1)

[1508] A means for acquiring biometric data using a wearable device worn by a user;

[1509] means for temporarily storing the acquired biometric data in the terminal and transmitting it to a server using an encryption protocol;

[1510] means for encrypting data collected in the terminal and transmitting the data to a server using a communication protocol;

[1511] means for analyzing the encrypted data and inferring an emotional state using an emotion engine;

[1512] means for generating music using a generative AI model based on the estimated emotional state;

[1513] means for encrypting the generated music data and transmitting the data to a terminal;

[1514] means for playing the decrypted music on the terminal through a media player and providing it to the user;

[1515] A method for tracking and analyzing daily emotional data over the long term and providing feedback to users.

[1516] A system including:

[1517] (Claim 2)

[1518] means for acquiring the voice data, temporarily storing it on the terminal, and transmitting it to a server using an encryption protocol;

[1519] means for estimating an emotional state by analyzing the acquired voice data;

[1520] means for generating music based on the estimated emotional state;

[1521] means for providing the generated music to a user;

[1522] 10. The system of claim 1, comprising:

[1523] (Claim 3)

[1524] means for customizing music based on the estimated emotional state, taking into account the user's preferences and cultural background;

[1525] a means for generating customized music;

[1526] 10. The system of claim 1, comprising:

[1527] "Application example 2 when combining emotion engines"

[1528] (Claim 1)

[1529] a means for acquiring biometric data;

[1530] means for encrypting the acquired biometric data;

[1531] means for transmitting the encrypted biometric data to a server;

[1532] means for decoding and analyzing the biometric data at the server;

[1533] means for acquiring audio data;

[1534] means for encrypting the captured audio data;

[1535] means for transmitting the encrypted audio data to a server;

[1536] means for decoding and analyzing the audio data at the server;

[1537] means for estimating an emotional state by combining biometric data and voice data;

[1538] means for generating music based on the estimated emotional state;

[1539] means for encrypting the generated music and transmitting it to a user terminal;

[1540] means for decoding and providing music generated at a user terminal;

[1541] A system including:

[1542] (Claim 2)

[1543] means for customizing music based on the estimated emotional state, taking into account the user's preferences and cultural background;

[1544] a means for generating customized music;

[1545] 10. The system of claim 1, comprising:

[1546] (Claim 3)

[1547] A means for acquiring biometric data and voice data of customers in real time in a physical store and analyzing their emotional state;

[1548] A means for generating and providing background music appropriate to the emotions of customers based on the analyzed emotional state;

[1549] 10. The system of claim 1, comprising: [Explanation of symbols]

[1550] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring biometric data; means for analyzing the acquired biometric data and estimating an emotional state; means for generating music based on the estimated emotional state; means for providing the generated music to a user; A system including:

2. means for acquiring audio data; means for analyzing the acquired voice data and estimating an emotional state; means for generating music based on the estimated emotional state; means for providing the generated music to a user; The system of claim 1 , comprising:

3. means for customizing music based on the estimated emotional state, taking into account the user's preferences and cultural background; a means for generating customized music; The system of claim 1 , comprising:

Images