system

Abstract

Provide a system. A means for inputting patient information; means for learning patient preferences based on input patient information; A means of listing suitable music; a means for playing the listed music; a means for monitoring the patient's response while the music is being played; a means for providing feedback on the monitored response; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's aging society, the number of dementia patients is increasing. This is accompanied by serious issues such as communication difficulties due to memory loss and emotional fluctuations, and an increased sense of isolation. Standard music therapy requires specialized knowledge, making it difficult to select the most appropriate music for each patient. Caregivers also face a significant burden in selecting and managing playlists. Given these circumstances, effective methods are needed to improve the quality of life of dementia patients and reduce the burden on their families and caregivers. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for listing appropriate music, a means for playing the listed music, a means for monitoring the patient's reaction during music playback, and a means for providing feedback on the monitored reaction. This system uses AI to automatically generate music playlists based on the individual preferences of dementia patients, promoting relaxation, emotional stability, and communication. It also reduces the burden on caregivers and family members by making the system easy to manage.

[0006] "Patient information" is basic data about a dementia patient, including name, date of birth, and favorite music genre.

[0007] "Preference learning" is the process of using artificial intelligence algorithms to identify a patient's preferred music genres and artists based on information provided by the patient.

[0008] The "means of listing appropriate music" is a process in which an artificial intelligence algorithm analyzes the patient's preferences and selects recommended songs based on the results.

[0009] "Means for playing music" refers to music playback devices or software that allow patients to actually listen to the listed songs.

[0010] "Response monitoring measures" are the process of observing the patient's facial expressions and behavior while the music is playing and collecting their responses as data.

[0011] "Feedback means" refers to the process of transmitting monitored patient response data to a server, which provides information to improve the algorithm. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during the music playback, and a means for providing feedback.

[0034] User operations

[0035] 1. Install and launch the app

[0036] The user downloads, installs, and launches the app, then enters the required information on the login screen and logs in.

[0037] 2. Enter patient information

[0038] The user clicks the "Register New Patient" button and enters basic information such as the patient's name, date of birth, preferred music genre, etc. They then press the "Next" button to proceed to a screen where additional details can be entered.

[0039] 3. Add episodes

[0040] The user clicks the "Add Episode" button and inputs a song the patient liked, such as "Like the Flow of a River" by Hibari Misora, and an episode related to that song. The user then clicks the "Save" button to complete the input.

[0041] Server Processing

[0042] 1. Receiving and storing patient information

[0043] The server receives the patient information sent by the user and stores it in a database.

[0044] 2. Launching the AI ​​algorithm

[0045] When the server receives new patient information, it activates an AI algorithm that builds a profile based on the patient's preferences and anecdotes.

[0046] 3. Listing Music Candidates

[0047] The server calls a music streaming service API using keywords such as "Misora ​​Hibari" and "Showa pop songs," lists relevant songs, and generates playlist candidates.

[0048] 4. Accumulation of learning data

[0049] The server-generated playlists and associated patient information are stored in a database and used to improve the AI ​​model in the future.

[0050] Terminal handling

[0051] 1. Receiving and displaying playlists

[0052] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0053] 2. User selection of playlist

[0054] The user selects "Like the Flow of a River" and "Love's Vacation" from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[0055] 3. Music playback and monitoring

[0056] The device plays songs from the selected playlist, allowing the patient to listen to music. During playback, the device monitors the patient's facial expressions with a camera to detect smiling or relaxed expressions.

[0057] 4. Submitting Feedback

[0058] The monitoring results are fed back to the server, which then performs additional learning on the AI ​​model based on this feedback.

[0059] Specific examples

[0060] 1. Enter patient information

[0061] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0062] 2. Server Processing

[0063] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0064] 3. Generate a playlist

[0065] The device displays playlist suggestions to the user, who then selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device detects the patient's smile and sends the monitoring results back to the server, which uses this information to improve the AI ​​model.

[0066] This will enhance the effectiveness of music therapy for dementia patients and improve their relaxation and communication.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] Installation and Login

[0070] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0071] Step 2:

[0072] Entering patient information

[0073] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0074] Step 3:

[0075] Sending patient information

[0076] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[0077] Step 4:

[0078] Receiving and storing information

[0079] The server receives the patient information sent from the terminal and stores the received information in a database.

[0080] Step 5:

[0081] Launching AI algorithms

[0082] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0083] Step 6:

[0084] Listing music candidates

[0085] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[0086] Step 7:

[0087] Sending a Playlist

[0088] The server transmits the generated playlist candidates to the terminal.

[0089] Step 8:

[0090] Receiving and displaying playlists

[0091] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[0092] Step 9:

[0093] User-selected playlist

[0094] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0095] Step 10:

[0096] Playing music

[0097] The device will play the songs from the selected playlist, allowing the user to listen to music.

[0098] Step 11:

[0099] Reaction monitoring

[0100] While the device plays music, it monitors the patient's facial expressions and behavior with a camera and records the monitored information (e.g., smiling or relaxed facial expressions).

[0101] Step 12:

[0102] Send Feedback

[0103] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data and uses it to improve future playlist generation.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] Traditional music therapy has faced the challenge of selecting effective music based on the preferences of dementia patients. In particular, there has been a lack of systems that can analyze the preferences and reactions of dementia patients in real time and provide appropriate music based on that analysis. As a result, the effectiveness of music therapy in improving patients' quality of life has been limited.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for creating a list of appropriate music, a means for monitoring the patient's reaction while music is being played, a means for providing feedback on the monitored reaction, and a means for storing the created playlist and related data for use in future improvements to the AI ​​model. This makes it possible to provide appropriate music according to the individual preferences of each patient, maximizing the effects of music therapy and improving the quality of life of dementia patients.

[0109] "Patient information" is personal data about a dementia patient, including basic information such as name, date of birth, and preferred music genre.

[0110] "Means for learning preferences" refers to technology that uses artificial intelligence algorithms to analyze a patient's preferences and tendencies based on input patient information and build a profile.

[0111] "Means for listing appropriate music" refers to technology that uses the API of a music streaming service to select relevant songs based on the patient's preferences and generate playlist candidates.

[0112] "Means for playing" refers to a device or software for playing the listed music.

[0113] "Monitoring means" refers to technology that uses cameras and sensors to observe the patient's facial expressions and behavior while music is playing, and collects data in real time.

[0114] "Feedback means" refers to a technology that sends monitored patient reaction data to a server and uses it for additional learning of the artificial intelligence model.

[0115] "Means for storing the generated playlists and associated data" refers to technology that stores playlist suggestions and patient response data in a database and uses them to improve future AI models.

[0116] "Artificial intelligence algorithms" refer to technologies that use machine learning and deep learning to find patterns in large amounts of data and make predictions based on patient preferences and responses.

[0117] A "music streaming service API" is a program interface provided by an external music distribution service, and refers to a mechanism for obtaining song information based on specific keywords.

[0118] This invention is a music therapy support system for improving the quality of life of dementia patients. This system involves a series of processes: inputting patient information, learning preferences, listing appropriate music, playing music, monitoring the patient's response, and providing feedback.

[0119] System configuration

[0120] Hardware and Software

[0121] 1. Server

[0122] Hardware: High-performance server machine (e.g., Dell PowerEdge server)

[0123] software:

[0124] Database: MySQL (registered trademark)

[0125] AI algorithms: Python®-based machine learning libraries (e.g., TENSORFLOW®, PyTorch)

[0126] Music streaming service APIs: Spotify API and Apple Music API

[0127] 2. Terminal

[0128] Hardware: Smartphones and tablets (e.g., iPhone (registered trademark), iPad (registered trademark), Android (registered trademark) devices)

[0129] software:

[0130] Application: Music therapy support app

[0131] Monitoring technology: camera modules (e.g., built-in device cameras), facial recognition software

[0132] System Operation

[0133] 1. Enter patient information

[0134] Users install the app on their device, launch it, and after logging in, enter basic information such as the patient's name, date of birth, and favorite music genre on the "New Patient Registration" screen.

[0135] Examples:

[0136] Enter the patient's name "Yamada Taro," date of birth "July 15, 1940," and favorite music genre "Showa era pop music."

[0137] The episode "Misora ​​Hibari's 'Like the Flow of a River'" has been added.

[0138] 2. Learning preferences

[0139] The server receives the patient information sent by the user, stores it in a database, and then activates an AI algorithm to build a profile based on the patient's preferences and anecdotes.

[0140] Examples:

[0141] The server runs an AI algorithm based on new patient information to obtain song information related to "Showa pop songs" and "Misora ​​Hibari."

[0142] 3. Listing Music Candidates

[0143] The server calls the music streaming service API, lists relevant songs, and generates playlist suggestions.

[0144] Examples:

[0145] The server calls the Spotify API to compile a list of songs related to "Misora ​​Hibari" and "Showa era pop songs" and generates playlist candidates.

[0146] 4. Playing Music

[0147] The device displays playlist candidates to the user, and the user selects the desired songs. When the user presses the "Start Play" button, the music begins to play.

[0148] Examples:

[0149] The user selects "Like the Flow of a River" and "Love Vacation" and presses the "Start Play" button.

[0150] 5. Monitoring patient response

[0151] The device uses a camera to monitor the patient's facial expressions during playback and detects smiling and relaxed expressions.

[0152] Examples:

[0153] The device's camera captures a picture of the patient's face, and facial recognition software detects a smile.

[0154] 6. Feedback

[0155] The device feeds back the monitoring results to the server, which then uses this information to perform additional learning on the AI ​​model.

[0156] Examples:

[0157] The device sends the detected smile data to the server, which then uses this data to retrain the AI ​​model.

[0158] Operating Procedure

[0159] Specific examples of operation procedures

[0160] 1. Install and launch the app

[0161] The user downloads and installs the app on their device.

[0162] Once the installation is complete, tap the app icon to launch it. Enter the required information on the login screen to complete the login process.

[0163] Specific prompt examples:

[0164] Download and install the app, then enter your details on the login screen to log in.

[0165] 2. Enter patient information

[0166] The user clicks the "Register New Patient" button, enters the required information, then clicks the "Add Episode" button, enters the specific episode, and saves it.

[0167] Specific prompt examples:

[0168] When entering basic patient information, click the "Register New Patient" button and enter the required information.

[0169] This will enable music therapy for dementia patients to be managed efficiently and provide music tailored to the individual preferences of each patient, thereby enhancing the effectiveness of the therapy.

[0170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0171] Step 1: Install and launch the app

[0172] The user downloads and installs the app on their device.

[0173] Input: Download link

[0174] Output: Installed apps

[0175] Specific behavior:

[0176] The user searches for a "music therapy support app" in the app store, downloads and installs it, taps the app icon to launch the app, and enters their username and password on the login screen to complete the login process.

[0177] Step 2: Enter patient information

[0178] The user enters basic patient information on the "New Patient Registration" screen.

[0179] Input: Patient's name, date of birth, favorite music genre

[0180] Output: Entered patient information

[0181] Specific behavior:

[0182] The user clicks the "Register New Patient" button and enters his name "Yamada Taro," his date of birth "July 15, 1940," and his favorite music genre "Showa pop music."

[0183] Step 3: Add Episodes

[0184] The user enters and saves an episode related to the patient.

[0185] Input: Specific episode

[0186] Output: Added episode information

[0187] Specific behavior:

[0188] The user clicks the "Add Episode" button, enters the episode "Misora ​​Hibari's 'Like the Flow of a River'", and clicks the "Save" button to complete the process.

[0189] Step 4: Receiving and storing patient information (server)

[0190] The server stores the received patient information and episode information in a database.

[0191] Input: Patient and episode information

[0192] Output: Saved data

[0193] Specific behavior:

[0194] The server receives the patient information sent by the user and stores it in a MySQL database.

[0195] Step 5: Launching the AI ​​algorithm (server)

[0196] The server runs an artificial intelligence algorithm based on the patient information.

[0197] Input: Patient preference information

[0198] Output: Patient profile

[0199] Specific behavior:

[0200] The server runs a Python-based AI model that analyzes the patient's preferences and anecdotes to generate a profile.

[0201] Step 6: Listing music candidates (server)

[0202] The server uses the music streaming service API to retrieve related songs and generate playlist suggestions.

[0203] Input: Patient profile (music preferences and keywords)

[0204] Output: Music playlist suggestions

[0205] Specific behavior:

[0206] The server calls the Spotify API, retrieves related songs based on the keywords "Misora ​​Hibari" and "Showa pop songs," and generates playlist candidates.

[0207] Step 7: Receive and display the playlist (on your device)

[0208] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0209] Input: Playlist candidate data

[0210] Output: Playlist suggestions displayed

[0211] Specific behavior:

[0212] The device retrieves the playlist data from the server and displays it on the app screen.

[0213] Step 8: User selects playlist (User)

[0214] The user selects a song from the playlist candidates and starts playback.

[0215] Input: Selected song

[0216] Output: Finalized playlist

[0217] Specific behavior:

[0218] The user selects "Like the Flow of a River" and "Love Vacation" on the playlist screen and presses the "Start Play" button.

[0219] Step 9: Playing and Monitoring Music (Device)

[0220] The device plays the selected song and monitors the patient's response.

[0221] Input: Confirmed playlist

[0222] Output: Patient monitoring data

[0223] Specific behavior:

[0224] While the device plays music, it uses a camera module to monitor the patient's facial expressions and detect smiling or relaxed expressions.

[0225] Step 10: Send Feedback (Device)

[0226] The device sends the monitoring results to the server, which then uses them for further learning of the AI ​​model.

[0227] Input: Monitoring data

[0228] Output: Update data for the AI ​​model

[0229] Specific behavior:

[0230] The monitoring data detected by the device is sent to the server, which then uses this data to retrain the AI ​​model.

[0231] This enables the system to efficiently provide music tailored to the individual preferences of dementia patients, maximizing the effectiveness of music therapy.

[0232] (Application example 1)

[0233] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] It is known that music therapy is effective in helping elderly people and dementia patients maintain a better mental state in their daily lives. However, there is no system yet that can provide optimal music for each patient and evaluate its effectiveness in real time. Furthermore, when elderly people listen to music, it is necessary not only to play the music but also to verify whether the music actually brings relaxation and a sense of happiness. To address this issue, we aim to provide a system that selects music based on individual patient information and evaluates its effectiveness in real time and provides feedback.

[0235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0236] In this invention, the server

[0237] a means for inputting patient information;

[0238] means for learning patient preferences based on input patient information;

[0239] A means of listing suitable music;

[0240] a means for playing the listed music;

[0241] a means for monitoring the patient's response while the music is being played;

[0242] a means for providing feedback on the monitored response;

[0243] A means of analyzing facial expressions in real time; and

[0244] A means of providing feedback using generative AI models; and

[0245] This makes it possible to provide optimal music for individual patients, evaluate its effectiveness in real time, and provide feedback.

[0246] The "means for inputting patient information" is an interface for inputting basic data such as the patient's name, date of birth, and musical preferences.

[0247] The "means for learning patient preferences based on input patient information" is a system that uses an artificial intelligence algorithm to learn a patient's musical preferences based on patient information.

[0248] The "means for listing appropriate music" is a system that suggests music that may be of interest in the form of a list based on learned preferences.

[0249] The "means for playing the listed music" is an audio playback system that actually plays the selected music so that the patient can listen to it.

[0250] The "means for monitoring the patient's reactions while music is being played" is a system that uses cameras and sensors to observe and record the patient's facial expressions and behavior while music is being played.

[0251] The "means for feeding back monitored responses" refers to a feedback mechanism that sends the monitoring results to a server or artificial intelligence system to help select the next piece of music or improve treatment methods.

[0252] The "means for analyzing facial expressions in real time" is a system for analyzing facial expressions from a patient's face photographed in real time using a camera and evaluating the results.

[0253] "Means for providing feedback using a generative AI model" refers to a mechanism for providing optimal feedback based on data analyzed using a generative AI model, thereby improving the system.

[0254] System Program

[0255] The invention can be implemented using a smartphone application with the following functions: The application includes inputting and learning patient information, listing and playing appropriate music, monitoring responses, and providing feedback.

[0256] Program processing

[0257] The application begins with a user entering patient information (such as name, date of birth, and favorite music genre) on a smartphone. This information is then sent to a server and stored in a database. The server then uses an artificial intelligence algorithm to learn the patient's preferences and presents a list of appropriate music based on that information.

[0258] Hardware and Software

[0259] Hardware: Smartphone's built-in camera

[0260] Software: OpenCV (image processing library), Keras (deep learning library), emotion_model.h5 (emotion recognition model)

[0261] Server processing: AI algorithms, databases

[0262] The server uses the generative AI model to generate a list of music based on the newly entered patient information and sends it back to the application, which then displays the list of music suggestions to the user and plays the music selected by the user.

[0263] Specific processing examples

[0264] While music is playing, the smartphone camera monitors the patient's reactions. Using OpenCV for real-time facial analysis, the captured facial image data is fed into an emotion recognition model. The model identifies facial expressions such as smiling or relaxed, and based on that, evaluates whether the currently playing music is appropriate for the patient.

[0265] The monitoring results are fed back to the server and used by the generative AI model to select music for future use and improve the system. For example, if a patient shows many relaxed facial expressions while listening to a particular piece of music, that piece of music will likely be recommended to similar patients.

[0266] Prompt Sentence Examples

[0267] "Capture a photo, analyze the emotion in real time, and display the results."

[0268] "Use an AI model to determine in real time whether an elderly person is relaxed and provide feedback on the results."

[0269] This invention will enhance the effectiveness of music therapy and improve the quality of daily life for elderly people and dementia patients.

[0270] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0271] Step 1:

[0272] The user launches the smartphone application and enters patient information. Specifically, a form is displayed in which the patient can enter their name, date of birth, and favorite music genre. This information is collected as input data.

[0273] Step 2:

[0274] The patient information entered by the user is sent from the terminal to the server, where it is saved in a database. The input data is sent in JSON format and stored in the database. The data processing performed here involves converting the input data into the correct format.

[0275] Step 3:

[0276] Based on the newly saved patient information, the server runs an artificial intelligence algorithm to learn the patient's preferences. The algorithm analyzes the input data and creates a profile of the patient's musical preferences based on the learning database. The output is a list of music genres and artists that are suitable for the patient.

[0277] Step 4:

[0278] The server uses the generative AI model to generate a list of suitable music. Specifically, it calls the API of a music streaming service based on the patient's preference profile to retrieve candidate songs. The output is a list of multiple songs suitable for the patient.

[0279] Step 5:

[0280] The terminal receives the music list sent from the server and displays it to the user. The user selects a song to play from the displayed list. The input here is the music list, and the output is the song selected by the user.

[0281] Step 6:

[0282] The device plays the selected music. During playback, the patient's facial expression is monitored in real time using the smartphone camera. The input data is a facial image captured by the camera, and the OpenCV library is used to detect the facial region and preprocess the image.

[0283] Step 7:

[0284] The device inputs the preprocessed facial image data into the emotion recognition model (emotion_model.h5) and analyzes facial expressions in real time. The output is the detected emotion (e.g., smiling, relaxed, angry, etc.).

[0285] Step 8:

[0286] The monitoring results are fed back to the server, which uses this feedback data to analyze the patient's reactions using a generative AI model to improve future music recommendation algorithms. The input is emotional data, and the output is updates to the learning database.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during playback, a feedback means, and an emotion engine for recognizing the user's emotions.

[0289] User operations

[0290] 1. Install and launch the app

[0291] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0292] 2. Enter patient information

[0293] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0294] Server Processing

[0295] 1. Receiving and storing patient information

[0296] The server receives the patient information sent from the terminal and stores it in a database.

[0297] 2. Launching the AI ​​algorithm

[0298] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0299] 3. Listing Music Candidates

[0300] The server calls the music streaming service API based on the profile, lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop," and generates playlist candidates.

[0301] 4. Submit a playlist

[0302] The server transmits the generated playlist candidates to the terminal.

[0303] Terminal handling

[0304] 1. Receiving and displaying playlists

[0305] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0306] 2. User selection of playlist

[0307] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0308] 3. Playing music

[0309] The terminal plays the songs from the selected playlist, allowing the patient to listen to music.

[0310] 4. Monitoring and analysis using an emotion engine

[0311] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state. The emotion engine then classifies the monitored emotional data to identify emotional states such as relaxation, joy, and sadness.

[0312] 5. Automatic adjustment

[0313] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state, for example, if the patient is determined to be in a relaxed state, it will adjust the music played to continue playing songs that help maintain that state.

[0314] 6. Submitting Feedback

[0315] The device feeds back the monitoring results to the server, which receives this feedback data and updates the AI ​​model to improve the accuracy of future playlist generation.

[0316] Specific examples

[0317] 1. Enter patient information

[0318] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0319] 2. Server Processing

[0320] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0321] 3. Generate a playlist

[0322] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and detect whether they are relaxed.

[0323] 4. The Role of the Emotional Engine

[0324] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[0325] 5. Submitting Feedback

[0326] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[0327] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[0328] The processing flow will be explained below.

[0329] Step 1:

[0330] Installing the app and logging in

[0331] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0332] Step 2:

[0333] Entering patient information

[0334] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0335] Step 3:

[0336] Sending patient information

[0337] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[0338] Step 4:

[0339] Receiving and storing information

[0340] The server receives the patient information sent from the terminal and stores the received information in a database.

[0341] Step 5:

[0342] Launching AI algorithms

[0343] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0344] Step 6:

[0345] Listing music candidates

[0346] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[0347] Step 7:

[0348] Sending a Playlist

[0349] The server transmits the generated playlist candidates to the terminal.

[0350] Step 8:

[0351] Receiving and displaying playlists

[0352] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[0353] Step 9:

[0354] User-selected playlist

[0355] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0356] Step 10:

[0357] Playing music

[0358] The device will play the songs from the selected playlist, allowing the user to listen to music.

[0359] Step 11:

[0360] Emotion engine monitoring and analysis

[0361] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state, which is then classified as relaxation, joy, etc.

[0362] Step 12:

[0363] automatic adjustment

[0364] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state: for example, if the patient is determined to be relaxed, songs that help maintain that state will be played next.

[0365] Step 13:

[0366] Send Feedback

[0367] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data to improve the accuracy of future playlist generation.

[0368] Specific examples

[0369] 1. Enter patient information

[0370] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0371] 2. Server Processing

[0372] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0373] 3. Generate a playlist

[0374] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice to determine whether they are relaxed.

[0375] 4. The Role of the Emotional Engine

[0376] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[0377] 5. Submitting Feedback

[0378] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[0379] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[0380] Example 2

[0381] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0382] The challenge is to provide a system that maximizes the effectiveness of music therapy, which aims to improve the quality of life of dementia patients. In particular, because it is difficult to automatically select and adjust music according to the patient's emotional state, real-time analysis and automatic adjustment technology using an emotion engine are required to solve this problem. Another challenge is to build an efficient system using artificial intelligence algorithms to learn the patient's preferences.

[0383] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0384] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for listing appropriate music, a means for playing the listed music, a means for monitoring the patient's reaction during music playback, a means for providing feedback on the monitored reaction, a means for using an emotion engine to analyze the user's emotions in real time, and a means for automatically adjusting the music being played based on the analyzed emotional state. This makes it possible to maximize the effectiveness of music therapy for dementia patients and automatically adjust the music experience according to the patient's emotional state.

[0385] "Patient information" refers to basic data related to an individual patient, such as the patient's name, date of birth, and preferred music genre.

[0386] "Preference learning means" refers to the process or algorithm used to learn music that matches a patient's preferences based on information input by the patient.

[0387] "Music listing method" refers to the process of selecting appropriate music based on the patient's preferences and compiling it into a playlist.

[0388] "Means for playing music" refers to the ability to actually play songs from a selected music playlist and have the patient listen to them.

[0389] "Monitoring means" refers to the process and equipment used to monitor and collect data on the patient's responses while the music is playing.

[0390] "Feedback means" refers to the process of analyzing the patient's reaction data obtained through monitoring, feeding the results back into the system, and using them to determine the next music selection and playback method.

[0391] "Emotion engine" refers to software and algorithms that analyze a patient's facial expressions and voice in real time to identify their emotional state.

[0392] "Means for automatically adjusting" means a function for automatically changing the selection of music being played or the playback sequence based on the emotional state identified by the emotional engine.

[0393] This invention provides a music therapy support system aimed at improving the quality of life of dementia patients, and is implemented by combining the following means. Each means will be explained based on the role of the server, terminal, and user.

[0394] Server Roles

[0395] Receiving and storing patient information

[0396] The server receives the patient information sent from the device and stores it in a database, including the patient's name, date of birth, and preferred music genre.

[0397] Launching AI algorithms and building profiles

[0398] When the server receives new patient information, it invokes an AI algorithm using a Python machine learning library (e.g., scikit-learn) to build a profile for that patient based on their preferences and the anecdotes they provide.

[0399] Listing music candidates

[0400] Based on the constructed profile, the server calls the music streaming service API (e.g., Spotify API), lists related songs, and generates playlist suggestions.

[0401] Sending a Playlist

[0402] The generated playlist candidates are sent to the device in JSON format.

[0403] Device Role

[0404] Installing and launching the app

[0405] Users download and install a dedicated app on their smartphone or tablet, enter the required information on the login screen, and log in.

[0406] Entering patient information

[0407] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre, then enters more detailed information such as an episode related to a specific song, and taps the "Save" button to complete the process.

[0408] Receiving and displaying playlists

[0409] The device receives the playlist suggestions sent from the server and displays them to the user. The displayed information includes song titles, artist names, album art, and so on.

[0410] User selects and plays a playlist

[0411] The user selects a song to play from the list of suggestions and taps the "Start Play" button to confirm the playlist. Playback begins and the patient begins listening to the music.

[0412] Monitoring and analysis of patient responses

[0413] Real-time analysis by emotion engine

[0414] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to identify their emotional state. The emotion engine collects data on facial features and tone of voice to classify emotional states such as relaxation, joy, and sadness.

[0415] Auto-tuning and feedback

[0416] automatic adjustment

[0417] The emotion engine automatically adjusts the music being played based on the analysis results. For example, if it determines that the patient is relaxed, it will adjust the music to continue playing to help maintain that state.

[0418] Send Feedback

[0419] The device feeds back the monitoring results to the server in real time, and the server stores the feedback in a database and uses it as training data to help generate the next playlist.

[0420] Specific examples

[0421] Prompts for entering patient information and generating a playlist

[0422] The user starts the app and enters their name ("Yamada Taro"), their date of birth ("July 15, 1940"), their favorite music ("Showa pop"), and an episode related to a specific song by a certain famous artist.

[0423] Based on this information, the server creates a list of songs related to "Showa pop music" and "certain famous artists" and generates a playlist of candidates. The user selects a "specific song name" and "another song name" from the playlist sent to the device and begins playback.

[0424] Prompt Sentence Examples

[0425] "Please generate a relaxing playlist for a male dementia patient over 70 years old who likes Showa-era pop music. For example, please include a specific song by a certain famous artist or a hit song from the Showa era."

[0426] This system can maximize the effectiveness of music therapy for dementia patients and provide a music experience that suits their emotional state. Real-time analysis and automatic adjustment functions based on an emotion engine are expected to further enhance the effect.

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Step 1:

[0429] Installing and launching the app

[0430] Users download and install a dedicated app on their smartphone or tablet, then launch the app and log in by entering their username and password on the login screen.

[0431] Input: None

[0432] Output: None

[0433] Specific operation: A user downloads the app from the Google Play Store or Apple App Store, launches it after installation, and then enters the required information on the login screen to log in.

[0434] Step 2:

[0435] Entering patient information

[0436] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre. They then press the "Next" button to enter more detailed information, such as an episode related to a specific song, and tap the "Save" button to complete the process.

[0437] Input: Patient information (name, date of birth, music preferences, episodes)

[0438] Output: Patient information is saved on the device

[0439] Specific operation: The user enters the required information into the form, presses the button to proceed to the next screen, and when the series of information input is completed, presses the save button to save the information in the terminal.

[0440] Step 3:

[0441] Transmitting and storing patient information

[0442] The terminal transmits the stored patient information to a server, which receives the information and stores it in a database.

[0443] Input: Patient information stored on the device

[0444] Output: Patient information stored in the server database

[0445] Specific operation: The terminal sends patient information to the server via an HTTP request, and the server receives the information and stores it in the database.

[0446] Step 4:

[0447] Launching AI algorithms and building profiles

[0448] When the server receives new patient information, it activates an AI algorithm that builds a patient profile based on the patient's preferences and anecdotes.

[0449] Input: Patient information stored in the server database

[0450] Output: Patient preference profile

[0451] Specific operation: The server uses a Python machine learning library (e.g., scikit-learn) to generate a patient preference profile.

[0452] Step 5:

[0453] Listing music candidates

[0454] The server's AI model calls the music streaming service API based on the constructed profile, lists appropriate songs, and generates playlist suggestions.

[0455] Input: Patient Preference Profile

[0456] Output: Playlist suggestions

[0457] Specific operation: The server calls the Spotify API, searches for songs using keywords such as "Showa pop music," and compiles the generated song list in JSON format as playlist candidates.

[0458] Step 6:

[0459] Sending a Playlist

[0460] The server transmits the generated playlist candidates to the terminal.

[0461] Input: Playlist suggestions

[0462] Output: Playlist suggestions sent to device

[0463] Specific operation: The server sends playlist candidates to the terminal as an HTTP response.

[0464] Step 7:

[0465] Receiving and displaying playlists

[0466] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0467] Input: Playlist suggestions from the server

[0468] Output: Playlist suggestions displayed to the user

[0469] Specific operation: The device displays the received playlist candidates to the user on a GUI, where the user can check information such as song title, artist name, and album art.

[0470] Step 8:

[0471] User selects and plays a playlist

[0472] The user selects songs to play from the displayed playlist candidates and taps the "Start Playing" button to confirm the playlist.

[0473] Input: Song selected by user

[0474] Output: Song that started playing

[0475] Specific behavior: The user selects a song from the list and presses the play button to play the music. The music will be played through the device's speakers or headphones.

[0476] Step 9:

[0477] Emotion engine monitoring and analysis

[0478] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state.

[0479] Input: facial expression data and voice data of the patient being played back

[0480] Output: Patient's emotional state (relaxed, happy, sad, etc.)

[0481] Specific operation: The device uses the built-in camera and microphone to collect facial expression and voice data, which the emotion engine analyzes to identify the user's emotional state.

[0482] Step 10:

[0483] automatic adjustment

[0484] The emotion engine automatically adjusts songs in the currently playing playlist based on the determined emotional state.

[0485] Input: Emotional state determined by the emotion engine

[0486] Output: Auto-tuned music playback sequence

[0487] Specific operation: The emotion engine adjusts the selection of the next song and the playback order, for example by changing the currently playing song to one with a relaxing effect.

[0488] Step 11:

[0489] Send Feedback

[0490] The device sends the monitoring results as feedback to the server, which receives them and stores them in a database to help improve the AI ​​model.

[0491] Input: Monitoring results

[0492] Output: The AI ​​model on the server is updated.

[0493] Specific operation: The server receives the monitoring results sent from the device and stores them in a database, which improves the accuracy of the next playlist generation.

[0494] (Application example 2)

[0495] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0496] Current music therapy systems have difficulty selecting appropriate music based on the individual preferences of dementia patients. Furthermore, there is no way to monitor the impact of the music being played on the patient's emotional state in real time and automatically adjust the music selection based on that information. This makes it difficult to maximize the effectiveness of music therapy.

[0497] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0498] In this invention, the server includes means for inputting patient information, means for learning the patient's preferences based on the input patient information, means for listing appropriate music, means for playing the listed music, means for monitoring the patient's reaction during music playback, means for feeding back the monitored reaction, means for automatically adjusting the music based on the reaction data, and means for analyzing emotions in real time using a generative AI model, thereby making it possible to select music that matches the patient's preferences and automatically adjust the music in real time according to the patient's emotional state.

[0499] "Means for inputting patient information" refers to an interface that allows a user to input information such as the patient's name, date of birth, and preferred music genre.

[0500] "Means for learning patient preferences" refers to an artificial intelligence algorithm that learns the patient's preferred music and episodes based on input patient information.

[0501] "Means for listing appropriate music" refers to a mechanism for listing music suitable for a patient based on learned preferences and generating a playlist.

[0502] "Means for playing the listed music" refers to a function for playing the selected music using an audio device.

[0503] "Means for monitoring patient reactions while music is being played" refers to a system for monitoring the patient's facial expressions, voice, heart rate, etc. in real time while music is being played and analyzing their emotional state.

[0504] "Means for providing feedback on monitored responses" refers to a feedback function that records the monitored emotional state of the patient in a database and uses this information to improve the overall operation of the system and music selection.

[0505] "Means for automatically adjusting music based on reaction data" refers to a mechanism for changing and adjusting currently playing music in real time based on monitored emotional data.

[0506] "Generative AI model" refers to a trained artificial intelligence model used to make music recommendations and reaction analysis based on input data.

[0507] "Real-time emotion analysis means" refers to a system that analyzes a patient's facial expressions, voice, and heart rate in real time to identify their current emotional state.

[0508] This invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reactions during playback, a means for providing feedback, and a generative AI model for analyzing emotions in real time.

[0509] Server Processing

[0510] The server receives patient information and stores it in a database. When new patient information is received, the server activates an artificial intelligence (AI) algorithm to build a profile based on the patient's preferences and input episodes. Based on the profiled data, the server calls a music streaming service API to list appropriate songs. Finally, the generated playlist suggestions are sent to the user's device.

[0511] Terminal handling

[0512] The device displays the playlist candidates received from the server to the user. The user selects songs to play from the displayed playlist candidates. While the music is playing, the device uses a generative AI model to analyze the patient's facial expressions, voice, heart rate, etc. in real time to determine their emotional state. Based on the emotional state, the device automatically adjusts the current playlist to play songs that will help the patient maintain a relaxed state. The monitored emotional data is fed back to the server and used to improve the accuracy of the next playlist generation.

[0513] Hardware and Software Used

[0514] Hardware: A webcam and heart rate sensor are used to collect patient biometric information, as well as speakers and a headset for playing music.

[0515] Software: Python, OpenCV, TensorFlow, and DeepFace libraries are used. Data processing and AI operations are performed using Python programs, and face recognition and emotion analysis are performed using OpenCV and DeepFace libraries.

[0516] Specific examples

[0517] The server receives the patient's name ("Yamada Taro"), date of birth (July 15, 1940), and favorite music genre ("Showa pop"), and generates playlist candidates based on this. The device displays the playlist candidates, and songs like "Like the Flow of a River" and "Love Vacation" are selected. During playback, the generative AI model analyzes the patient's facial expressions and voice to detect whether they are relaxed. If it is determined that they are relaxed, it automatically changes to music with a similar effect.

[0518] Prompt Sentence Examples

[0519] Taro Yamada's favorite music genre is "Showa pop," and there is an episode related to a specific song, "Like the Flow of a River." Based on this, create an AI model that can recommend appropriate music. Also, include a system that monitors the subject's emotional state in real time and automatically adjusts the music list.

[0520] As described above, the present invention improves the quality of life of dementia patients by recommending music based on the patient's individual preferences and analyzing and adjusting their emotional state in real time.

[0521] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0522] Step 1:

[0523] The user installs the app on their smartphone or device and launches it. The user enters the required information (e.g., username and password) on the login screen and logs in.

[0524] Input: User login information

[0525] Output: Login successful message, main menu displayed

[0526] Specific operation: When the user enters login information and presses the send button, the device sends an authentication request to the authentication server, and if authentication is successful, the main menu is displayed.

[0527] Step 2:

[0528] The user enters new patient information by clicking the "Register New Patient" button, entering basic information such as the patient's name, date of birth, and favorite music genre, and then clicking the "Next" button to proceed to the detailed information entry screen.

[0529] Input: Patient's name, date of birth, favorite music genre, detailed episode

[0530] Output: Patient information data

[0531] Specific operation: When the user enters patient information and presses the save button, the terminal sends and saves this information to the database.

[0532] Step 3:

[0533] The server receives the patient information sent from the device and stores it in a database. The reception of new patient information triggers the AI ​​algorithm.

[0534] Input: Patient information data

[0535] Output: AI algorithm execution flag, patient profile

[0536] How it works: When the server receives patient information, it writes it into a database and then runs an AI algorithm to generate a profile based on the patient's preferences.

[0537] Step 4:

[0538] Based on the generated patient profile, the server calls a music streaming service API to list relevant songs.

[0539] Input: Patient Profile

[0540] Output: Playlist suggestions

[0541] How it works: The AI ​​algorithm analyzes profile data, uses music streaming service APIs to collect related songs, and generates a list of playlist suggestions.

[0542] Step 5:

[0543] The server transmits the generated playlist candidates to the terminal, which receives them and displays them to the user.

[0544] Input: Playlist suggestions

[0545] Output: Playlist suggestions displayed on the device

[0546] Specific operation: The server transmits the playlist data to the terminal, and the terminal displays the received data on the user interface.

[0547] Step 6:

[0548] The user selects songs to play from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[0549] Input: User song selection

[0550] Output: Playback start signal, selected playlist

[0551] Specific operation: When the user selects a song and presses the start playback button, the data of the selected playlist is sent to the playback engine.

[0552] Step 7:

[0553] The device plays songs from the selected playlist, while the patient's facial expressions, voice, and heart rate are analyzed in real time using the HMD, camera, and heart rate sensor.

[0554] Input: Selected playlist, patient vitals

[0555] Output: Playing music, real-time analysis data

[0556] Specific operation: The device starts the playback engine and begins playing the music, and at the same time, it acquires data using a biometric information collection device and has it analyzed by the generative AI model.

[0557] Step 8:

[0558] Automatically adjusts songs in the current playlist based on real-time analyzed emotional data.

[0559] Input: Parsed emotion data

[0560] Output: Adjusted song list, next song to play

[0561] How it works: The generative AI model evaluates the currently playing song based on emotional data and automatically changes or adjusts the next song if necessary.

[0562] Step 9:

[0563] The monitoring results are fed back to the server to improve the accuracy of future playlist generation.

[0564] Input: Real-time analysis data and adjustment data

[0565] Output: Updated AI model, feedback data saved

[0566] Specific operation: The device sends emotion analysis data and adjustment data to the server, which stores it in a database and uses it to update the AI ​​model.

[0567] Through these steps, this music therapy support system provides music based on the individual preferences of dementia patients and analyzes and adjusts their emotions in real time, thereby improving their quality of life.

[0568] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0569] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0570] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0571] [Second embodiment]

[0572] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0573] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0574] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0575] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0576] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0577] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0578] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0579] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0580] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0581] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0582] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0583] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0584] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during the music playback, and a means for providing feedback.

[0585] User operations

[0586] 1. Install and launch the app

[0587] The user downloads, installs, and launches the app, then enters the required information on the login screen and logs in.

[0588] 2. Enter patient information

[0589] The user clicks the "Register New Patient" button and enters basic information such as the patient's name, date of birth, preferred music genre, etc. They then press the "Next" button to proceed to a screen where additional details can be entered.

[0590] 3. Add episodes

[0591] The user clicks the "Add Episode" button and inputs a song the patient liked, such as "Like the Flow of a River" by Hibari Misora, and an episode related to that song. The user then clicks the "Save" button to complete the input.

[0592] Server Processing

[0593] 1. Receiving and storing patient information

[0594] The server receives the patient information sent by the user and stores it in a database.

[0595] 2. Launching the AI ​​algorithm

[0596] When the server receives new patient information, it activates an AI algorithm that builds a profile based on the patient's preferences and anecdotes.

[0597] 3. Listing Music Candidates

[0598] The server calls a music streaming service API using keywords such as "Misora ​​Hibari" and "Showa pop songs," lists relevant songs, and generates playlist candidates.

[0599] 4. Accumulation of learning data

[0600] The server-generated playlists and associated patient information are stored in a database and used to improve the AI ​​model in the future.

[0601] Terminal handling

[0602] 1. Receiving and displaying playlists

[0603] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0604] 2. User selection of playlist

[0605] The user selects "Like the Flow of a River" and "Love's Vacation" from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[0606] 3. Music playback and monitoring

[0607] The device plays songs from the selected playlist, allowing the patient to listen to music. During playback, the device monitors the patient's facial expressions with a camera to detect smiling or relaxed expressions.

[0608] 4. Submitting Feedback

[0609] The monitoring results are fed back to the server, which then performs additional learning on the AI ​​model based on this feedback.

[0610] Specific examples

[0611] 1. Enter patient information

[0612] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0613] 2. Server Processing

[0614] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0615] 3. Generate a playlist

[0616] The device displays playlist suggestions to the user, who then selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device detects the patient's smile and sends the monitoring results back to the server, which uses this information to improve the AI ​​model.

[0617] This will enhance the effectiveness of music therapy for dementia patients and improve their relaxation and communication.

[0618] The processing flow will be explained below.

[0619] Step 1:

[0620] Installation and Login

[0621] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0622] Step 2:

[0623] Entering patient information

[0624] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0625] Step 3:

[0626] Sending patient information

[0627] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[0628] Step 4:

[0629] Receiving and storing information

[0630] The server receives the patient information sent from the terminal and stores the received information in a database.

[0631] Step 5:

[0632] Launching AI algorithms

[0633] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0634] Step 6:

[0635] Listing music candidates

[0636] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[0637] Step 7:

[0638] Sending a Playlist

[0639] The server transmits the generated playlist candidates to the terminal.

[0640] Step 8:

[0641] Receiving and displaying playlists

[0642] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[0643] Step 9:

[0644] User-selected playlist

[0645] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0646] Step 10:

[0647] Playing music

[0648] The device will play the songs from the selected playlist, allowing the user to listen to music.

[0649] Step 11:

[0650] Reaction monitoring

[0651] While the device plays music, it monitors the patient's facial expressions and behavior with a camera and records the monitored information (e.g., smiling or relaxed facial expressions).

[0652] Step 12:

[0653] Send Feedback

[0654] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data and uses it to improve future playlist generation.

[0655] Example 1

[0656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Traditional music therapy has faced the challenge of selecting effective music based on the preferences of dementia patients. In particular, there has been a lack of systems that can analyze the preferences and reactions of dementia patients in real time and provide appropriate music based on that analysis. As a result, the effectiveness of music therapy in improving patients' quality of life has been limited.

[0658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0659] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for creating a list of appropriate music, a means for monitoring the patient's reaction while music is being played, a means for providing feedback on the monitored reaction, and a means for storing the created playlist and related data for use in future improvements to the AI ​​model. This makes it possible to provide appropriate music according to the individual preferences of each patient, maximizing the effects of music therapy and improving the quality of life of dementia patients.

[0660] "Patient information" is personal data about a dementia patient, including basic information such as name, date of birth, and preferred music genre.

[0661] "Means for learning preferences" refers to technology that uses artificial intelligence algorithms to analyze a patient's preferences and tendencies based on input patient information and build a profile.

[0662] "Means for listing appropriate music" refers to technology that uses the API of a music streaming service to select relevant songs based on the patient's preferences and generate playlist candidates.

[0663] "Means for playing" refers to a device or software for playing the listed music.

[0664] "Monitoring means" refers to technology that uses cameras and sensors to observe the patient's facial expressions and behavior while music is playing, and collects data in real time.

[0665] "Feedback means" refers to a technology that sends monitored patient reaction data to a server and uses it for additional learning of the artificial intelligence model.

[0666] "Means for storing the generated playlists and associated data" refers to technology that stores playlist suggestions and patient response data in a database and uses them to improve future AI models.

[0667] "Artificial intelligence algorithms" refer to technologies that use machine learning and deep learning to find patterns in large amounts of data and make predictions based on patient preferences and responses.

[0668] A "music streaming service API" is a program interface provided by an external music distribution service, and refers to a mechanism for obtaining song information based on specific keywords.

[0669] This invention is a music therapy support system for improving the quality of life of dementia patients. This system involves a series of processes: inputting patient information, learning preferences, listing appropriate music, playing music, monitoring the patient's response, and providing feedback.

[0670] System configuration

[0671] Hardware and Software

[0672] 1. Server

[0673] Hardware: High-performance server machine (e.g., Dell PowerEdge server)

[0674] software:

[0675] Database: MySQL

[0676] AI algorithms: Python-based machine learning libraries (e.g., TensorFlow, PyTorch)

[0677] Music streaming service APIs: Spotify API and Apple Music API

[0678] 2. Terminal

[0679] Hardware: Smartphones and tablets (e.g. iPhone, iPad, Android devices)

[0680] software:

[0681] Application: Music therapy support app

[0682] Monitoring technology: camera modules (e.g., built-in device cameras), facial recognition software

[0683] System Operation

[0684] 1. Enter patient information

[0685] Users install the app on their device, launch it, and after logging in, enter basic information such as the patient's name, date of birth, and favorite music genre on the "New Patient Registration" screen.

[0686] Examples:

[0687] Enter the patient's name "Yamada Taro," date of birth "July 15, 1940," and favorite music genre "Showa era pop music."

[0688] The episode "Misora ​​Hibari's 'Like the Flow of a River'" has been added.

[0689] 2. Learning preferences

[0690] The server receives the patient information sent by the user, stores it in a database, and then activates an AI algorithm to build a profile based on the patient's preferences and anecdotes.

[0691] Examples:

[0692] The server runs an AI algorithm based on new patient information to obtain song information related to "Showa pop songs" and "Misora ​​Hibari."

[0693] 3. Listing Music Candidates

[0694] The server calls the music streaming service API, lists relevant songs, and generates playlist suggestions.

[0695] Examples:

[0696] The server calls the Spotify API to compile a list of songs related to "Misora ​​Hibari" and "Showa era pop songs" and generates playlist candidates.

[0697] 4. Playing Music

[0698] The device displays playlist candidates to the user, and the user selects the desired songs. When the user presses the "Start Play" button, the music begins to play.

[0699] Examples:

[0700] The user selects "Like the Flow of a River" and "Love Vacation" and presses the "Start Play" button.

[0701] 5. Monitoring patient response

[0702] The device uses a camera to monitor the patient's facial expressions during playback and detects smiling and relaxed expressions.

[0703] Examples:

[0704] The device's camera captures a picture of the patient's face, and facial recognition software detects a smile.

[0705] 6. Feedback

[0706] The device feeds back the monitoring results to the server, which then uses this information to perform additional learning on the AI ​​model.

[0707] Examples:

[0708] The device sends the detected smile data to the server, which then uses this data to retrain the AI ​​model.

[0709] Operating Procedure

[0710] Specific examples of operation procedures

[0711] 1. Install and launch the app

[0712] The user downloads and installs the app on their device.

[0713] Once the installation is complete, tap the app icon to launch it. Enter the required information on the login screen to complete the login process.

[0714] Specific prompt examples:

[0715] Download and install the app, then enter your details on the login screen to log in.

[0716] 2. Enter patient information

[0717] The user clicks the "Register New Patient" button, enters the required information, then clicks the "Add Episode" button, enters the specific episode, and saves it.

[0718] Specific prompt examples:

[0719] When entering basic patient information, click the "Register New Patient" button and enter the required information.

[0720] This will enable music therapy for dementia patients to be managed efficiently and provide music tailored to the individual preferences of each patient, thereby enhancing the effectiveness of the therapy.

[0721] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0722] Step 1: Install and launch the app

[0723] The user downloads and installs the app on their device.

[0724] Input: Download link

[0725] Output: Installed apps

[0726] Specific behavior:

[0727] The user searches for a "music therapy support app" in the app store, downloads and installs it, taps the app icon to launch the app, and enters their username and password on the login screen to complete the login process.

[0728] Step 2: Enter patient information

[0729] The user enters basic patient information on the "New Patient Registration" screen.

[0730] Input: Patient's name, date of birth, favorite music genre

[0731] Output: Entered patient information

[0732] Specific behavior:

[0733] The user clicks the "Register New Patient" button and enters his name "Yamada Taro," his date of birth "July 15, 1940," and his favorite music genre "Showa pop music."

[0734] Step 3: Add Episodes

[0735] The user enters and saves an episode related to the patient.

[0736] Input: Specific episode

[0737] Output: Added episode information

[0738] Specific behavior:

[0739] The user clicks the "Add Episode" button, enters the episode "Misora ​​Hibari's 'Like the Flow of a River'", and clicks the "Save" button to complete the process.

[0740] Step 4: Receiving and storing patient information (server)

[0741] The server stores the received patient information and episode information in a database.

[0742] Input: Patient and episode information

[0743] Output: Saved data

[0744] Specific behavior:

[0745] The server receives the patient information sent by the user and stores it in a MySQL database.

[0746] Step 5: Launching the AI ​​algorithm (server)

[0747] The server runs an artificial intelligence algorithm based on the patient information.

[0748] Input: Patient preference information

[0749] Output: Patient profile

[0750] Specific behavior:

[0751] The server runs a Python-based AI model that analyzes the patient's preferences and anecdotes to generate a profile.

[0752] Step 6: Listing music candidates (server)

[0753] The server uses the music streaming service API to retrieve related songs and generate playlist suggestions.

[0754] Input: Patient profile (music preferences and keywords)

[0755] Output: Music playlist suggestions

[0756] Specific behavior:

[0757] The server calls the Spotify API, retrieves related songs based on the keywords "Misora ​​Hibari" and "Showa pop songs," and generates playlist candidates.

[0758] Step 7: Receive and display the playlist (on your device)

[0759] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0760] Input: Playlist candidate data

[0761] Output: Playlist suggestions displayed

[0762] Specific behavior:

[0763] The device retrieves the playlist data from the server and displays it on the app screen.

[0764] Step 8: User selects playlist (User)

[0765] The user selects a song from the playlist candidates and starts playback.

[0766] Input: Selected song

[0767] Output: Finalized playlist

[0768] Specific behavior:

[0769] The user selects "Like the Flow of a River" and "Love Vacation" on the playlist screen and presses the "Start Play" button.

[0770] Step 9: Playing and Monitoring Music (Device)

[0771] The device plays the selected song and monitors the patient's response.

[0772] Input: Confirmed playlist

[0773] Output: Patient monitoring data

[0774] Specific behavior:

[0775] While the device plays music, it uses a camera module to monitor the patient's facial expressions and detect smiling or relaxed expressions.

[0776] Step 10: Send Feedback (Device)

[0777] The device sends the monitoring results to the server, which then uses them for further learning of the AI ​​model.

[0778] Input: Monitoring data

[0779] Output: Update data for the AI ​​model

[0780] Specific behavior:

[0781] The monitoring data detected by the device is sent to the server, which then uses this data to retrain the AI ​​model.

[0782] This enables the system to efficiently provide music tailored to the individual preferences of dementia patients, maximizing the effectiveness of music therapy.

[0783] (Application example 1)

[0784] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0785] It is known that music therapy is effective in helping elderly people and dementia patients maintain a better mental state in their daily lives. However, there is no system yet that can provide optimal music for each patient and evaluate its effectiveness in real time. Furthermore, when elderly people listen to music, it is necessary not only to play the music but also to verify whether the music actually brings relaxation and a sense of happiness. To address this issue, we aim to provide a system that selects music based on individual patient information and evaluates its effectiveness in real time and provides feedback.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0787] In this invention, the server

[0788] a means for inputting patient information;

[0789] means for learning patient preferences based on input patient information;

[0790] A means of listing suitable music;

[0791] a means for playing the listed music;

[0792] a means for monitoring the patient's response while the music is being played;

[0793] a means for providing feedback on the monitored response;

[0794] A means of analyzing facial expressions in real time; and

[0795] A means of providing feedback using generative AI models; and

[0796] This makes it possible to provide optimal music for individual patients, evaluate its effectiveness in real time, and provide feedback.

[0797] The "means for inputting patient information" is an interface for inputting basic data such as the patient's name, date of birth, and musical preferences.

[0798] The "means for learning patient preferences based on input patient information" is a system that uses an artificial intelligence algorithm to learn a patient's musical preferences based on patient information.

[0799] The "means for listing appropriate music" is a system that suggests music that may be of interest in the form of a list based on learned preferences.

[0800] The "means for playing the listed music" is an audio playback system that actually plays the selected music so that the patient can listen to it.

[0801] The "means for monitoring the patient's reactions while music is being played" is a system that uses cameras and sensors to observe and record the patient's facial expressions and behavior while music is being played.

[0802] The "means for feeding back monitored responses" refers to a feedback mechanism that sends the monitoring results to a server or artificial intelligence system to help select the next piece of music or improve treatment methods.

[0803] The "means for analyzing facial expressions in real time" is a system for analyzing facial expressions from a patient's face photographed in real time using a camera and evaluating the results.

[0804] "Means for providing feedback using a generative AI model" refers to a mechanism for providing optimal feedback based on data analyzed using a generative AI model, thereby improving the system.

[0805] System Program

[0806] The invention can be implemented using a smartphone application with the following functions: The application includes inputting and learning patient information, listing and playing appropriate music, monitoring responses, and providing feedback.

[0807] Program processing

[0808] The application begins with a user entering patient information (such as name, date of birth, and favorite music genre) on a smartphone. This information is then sent to a server and stored in a database. The server then uses an artificial intelligence algorithm to learn the patient's preferences and presents a list of appropriate music based on that information.

[0809] Hardware and Software

[0810] Hardware: Smartphone's built-in camera

[0811] Software: OpenCV (image processing library), Keras (deep learning library), emotion_model.h5 (emotion recognition model)

[0812] Server processing: AI algorithms, databases

[0813] The server uses the generative AI model to generate a list of music based on the newly entered patient information and sends it back to the application, which then displays the list of music suggestions to the user and plays the music selected by the user.

[0814] Specific processing examples

[0815] While music is playing, the smartphone camera monitors the patient's reactions. Using OpenCV for real-time facial analysis, the captured facial image data is fed into an emotion recognition model. The model identifies facial expressions such as smiling or relaxed, and based on that, evaluates whether the currently playing music is appropriate for the patient.

[0816] The monitoring results are fed back to the server and used by the generative AI model to select music for future use and improve the system. For example, if a patient shows many relaxed facial expressions while listening to a particular piece of music, that piece of music will likely be recommended to similar patients.

[0817] Prompt Sentence Examples

[0818] "Capture a photo, analyze the emotion in real time, and display the results."

[0819] "Use an AI model to determine in real time whether an elderly person is relaxed and provide feedback on the results."

[0820] This invention will enhance the effectiveness of music therapy and improve the quality of daily life for elderly people and dementia patients.

[0821] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0822] Step 1:

[0823] The user launches the smartphone application and enters patient information. Specifically, a form is displayed in which the patient can enter their name, date of birth, and favorite music genre. This information is collected as input data.

[0824] Step 2:

[0825] The patient information entered by the user is sent from the terminal to the server, where it is saved in a database. The input data is sent in JSON format and stored in the database. The data processing performed here involves converting the input data into the correct format.

[0826] Step 3:

[0827] Based on the newly saved patient information, the server runs an artificial intelligence algorithm to learn the patient's preferences. The algorithm analyzes the input data and creates a profile of the patient's musical preferences based on the learning database. The output is a list of music genres and artists that are suitable for the patient.

[0828] Step 4:

[0829] The server uses the generative AI model to generate a list of suitable music. Specifically, it calls the API of a music streaming service based on the patient's preference profile to retrieve candidate songs. The output is a list of multiple songs suitable for the patient.

[0830] Step 5:

[0831] The terminal receives the music list sent from the server and displays it to the user. The user selects a song to play from the displayed list. The input here is the music list, and the output is the song selected by the user.

[0832] Step 6:

[0833] The device plays the selected music. During playback, the patient's facial expression is monitored in real time using the smartphone camera. The input data is a facial image captured by the camera, and the OpenCV library is used to detect the facial region and preprocess the image.

[0834] Step 7:

[0835] The device inputs the preprocessed facial image data into the emotion recognition model (emotion_model.h5) and analyzes facial expressions in real time. The output is the detected emotion (e.g., smiling, relaxed, angry, etc.).

[0836] Step 8:

[0837] The monitoring results are fed back to the server, which uses this feedback data to analyze the patient's reactions using a generative AI model to improve future music recommendation algorithms. The input is emotional data, and the output is updates to the learning database.

[0838] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0839] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during playback, a feedback means, and an emotion engine for recognizing the user's emotions.

[0840] User operations

[0841] 1. Install and launch the app

[0842] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0843] 2. Enter patient information

[0844] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0845] Server Processing

[0846] 1. Receiving and storing patient information

[0847] The server receives the patient information sent from the terminal and stores it in a database.

[0848] 2. Launching the AI ​​algorithm

[0849] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0850] 3. Listing Music Candidates

[0851] The server calls the music streaming service API based on the profile, lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop," and generates playlist candidates.

[0852] 4. Submit a playlist

[0853] The server transmits the generated playlist candidates to the terminal.

[0854] Terminal handling

[0855] 1. Receiving and displaying playlists

[0856] The terminal receives the playlist candidates sent from the server and displays them to the user.

[0857] 2. User selection of playlist

[0858] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0859] 3. Playing music

[0860] The terminal plays the songs from the selected playlist, allowing the patient to listen to music.

[0861] 4. Monitoring and analysis using an emotion engine

[0862] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state. The emotion engine then classifies the monitored emotional data to identify emotional states such as relaxation, joy, and sadness.

[0863] 5. Automatic adjustment

[0864] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state, for example, if the patient is determined to be in a relaxed state, it will adjust the music played to continue playing songs that help maintain that state.

[0865] 6. Submitting Feedback

[0866] The device feeds back the monitoring results to the server, which receives this feedback data and updates the AI ​​model to improve the accuracy of future playlist generation.

[0867] Specific examples

[0868] 1. Enter patient information

[0869] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0870] 2. Server Processing

[0871] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0872] 3. Generate a playlist

[0873] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and detect whether they are relaxed.

[0874] 4. The Role of the Emotional Engine

[0875] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[0876] 5. Submitting Feedback

[0877] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[0878] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[0879] The processing flow will be explained below.

[0880] Step 1:

[0881] Installing the app and logging in

[0882] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[0883] Step 2:

[0884] Entering patient information

[0885] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[0886] Step 3:

[0887] Sending patient information

[0888] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[0889] Step 4:

[0890] Receiving and storing information

[0891] The server receives the patient information sent from the terminal and stores the received information in a database.

[0892] Step 5:

[0893] Launching AI algorithms

[0894] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[0895] Step 6:

[0896] Listing music candidates

[0897] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[0898] Step 7:

[0899] Sending a Playlist

[0900] The server transmits the generated playlist candidates to the terminal.

[0901] Step 8:

[0902] Receiving and displaying playlists

[0903] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[0904] Step 9:

[0905] User-selected playlist

[0906] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[0907] Step 10:

[0908] Playing music

[0909] The device will play the songs from the selected playlist, allowing the user to listen to music.

[0910] Step 11:

[0911] Emotion engine monitoring and analysis

[0912] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state, which is then classified as relaxation, joy, etc.

[0913] Step 12:

[0914] automatic adjustment

[0915] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state: for example, if the patient is determined to be relaxed, songs that help maintain that state will be played next.

[0916] Step 13:

[0917] Send Feedback

[0918] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data to improve the accuracy of future playlist generation.

[0919] Specific examples

[0920] 1. Enter patient information

[0921] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[0922] 2. Server Processing

[0923] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[0924] 3. Generate a playlist

[0925] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice to determine whether they are relaxed.

[0926] 4. The Role of the Emotional Engine

[0927] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[0928] 5. Submitting Feedback

[0929] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[0930] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[0931] Example 2

[0932] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0933] The challenge is to provide a system that maximizes the effectiveness of music therapy, which aims to improve the quality of life of dementia patients. In particular, because it is difficult to automatically select and adjust music according to the patient's emotional state, real-time analysis and automatic adjustment technology using an emotion engine are required to solve this problem. Another challenge is to build an efficient system using artificial intelligence algorithms to learn the patient's preferences.

[0934] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0935] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for listing appropriate music, a means for playing the listed music, a means for monitoring the patient's reaction during music playback, a means for providing feedback on the monitored reaction, a means for using an emotion engine to analyze the user's emotions in real time, and a means for automatically adjusting the music being played based on the analyzed emotional state. This makes it possible to maximize the effectiveness of music therapy for dementia patients and automatically adjust the music experience according to the patient's emotional state.

[0936] "Patient information" refers to basic data related to an individual patient, such as the patient's name, date of birth, and preferred music genre.

[0937] "Preference learning means" refers to the process or algorithm used to learn music that matches a patient's preferences based on information input by the patient.

[0938] "Music listing method" refers to the process of selecting appropriate music based on the patient's preferences and compiling it into a playlist.

[0939] "Means for playing music" refers to the ability to actually play songs from a selected music playlist and have the patient listen to them.

[0940] "Monitoring means" refers to the process and equipment used to monitor and collect data on the patient's responses while the music is playing.

[0941] "Feedback means" refers to the process of analyzing the patient's reaction data obtained through monitoring, feeding the results back into the system, and using them to determine the next music selection and playback method.

[0942] "Emotion engine" refers to software and algorithms that analyze a patient's facial expressions and voice in real time to identify their emotional state.

[0943] "Means for automatically adjusting" means a function for automatically changing the selection of music being played or the playback sequence based on the emotional state identified by the emotional engine.

[0944] This invention provides a music therapy support system aimed at improving the quality of life of dementia patients, and is implemented by combining the following means. Each means will be explained based on the role of the server, terminal, and user.

[0945] Server Roles

[0946] Receiving and storing patient information

[0947] The server receives the patient information sent from the device and stores it in a database, including the patient's name, date of birth, and preferred music genre.

[0948] Launching AI algorithms and building profiles

[0949] When the server receives new patient information, it invokes an AI algorithm using a Python machine learning library (e.g., scikit-learn) to build a profile for that patient based on their preferences and the anecdotes they provide.

[0950] Listing music candidates

[0951] Based on the constructed profile, the server calls the music streaming service API (e.g., Spotify API), lists related songs, and generates playlist suggestions.

[0952] Sending a Playlist

[0953] The generated playlist candidates are sent to the device in JSON format.

[0954] Device Role

[0955] Installing and launching the app

[0956] Users download and install a dedicated app on their smartphone or tablet, enter the required information on the login screen, and log in.

[0957] Entering patient information

[0958] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre, then enters more detailed information such as an episode related to a specific song, and taps the "Save" button to complete the process.

[0959] Receiving and displaying playlists

[0960] The device receives the playlist suggestions sent from the server and displays them to the user. The displayed information includes song titles, artist names, album art, and so on.

[0961] User selects and plays a playlist

[0962] The user selects a song to play from the list of suggestions and taps the "Start Play" button to confirm the playlist. Playback begins and the patient begins listening to the music.

[0963] Monitoring and analysis of patient responses

[0964] Real-time analysis by emotion engine

[0965] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to identify their emotional state. The emotion engine collects data on facial features and tone of voice to classify emotional states such as relaxation, joy, and sadness.

[0966] Auto-tuning and feedback

[0967] automatic adjustment

[0968] The emotion engine automatically adjusts the music being played based on the analysis results. For example, if it determines that the patient is relaxed, it will adjust the music to continue playing to help maintain that state.

[0969] Send Feedback

[0970] The device feeds back the monitoring results to the server in real time, and the server stores the feedback in a database and uses it as training data to help generate the next playlist.

[0971] Specific examples

[0972] Prompts for entering patient information and generating a playlist

[0973] The user starts the app and enters their name ("Yamada Taro"), their date of birth ("July 15, 1940"), their favorite music ("Showa pop"), and an episode related to a specific song by a certain famous artist.

[0974] Based on this information, the server creates a list of songs related to "Showa pop music" and "certain famous artists" and generates a playlist of candidates. The user selects a "specific song name" and "another song name" from the playlist sent to the device and begins playback.

[0975] Prompt Sentence Examples

[0976] "Please generate a relaxing playlist for a male dementia patient over 70 years old who likes Showa-era pop music. For example, please include a specific song by a certain famous artist or a hit song from the Showa era."

[0977] This system can maximize the effectiveness of music therapy for dementia patients and provide a music experience that suits their emotional state. Real-time analysis and automatic adjustment functions based on an emotion engine are expected to further enhance the effect.

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1:

[0980] Installing and launching the app

[0981] Users download and install a dedicated app on their smartphone or tablet, then launch the app and log in by entering their username and password on the login screen.

[0982] Input: None

[0983] Output: None

[0984] Specific operation: A user downloads the app from the Google Play Store or Apple App Store, launches it after installation, and then logs in by entering the required information on the login screen.

[0985] Step 2:

[0986] Entering patient information

[0987] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre. They then press the "Next" button to enter more detailed information, such as an episode related to a specific song, and tap the "Save" button to complete the process.

[0988] Input: Patient information (name, date of birth, music preferences, episodes)

[0989] Output: Patient information is saved on the device

[0990] Specific operation: The user enters the required information into the form, presses the button to proceed to the next screen, and when the series of information input is completed, presses the save button to save the information in the terminal.

[0991] Step 3:

[0992] Transmitting and storing patient information

[0993] The terminal transmits the stored patient information to a server, which receives the information and stores it in a database.

[0994] Input: Patient information stored on the device

[0995] Output: Patient information stored in the server database

[0996] Specific operation: The terminal sends patient information to the server via an HTTP request, and the server receives the information and stores it in the database.

[0997] Step 4:

[0998] Launching AI algorithms and building profiles

[0999] When the server receives new patient information, it activates an AI algorithm that builds a patient profile based on the patient's preferences and anecdotes.

[1000] Input: Patient information stored in the server database

[1001] Output: Patient preference profile

[1002] Specific operation: The server uses a Python machine learning library (e.g., scikit-learn) to generate a patient preference profile.

[1003] Step 5:

[1004] Listing music candidates

[1005] The server's AI model calls the music streaming service API based on the constructed profile, lists appropriate songs, and generates playlist suggestions.

[1006] Input: Patient Preference Profile

[1007] Output: Playlist suggestions

[1008] Specific operation: The server calls the Spotify API, searches for songs using keywords such as "Showa pop music," and compiles the generated song list in JSON format as playlist candidates.

[1009] Step 6:

[1010] Sending a Playlist

[1011] The server transmits the generated playlist candidates to the terminal.

[1012] Input: Playlist suggestions

[1013] Output: Playlist suggestions sent to device

[1014] Specific operation: The server sends playlist candidates to the terminal as an HTTP response.

[1015] Step 7:

[1016] Receiving and displaying playlists

[1017] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1018] Input: Playlist suggestions from the server

[1019] Output: Playlist suggestions displayed to the user

[1020] Specific operation: The device displays the received playlist candidates to the user on a GUI, where the user can check information such as song title, artist name, and album art.

[1021] Step 8:

[1022] User selects and plays a playlist

[1023] The user selects songs to play from the displayed playlist candidates and taps the "Start Playing" button to confirm the playlist.

[1024] Input: Song selected by user

[1025] Output: Song that started playing

[1026] Specific behavior: The user selects a song from the list and presses the play button to play the music. The music will be played through the device's speakers or headphones.

[1027] Step 9:

[1028] Emotion engine monitoring and analysis

[1029] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state.

[1030] Input: facial expression data and voice data of the patient being played back

[1031] Output: Patient's emotional state (relaxed, happy, sad, etc.)

[1032] Specific operation: The device uses the built-in camera and microphone to collect facial expression and voice data, which the emotion engine analyzes to identify the user's emotional state.

[1033] Step 10:

[1034] automatic adjustment

[1035] The emotion engine automatically adjusts songs in the currently playing playlist based on the determined emotional state.

[1036] Input: Emotional state determined by the emotion engine

[1037] Output: Auto-tuned music playback sequence

[1038] Specific operation: The emotion engine adjusts the selection of the next song and the playback order, for example by changing the currently playing song to one with a relaxing effect.

[1039] Step 11:

[1040] Send Feedback

[1041] The device sends the monitoring results as feedback to the server, which receives them and stores them in a database to help improve the AI ​​model.

[1042] Input: Monitoring results

[1043] Output: The AI ​​model on the server is updated.

[1044] Specific operation: The server receives the monitoring results sent from the device and stores them in a database, which improves the accuracy of the next playlist generation.

[1045] (Application example 2)

[1046] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1047] Current music therapy systems have difficulty selecting appropriate music based on the individual preferences of dementia patients. Furthermore, there is no way to monitor the impact of the music being played on the patient's emotional state in real time and automatically adjust the music selection based on that information. This makes it difficult to maximize the effectiveness of music therapy.

[1048] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1049] In this invention, the server includes means for inputting patient information, means for learning the patient's preferences based on the input patient information, means for listing appropriate music, means for playing the listed music, means for monitoring the patient's reaction during music playback, means for feeding back the monitored reaction, means for automatically adjusting the music based on the reaction data, and means for analyzing emotions in real time using a generative AI model, thereby making it possible to select music that matches the patient's preferences and automatically adjust the music in real time according to the patient's emotional state.

[1050] "Means for inputting patient information" refers to an interface that allows a user to input information such as the patient's name, date of birth, and preferred music genre.

[1051] "Means for learning patient preferences" refers to an artificial intelligence algorithm that learns the patient's preferred music and episodes based on input patient information.

[1052] "Means for listing appropriate music" refers to a mechanism for listing music suitable for a patient based on learned preferences and generating a playlist.

[1053] "Means for playing the listed music" refers to a function for playing the selected music using an audio device.

[1054] "Means for monitoring patient reactions while music is being played" refers to a system for monitoring the patient's facial expressions, voice, heart rate, etc. in real time while music is being played and analyzing their emotional state.

[1055] "Means for providing feedback on monitored responses" refers to a feedback function that records the monitored emotional state of the patient in a database and uses this information to improve the overall operation of the system and music selection.

[1056] "Means for automatically adjusting music based on reaction data" refers to a mechanism for changing and adjusting currently playing music in real time based on monitored emotional data.

[1057] "Generative AI model" refers to a trained artificial intelligence model used to make music recommendations and reaction analysis based on input data.

[1058] "Real-time emotion analysis means" refers to a system that analyzes a patient's facial expressions, voice, and heart rate in real time to identify their current emotional state.

[1059] This invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reactions during playback, a means for providing feedback, and a generative AI model for analyzing emotions in real time.

[1060] Server Processing

[1061] The server receives patient information and stores it in a database. When new patient information is received, the server activates an artificial intelligence (AI) algorithm to build a profile based on the patient's preferences and input episodes. Based on the profiled data, the server calls a music streaming service API to list appropriate songs. Finally, the generated playlist suggestions are sent to the user's device.

[1062] Terminal handling

[1063] The device displays the playlist candidates received from the server to the user. The user selects songs to play from the displayed playlist candidates. While the music is playing, the device uses a generative AI model to analyze the patient's facial expressions, voice, heart rate, etc. in real time to determine their emotional state. Based on the emotional state, the device automatically adjusts the current playlist to play songs that will help the patient maintain a relaxed state. The monitored emotional data is fed back to the server and used to improve the accuracy of the next playlist generation.

[1064] Hardware and Software Used

[1065] Hardware: A webcam and heart rate sensor are used to collect patient biometric information, as well as speakers and a headset for playing music.

[1066] Software: Python, OpenCV, TensorFlow, and DeepFace libraries are used. Data processing and AI operations are performed using Python programs, and face recognition and emotion analysis are performed using OpenCV and DeepFace libraries.

[1067] Specific examples

[1068] The server receives the patient's name ("Yamada Taro"), date of birth (July 15, 1940), and favorite music genre ("Showa pop"), and generates playlist candidates based on this. The device displays the playlist candidates, and songs like "Like the Flow of a River" and "Love Vacation" are selected. During playback, the generative AI model analyzes the patient's facial expressions and voice to detect whether they are relaxed. If it is determined that they are relaxed, it automatically changes to music with a similar effect.

[1069] Prompt Sentence Examples

[1070] Taro Yamada's favorite music genre is "Showa pop," and there is an episode related to a specific song, "Like the Flow of a River." Based on this, create an AI model that can recommend appropriate music. Also, include a system that monitors the subject's emotional state in real time and automatically adjusts the music list.

[1071] As described above, the present invention improves the quality of life of dementia patients by recommending music based on the patient's individual preferences and analyzing and adjusting their emotional state in real time.

[1072] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1073] Step 1:

[1074] The user installs the app on their smartphone or device and launches it. The user enters the required information (e.g., username and password) on the login screen and logs in.

[1075] Input: User login information

[1076] Output: Login successful message, main menu displayed

[1077] Specific operation: When the user enters login information and presses the send button, the device sends an authentication request to the authentication server, and if authentication is successful, the main menu is displayed.

[1078] Step 2:

[1079] The user enters new patient information by clicking the "Register New Patient" button, entering basic information such as the patient's name, date of birth, and favorite music genre, and then clicking the "Next" button to proceed to the detailed information entry screen.

[1080] Input: Patient's name, date of birth, favorite music genre, detailed episode

[1081] Output: Patient information data

[1082] Specific operation: When the user enters patient information and presses the save button, the terminal sends and saves this information to the database.

[1083] Step 3:

[1084] The server receives the patient information sent from the device and stores it in a database. The reception of new patient information triggers the AI ​​algorithm.

[1085] Input: Patient information data

[1086] Output: AI algorithm execution flag, patient profile

[1087] How it works: When the server receives patient information, it writes it into a database and then runs an AI algorithm to generate a profile based on the patient's preferences.

[1088] Step 4:

[1089] Based on the generated patient profile, the server calls a music streaming service API to list relevant songs.

[1090] Input: Patient Profile

[1091] Output: Playlist suggestions

[1092] How it works: The AI ​​algorithm analyzes profile data, uses music streaming service APIs to collect related songs, and generates a list of playlist suggestions.

[1093] Step 5:

[1094] The server transmits the generated playlist candidates to the terminal, which receives them and displays them to the user.

[1095] Input: Playlist suggestions

[1096] Output: Playlist suggestions displayed on the device

[1097] Specific operation: The server transmits the playlist data to the terminal, and the terminal displays the received data on the user interface.

[1098] Step 6:

[1099] The user selects songs to play from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[1100] Input: User song selection

[1101] Output: Playback start signal, selected playlist

[1102] Specific operation: When the user selects a song and presses the start playback button, the data of the selected playlist is sent to the playback engine.

[1103] Step 7:

[1104] The device plays songs from the selected playlist, while the patient's facial expressions, voice, and heart rate are analyzed in real time using the HMD, camera, and heart rate sensor.

[1105] Input: Selected playlist, patient vitals

[1106] Output: Playing music, real-time analysis data

[1107] Specific operation: The device starts the playback engine and begins playing the music, and at the same time, it acquires data using a biometric information collection device and has it analyzed by the generative AI model.

[1108] Step 8:

[1109] Automatically adjusts songs in the current playlist based on real-time analyzed emotional data.

[1110] Input: Parsed emotion data

[1111] Output: Adjusted song list, next song to play

[1112] How it works: The generative AI model evaluates the currently playing song based on emotional data and automatically changes or adjusts the next song if necessary.

[1113] Step 9:

[1114] The monitoring results are fed back to the server to improve the accuracy of future playlist generation.

[1115] Input: Real-time analysis data and adjustment data

[1116] Output: Updated AI model, feedback data saved

[1117] Specific operation: The device sends emotion analysis data and adjustment data to the server, which stores it in a database and uses it to update the AI ​​model.

[1118] Through these steps, this music therapy support system provides music based on the individual preferences of dementia patients and analyzes and adjusts their emotions in real time, thereby improving their quality of life.

[1119] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1120] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1121] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1122] [Third embodiment]

[1123] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1124] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1125] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1126] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1127] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1128] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1129] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1130] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1131] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1132] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1133] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1134] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1135] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during the music playback, and a means for providing feedback.

[1136] User operations

[1137] 1. Install and launch the app

[1138] The user downloads, installs, and launches the app, then enters the required information on the login screen and logs in.

[1139] 2. Enter patient information

[1140] The user clicks the "Register New Patient" button and enters basic information such as the patient's name, date of birth, preferred music genre, etc. They then press the "Next" button to proceed to a screen where additional details can be entered.

[1141] 3. Add episodes

[1142] The user clicks the "Add Episode" button and inputs a song the patient liked, such as "Like the Flow of a River" by Hibari Misora, and an episode related to that song. The user then clicks the "Save" button to complete the input.

[1143] Server Processing

[1144] 1. Receiving and storing patient information

[1145] The server receives the patient information sent by the user and stores it in a database.

[1146] 2. Launching the AI ​​algorithm

[1147] When the server receives new patient information, it activates an AI algorithm that builds a profile based on the patient's preferences and anecdotes.

[1148] 3. Listing Music Candidates

[1149] The server calls a music streaming service API using keywords such as "Misora ​​Hibari" and "Showa pop songs," lists relevant songs, and generates playlist candidates.

[1150] 4. Accumulation of learning data

[1151] The server-generated playlists and associated patient information are stored in a database and used to improve the AI ​​model in the future.

[1152] Terminal handling

[1153] 1. Receiving and displaying playlists

[1154] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1155] 2. User selection of playlist

[1156] The user selects "Like the Flow of a River" and "Love's Vacation" from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[1157] 3. Music playback and monitoring

[1158] The device plays songs from the selected playlist, allowing the patient to listen to music. During playback, the device monitors the patient's facial expressions with a camera to detect smiling or relaxed expressions.

[1159] 4. Submitting Feedback

[1160] The monitoring results are fed back to the server, which then performs additional learning on the AI ​​model based on this feedback.

[1161] Specific examples

[1162] 1. Enter patient information

[1163] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[1164] 2. Server Processing

[1165] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[1166] 3. Generate a playlist

[1167] The device displays playlist suggestions to the user, who then selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device detects the patient's smile and sends the monitoring results back to the server, which uses this information to improve the AI ​​model.

[1168] This will enhance the effectiveness of music therapy for dementia patients and improve their relaxation and communication.

[1169] The processing flow will be explained below.

[1170] Step 1:

[1171] Installation and Login

[1172] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1173] Step 2:

[1174] Entering patient information

[1175] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1176] Step 3:

[1177] Sending patient information

[1178] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[1179] Step 4:

[1180] Receiving and storing information

[1181] The server receives the patient information sent from the terminal and stores the received information in a database.

[1182] Step 5:

[1183] Launching AI algorithms

[1184] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1185] Step 6:

[1186] Listing music candidates

[1187] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[1188] Step 7:

[1189] Sending a Playlist

[1190] The server transmits the generated playlist candidates to the terminal.

[1191] Step 8:

[1192] Receiving and displaying playlists

[1193] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[1194] Step 9:

[1195] User-selected playlist

[1196] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[1197] Step 10:

[1198] Playing music

[1199] The device will play the songs from the selected playlist, allowing the user to listen to music.

[1200] Step 11:

[1201] Reaction monitoring

[1202] While the device plays music, it monitors the patient's facial expressions and behavior with a camera and records the monitored information (e.g., smiling or relaxed facial expressions).

[1203] Step 12:

[1204] Send Feedback

[1205] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data and uses it to improve future playlist generation.

[1206] Example 1

[1207] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] Traditional music therapy has faced the challenge of selecting effective music based on the preferences of dementia patients. In particular, there has been a lack of systems that can analyze the preferences and reactions of dementia patients in real time and provide appropriate music based on that analysis. As a result, the effectiveness of music therapy in improving patients' quality of life has been limited.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1210] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for creating a list of appropriate music, a means for monitoring the patient's reaction while music is being played, a means for providing feedback on the monitored reaction, and a means for storing the created playlist and related data for use in future improvements to the AI ​​model. This makes it possible to provide appropriate music according to the individual preferences of each patient, maximizing the effects of music therapy and improving the quality of life of dementia patients.

[1211] "Patient information" is personal data about a dementia patient, including basic information such as name, date of birth, and preferred music genre.

[1212] "Means for learning preferences" refers to technology that uses artificial intelligence algorithms to analyze a patient's preferences and tendencies based on input patient information and build a profile.

[1213] "Means for listing appropriate music" refers to technology that uses the API of a music streaming service to select relevant songs based on the patient's preferences and generate playlist candidates.

[1214] "Means for playing" refers to a device or software for playing the listed music.

[1215] "Monitoring means" refers to technology that uses cameras and sensors to observe the patient's facial expressions and behavior while music is playing, and collects data in real time.

[1216] "Feedback means" refers to a technology that sends monitored patient reaction data to a server and uses it for additional learning of the artificial intelligence model.

[1217] "Means for storing the generated playlists and associated data" refers to technology that stores playlist suggestions and patient response data in a database and uses them to improve future AI models.

[1218] "Artificial intelligence algorithms" refer to technologies that use machine learning and deep learning to find patterns in large amounts of data and make predictions based on patient preferences and responses.

[1219] A "music streaming service API" is a program interface provided by an external music distribution service, and refers to a mechanism for obtaining song information based on specific keywords.

[1220] This invention is a music therapy support system for improving the quality of life of dementia patients. This system involves a series of processes: inputting patient information, learning preferences, listing appropriate music, playing music, monitoring the patient's response, and providing feedback.

[1221] System configuration

[1222] Hardware and Software

[1223] 1. Server

[1224] Hardware: High-performance server machine (e.g., Dell PowerEdge server)

[1225] software:

[1226] Database: MySQL

[1227] AI algorithms: Python-based machine learning libraries (e.g., TensorFlow, PyTorch)

[1228] Music streaming service APIs: Spotify API and Apple Music API

[1229] 2. Terminal

[1230] Hardware: Smartphones and tablets (e.g. iPhone, iPad, Android devices)

[1231] software:

[1232] Application: Music therapy support app

[1233] Monitoring technology: camera modules (e.g., built-in device cameras), facial recognition software

[1234] System Operation

[1235] 1. Enter patient information

[1236] Users install the app on their device, launch it, and after logging in, enter basic information such as the patient's name, date of birth, and favorite music genre on the "New Patient Registration" screen.

[1237] Examples:

[1238] Enter the patient's name "Yamada Taro," date of birth "July 15, 1940," and favorite music genre "Showa era pop music."

[1239] The episode "Misora ​​Hibari's 'Like the Flow of a River'" has been added.

[1240] 2. Learning preferences

[1241] The server receives the patient information sent by the user, stores it in a database, and then activates an AI algorithm to build a profile based on the patient's preferences and anecdotes.

[1242] Examples:

[1243] The server runs an AI algorithm based on new patient information to obtain song information related to "Showa pop songs" and "Misora ​​Hibari."

[1244] 3. Listing Music Candidates

[1245] The server calls the music streaming service API, lists relevant songs, and generates playlist suggestions.

[1246] Examples:

[1247] The server calls the Spotify API to compile a list of songs related to "Misora ​​Hibari" and "Showa era pop songs" and generates playlist candidates.

[1248] 4. Playing Music

[1249] The device displays playlist candidates to the user, and the user selects the desired songs. When the user presses the "Start Play" button, the music begins to play.

[1250] Examples:

[1251] The user selects "Like the Flow of a River" and "Love Vacation" and presses the "Start Play" button.

[1252] 5. Monitoring patient response

[1253] The device uses a camera to monitor the patient's facial expressions during playback and detects smiling and relaxed expressions.

[1254] Examples:

[1255] The device's camera captures a picture of the patient's face, and facial recognition software detects a smile.

[1256] 6. Feedback

[1257] The device feeds back the monitoring results to the server, which then uses this information to perform additional learning on the AI ​​model.

[1258] Examples:

[1259] The device sends the detected smile data to the server, which then uses this data to retrain the AI ​​model.

[1260] Operating Procedure

[1261] Specific examples of operation procedures

[1262] 1. Install and launch the app

[1263] The user downloads and installs the app on their device.

[1264] Once the installation is complete, tap the app icon to launch it. Enter the required information on the login screen to complete the login process.

[1265] Specific prompt examples:

[1266] Download and install the app, then enter your details on the login screen to log in.

[1267] 2. Enter patient information

[1268] The user clicks the "Register New Patient" button, enters the required information, then clicks the "Add Episode" button, enters the specific episode, and saves it.

[1269] Specific prompt examples:

[1270] When entering basic patient information, click the "Register New Patient" button and enter the required information.

[1271] This will enable music therapy for dementia patients to be managed efficiently and provide music tailored to the individual preferences of each patient, thereby enhancing the effectiveness of the therapy.

[1272] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1273] Step 1: Install and launch the app

[1274] The user downloads and installs the app on their device.

[1275] Input: Download link

[1276] Output: Installed apps

[1277] Specific behavior:

[1278] The user searches for a "music therapy support app" in the app store, downloads and installs it, taps the app icon to launch the app, and enters their username and password on the login screen to complete the login process.

[1279] Step 2: Enter patient information

[1280] The user enters basic patient information on the "New Patient Registration" screen.

[1281] Input: Patient's name, date of birth, favorite music genre

[1282] Output: Entered patient information

[1283] Specific behavior:

[1284] The user clicks the "Register New Patient" button and enters his name "Yamada Taro," his date of birth "July 15, 1940," and his favorite music genre "Showa pop music."

[1285] Step 3: Add Episodes

[1286] The user enters and saves an episode related to the patient.

[1287] Input: Specific episode

[1288] Output: Added episode information

[1289] Specific behavior:

[1290] The user clicks the "Add Episode" button, enters the episode "Misora ​​Hibari's 'Like the Flow of a River'", and clicks the "Save" button to complete the process.

[1291] Step 4: Receiving and storing patient information (server)

[1292] The server stores the received patient information and episode information in a database.

[1293] Input: Patient and episode information

[1294] Output: Saved data

[1295] Specific behavior:

[1296] The server receives the patient information sent by the user and stores it in a MySQL database.

[1297] Step 5: Launching the AI ​​algorithm (server)

[1298] The server runs an artificial intelligence algorithm based on the patient information.

[1299] Input: Patient preference information

[1300] Output: Patient profile

[1301] Specific behavior:

[1302] The server runs a Python-based AI model that analyzes the patient's preferences and anecdotes to generate a profile.

[1303] Step 6: Listing music candidates (server)

[1304] The server uses the music streaming service API to retrieve related songs and generate playlist suggestions.

[1305] Input: Patient profile (music preferences and keywords)

[1306] Output: Music playlist suggestions

[1307] Specific behavior:

[1308] The server calls the Spotify API, retrieves related songs based on the keywords "Misora ​​Hibari" and "Showa pop songs," and generates playlist candidates.

[1309] Step 7: Receive and display the playlist (on your device)

[1310] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1311] Input: Playlist candidate data

[1312] Output: Playlist suggestions displayed

[1313] Specific behavior:

[1314] The device retrieves the playlist data from the server and displays it on the app screen.

[1315] Step 8: User selects playlist (User)

[1316] The user selects a song from the playlist candidates and starts playback.

[1317] Input: Selected song

[1318] Output: Finalized playlist

[1319] Specific behavior:

[1320] The user selects "Like the Flow of a River" and "Love Vacation" on the playlist screen and presses the "Start Play" button.

[1321] Step 9: Playing and Monitoring Music (Device)

[1322] The device plays the selected song and monitors the patient's response.

[1323] Input: Confirmed playlist

[1324] Output: Patient monitoring data

[1325] Specific behavior:

[1326] While the device plays music, it uses a camera module to monitor the patient's facial expressions and detect smiling or relaxed expressions.

[1327] Step 10: Send Feedback (Device)

[1328] The device sends the monitoring results to the server, which then uses them for further learning of the AI ​​model.

[1329] Input: Monitoring data

[1330] Output: Update data for the AI ​​model

[1331] Specific behavior:

[1332] The monitoring data detected by the device is sent to the server, which then uses this data to retrain the AI ​​model.

[1333] This enables the system to efficiently provide music tailored to the individual preferences of dementia patients, maximizing the effectiveness of music therapy.

[1334] (Application example 1)

[1335] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1336] It is known that music therapy is effective in helping elderly people and dementia patients maintain a better mental state in their daily lives. However, there is no system yet that can provide optimal music for each patient and evaluate its effectiveness in real time. Furthermore, when elderly people listen to music, it is necessary not only to play the music but also to verify whether the music actually brings relaxation and a sense of happiness. To address this issue, we aim to provide a system that selects music based on individual patient information and evaluates its effectiveness in real time and provides feedback.

[1337] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1338] In this invention, the server

[1339] a means for inputting patient information;

[1340] means for learning patient preferences based on input patient information;

[1341] A means of listing suitable music;

[1342] a means for playing the listed music;

[1343] a means for monitoring the patient's response while the music is being played;

[1344] a means for providing feedback on the monitored response;

[1345] A means of analyzing facial expressions in real time; and

[1346] A means of providing feedback using generative AI models; and

[1347] This makes it possible to provide optimal music for individual patients, evaluate its effectiveness in real time, and provide feedback.

[1348] The "means for inputting patient information" is an interface for inputting basic data such as the patient's name, date of birth, and musical preferences.

[1349] The "means for learning patient preferences based on input patient information" is a system that uses an artificial intelligence algorithm to learn a patient's musical preferences based on patient information.

[1350] The "means for listing appropriate music" is a system that suggests music that may be of interest in the form of a list based on learned preferences.

[1351] The "means for playing the listed music" is an audio playback system that actually plays the selected music so that the patient can listen to it.

[1352] The "means for monitoring the patient's reactions while music is being played" is a system that uses cameras and sensors to observe and record the patient's facial expressions and behavior while music is being played.

[1353] The "means for feeding back monitored responses" refers to a feedback mechanism that sends the monitoring results to a server or artificial intelligence system to help select the next piece of music or improve treatment methods.

[1354] The "means for analyzing facial expressions in real time" is a system for analyzing facial expressions from a patient's face photographed in real time using a camera and evaluating the results.

[1355] "Means for providing feedback using a generative AI model" refers to a mechanism for providing optimal feedback based on data analyzed using a generative AI model, thereby improving the system.

[1356] System Program

[1357] The invention can be implemented using a smartphone application with the following functions: The application includes inputting and learning patient information, listing and playing appropriate music, monitoring responses, and providing feedback.

[1358] Program processing

[1359] The application begins with a user entering patient information (such as name, date of birth, and favorite music genre) on a smartphone. This information is then sent to a server and stored in a database. The server then uses an artificial intelligence algorithm to learn the patient's preferences and presents a list of appropriate music based on that information.

[1360] Hardware and Software

[1361] Hardware: Smartphone's built-in camera

[1362] Software: OpenCV (image processing library), Keras (deep learning library), emotion_model.h5 (emotion recognition model)

[1363] Server processing: AI algorithms, databases

[1364] The server uses the generative AI model to generate a list of music based on the newly entered patient information and sends it back to the application, which then displays the list of music suggestions to the user and plays the music selected by the user.

[1365] Specific processing examples

[1366] While music is playing, the smartphone camera monitors the patient's reactions. Using OpenCV for real-time facial analysis, the captured facial image data is fed into an emotion recognition model. The model identifies facial expressions such as smiling or relaxed, and based on that, evaluates whether the currently playing music is appropriate for the patient.

[1367] The monitoring results are fed back to the server and used by the generative AI model to select music for future use and improve the system. For example, if a patient shows many relaxed facial expressions while listening to a particular piece of music, that piece of music will likely be recommended to similar patients.

[1368] Prompt Sentence Examples

[1369] "Capture a photo, analyze the emotion in real time, and display the results."

[1370] "Use an AI model to determine in real time whether an elderly person is relaxed and provide feedback on the results."

[1371] This invention will enhance the effectiveness of music therapy and improve the quality of daily life for elderly people and dementia patients.

[1372] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1373] Step 1:

[1374] The user launches the smartphone application and enters patient information. Specifically, a form is displayed in which the patient can enter their name, date of birth, and favorite music genre. This information is collected as input data.

[1375] Step 2:

[1376] The patient information entered by the user is sent from the terminal to the server, where it is saved in a database. The input data is sent in JSON format and stored in the database. The data processing performed here involves converting the input data into the correct format.

[1377] Step 3:

[1378] Based on the newly saved patient information, the server runs an artificial intelligence algorithm to learn the patient's preferences. The algorithm analyzes the input data and creates a profile of the patient's musical preferences based on the learning database. The output is a list of music genres and artists that are suitable for the patient.

[1379] Step 4:

[1380] The server uses the generative AI model to generate a list of suitable music. Specifically, it calls the API of a music streaming service based on the patient's preference profile to retrieve candidate songs. The output is a list of multiple songs suitable for the patient.

[1381] Step 5:

[1382] The terminal receives the music list sent from the server and displays it to the user. The user selects a song to play from the displayed list. The input here is the music list, and the output is the song selected by the user.

[1383] Step 6:

[1384] The device plays the selected music. During playback, the patient's facial expression is monitored in real time using the smartphone camera. The input data is a facial image captured by the camera, and the OpenCV library is used to detect the facial region and preprocess the image.

[1385] Step 7:

[1386] The device inputs the preprocessed facial image data into the emotion recognition model (emotion_model.h5) and analyzes facial expressions in real time. The output is the detected emotion (e.g., smiling, relaxed, angry, etc.).

[1387] Step 8:

[1388] The monitoring results are fed back to the server, which uses this feedback data to analyze the patient's reactions using a generative AI model to improve future music recommendation algorithms. The input is emotional data, and the output is updates to the learning database.

[1389] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1390] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during playback, a feedback means, and an emotion engine for recognizing the user's emotions.

[1391] User operations

[1392] 1. Install and launch the app

[1393] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1394] 2. Enter patient information

[1395] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1396] Server Processing

[1397] 1. Receiving and storing patient information

[1398] The server receives the patient information sent from the terminal and stores it in a database.

[1399] 2. Launching the AI ​​algorithm

[1400] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1401] 3. Listing Music Candidates

[1402] The server calls the music streaming service API based on the profile, lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop," and generates playlist candidates.

[1403] 4. Submit a playlist

[1404] The server transmits the generated playlist candidates to the terminal.

[1405] Terminal handling

[1406] 1. Receiving and displaying playlists

[1407] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1408] 2. User selection of playlist

[1409] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[1410] 3. Playing music

[1411] The terminal plays the songs from the selected playlist, allowing the patient to listen to music.

[1412] 4. Monitoring and analysis using an emotion engine

[1413] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state. The emotion engine then classifies the monitored emotional data to identify emotional states such as relaxation, joy, and sadness.

[1414] 5. Automatic adjustment

[1415] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state, for example, if the patient is determined to be in a relaxed state, it will adjust the music played to continue playing songs that help maintain that state.

[1416] 6. Submitting Feedback

[1417] The device feeds back the monitoring results to the server, which receives this feedback data and updates the AI ​​model to improve the accuracy of future playlist generation.

[1418] Specific examples

[1419] 1. Enter patient information

[1420] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[1421] 2. Server Processing

[1422] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[1423] 3. Generate a playlist

[1424] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and detect whether they are relaxed.

[1425] 4. The Role of the Emotional Engine

[1426] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[1427] 5. Submitting Feedback

[1428] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[1429] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[1430] The processing flow will be explained below.

[1431] Step 1:

[1432] Installing the app and logging in

[1433] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1434] Step 2:

[1435] Entering patient information

[1436] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1437] Step 3:

[1438] Sending patient information

[1439] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[1440] Step 4:

[1441] Receiving and storing information

[1442] The server receives the patient information sent from the terminal and stores the received information in a database.

[1443] Step 5:

[1444] Launching AI algorithms

[1445] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1446] Step 6:

[1447] Listing music candidates

[1448] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[1449] Step 7:

[1450] Sending a Playlist

[1451] The server transmits the generated playlist candidates to the terminal.

[1452] Step 8:

[1453] Receiving and displaying playlists

[1454] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[1455] Step 9:

[1456] User-selected playlist

[1457] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[1458] Step 10:

[1459] Playing music

[1460] The device will play the songs from the selected playlist, allowing the user to listen to music.

[1461] Step 11:

[1462] Emotion engine monitoring and analysis

[1463] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state, which is then classified as relaxation, joy, etc.

[1464] Step 12:

[1465] automatic adjustment

[1466] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state: for example, if the patient is determined to be relaxed, songs that help maintain that state will be played next.

[1467] Step 13:

[1468] Send Feedback

[1469] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data to improve the accuracy of future playlist generation.

[1470] Specific examples

[1471] 1. Enter patient information

[1472] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[1473] 2. Server Processing

[1474] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[1475] 3. Generate a playlist

[1476] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice to determine whether they are relaxed.

[1477] 4. The Role of the Emotional Engine

[1478] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[1479] 5. Submitting Feedback

[1480] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[1481] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[1482] Example 2

[1483] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1484] The challenge is to provide a system that maximizes the effectiveness of music therapy, which aims to improve the quality of life of dementia patients. In particular, because it is difficult to automatically select and adjust music according to the patient's emotional state, real-time analysis and automatic adjustment technology using an emotion engine are required to solve this problem. Another challenge is to build an efficient system using artificial intelligence algorithms to learn the patient's preferences.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1486] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for listing appropriate music, a means for playing the listed music, a means for monitoring the patient's reaction during music playback, a means for providing feedback on the monitored reaction, a means for using an emotion engine to analyze the user's emotions in real time, and a means for automatically adjusting the music being played based on the analyzed emotional state. This makes it possible to maximize the effectiveness of music therapy for dementia patients and automatically adjust the music experience according to the patient's emotional state.

[1487] "Patient information" refers to basic data related to an individual patient, such as the patient's name, date of birth, and preferred music genre.

[1488] "Preference learning means" refers to the process or algorithm used to learn music that matches a patient's preferences based on information input by the patient.

[1489] "Music listing method" refers to the process of selecting appropriate music based on the patient's preferences and compiling it into a playlist.

[1490] "Means for playing music" refers to the ability to actually play songs from a selected music playlist and have the patient listen to them.

[1491] "Monitoring means" refers to the process and equipment used to monitor and collect data on the patient's responses while the music is playing.

[1492] "Feedback means" refers to the process of analyzing the patient's reaction data obtained through monitoring, feeding the results back into the system, and using them to determine the next music selection and playback method.

[1493] "Emotion engine" refers to software and algorithms that analyze a patient's facial expressions and voice in real time to identify their emotional state.

[1494] "Means for automatically adjusting" means a function for automatically changing the selection of music being played or the playback sequence based on the emotional state identified by the emotional engine.

[1495] This invention provides a music therapy support system aimed at improving the quality of life of dementia patients, and is implemented by combining the following means. Each means will be explained based on the role of the server, terminal, and user.

[1496] Server Roles

[1497] Receiving and storing patient information

[1498] The server receives the patient information sent from the device and stores it in a database, including the patient's name, date of birth, and preferred music genre.

[1499] Launching AI algorithms and building profiles

[1500] When the server receives new patient information, it invokes an AI algorithm using a Python machine learning library (e.g., scikit-learn) to build a profile for that patient based on their preferences and the anecdotes they provide.

[1501] Listing music candidates

[1502] Based on the constructed profile, the server calls the music streaming service API (e.g., Spotify API), lists related songs, and generates playlist suggestions.

[1503] Sending a Playlist

[1504] The generated playlist candidates are sent to the device in JSON format.

[1505] Device Role

[1506] Installing and launching the app

[1507] Users download and install a dedicated app on their smartphone or tablet, enter the required information on the login screen, and log in.

[1508] Entering patient information

[1509] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre, then enters more detailed information such as an episode related to a specific song, and taps the "Save" button to complete the process.

[1510] Receiving and displaying playlists

[1511] The device receives the playlist suggestions sent from the server and displays them to the user. The displayed information includes song titles, artist names, album art, and so on.

[1512] User selects and plays a playlist

[1513] The user selects a song to play from the list of suggestions and taps the "Start Play" button to confirm the playlist. Playback begins and the patient begins listening to the music.

[1514] Monitoring and analysis of patient responses

[1515] Real-time analysis by emotion engine

[1516] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to identify their emotional state. The emotion engine collects data on facial features and tone of voice to classify emotional states such as relaxation, joy, and sadness.

[1517] Auto-tuning and feedback

[1518] automatic adjustment

[1519] The emotion engine automatically adjusts the music being played based on the analysis results. For example, if it determines that the patient is relaxed, it will adjust the music to continue playing to help maintain that state.

[1520] Send Feedback

[1521] The device feeds back the monitoring results to the server in real time, and the server stores the feedback in a database and uses it as training data to help generate the next playlist.

[1522] Specific examples

[1523] Prompts for entering patient information and generating a playlist

[1524] The user starts the app and enters their name ("Yamada Taro"), their date of birth ("July 15, 1940"), their favorite music ("Showa pop"), and an episode related to a specific song by a certain famous artist.

[1525] Based on this information, the server creates a list of songs related to "Showa pop music" and "certain famous artists" and generates a playlist of candidates. The user selects a "specific song name" and "another song name" from the playlist sent to the device and begins playback.

[1526] Prompt Sentence Examples

[1527] "Please generate a relaxing playlist for a male dementia patient over 70 years old who likes Showa-era pop music. For example, please include a specific song by a certain famous artist or a hit song from the Showa era."

[1528] This system can maximize the effectiveness of music therapy for dementia patients and provide a music experience that suits their emotional state. Real-time analysis and automatic adjustment functions based on an emotion engine are expected to further enhance the effect.

[1529] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1530] Step 1:

[1531] Installing and launching the app

[1532] Users download and install a dedicated app on their smartphone or tablet, then launch the app and log in by entering their username and password on the login screen.

[1533] Input: None

[1534] Output: None

[1535] Specific operation: A user downloads an app from the Google Play Store or Apple App Store, launches it after installation, and then logs in by entering the required information on the login screen.

[1536] Step 2:

[1537] Entering patient information

[1538] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre. They then press the "Next" button to enter more detailed information, such as an episode related to a specific song, and tap the "Save" button to complete the process.

[1539] Input: Patient information (name, date of birth, music preferences, episodes)

[1540] Output: Patient information is saved on the device

[1541] Specific operation: The user enters the required information into the form, presses the button to proceed to the next screen, and when the series of information input is completed, presses the save button to save the information in the terminal.

[1542] Step 3:

[1543] Transmitting and storing patient information

[1544] The terminal transmits the stored patient information to a server, which receives the information and stores it in a database.

[1545] Input: Patient information stored on the device

[1546] Output: Patient information stored in the server database

[1547] Specific operation: The terminal sends patient information to the server via an HTTP request, and the server receives the information and stores it in the database.

[1548] Step 4:

[1549] Launching AI algorithms and building profiles

[1550] When the server receives new patient information, it activates an AI algorithm that builds a patient profile based on the patient's preferences and anecdotes.

[1551] Input: Patient information stored in the server database

[1552] Output: Patient preference profile

[1553] Specific operation: The server uses a Python machine learning library (e.g., scikit-learn) to generate a patient preference profile.

[1554] Step 5:

[1555] Listing music candidates

[1556] The server's AI model calls the music streaming service API based on the constructed profile, lists appropriate songs, and generates playlist suggestions.

[1557] Input: Patient Preference Profile

[1558] Output: Playlist suggestions

[1559] Specific operation: The server calls the Spotify API, searches for songs using keywords such as "Showa pop music," and compiles the generated song list in JSON format as playlist candidates.

[1560] Step 6:

[1561] Sending a Playlist

[1562] The server transmits the generated playlist candidates to the terminal.

[1563] Input: Playlist suggestions

[1564] Output: Playlist suggestions sent to device

[1565] Specific operation: The server sends playlist candidates to the terminal as an HTTP response.

[1566] Step 7:

[1567] Receiving and displaying playlists

[1568] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1569] Input: Playlist suggestions from the server

[1570] Output: Playlist suggestions displayed to the user

[1571] Specific operation: The device displays the received playlist candidates to the user on a GUI, where the user can check information such as song title, artist name, and album art.

[1572] Step 8:

[1573] User selects and plays a playlist

[1574] The user selects songs to play from the displayed playlist candidates and taps the "Start Playing" button to confirm the playlist.

[1575] Input: Song selected by user

[1576] Output: Song that started playing

[1577] Specific behavior: The user selects a song from the list and presses the play button to play the music. The music will be played through the device's speakers or headphones.

[1578] Step 9:

[1579] Emotion engine monitoring and analysis

[1580] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state.

[1581] Input: facial expression data and voice data of the patient being played back

[1582] Output: Patient's emotional state (relaxed, happy, sad, etc.)

[1583] Specific operation: The device uses the built-in camera and microphone to collect facial expression and voice data, which the emotion engine analyzes to identify the user's emotional state.

[1584] Step 10:

[1585] automatic adjustment

[1586] The emotion engine automatically adjusts songs in the currently playing playlist based on the determined emotional state.

[1587] Input: Emotional state determined by the emotion engine

[1588] Output: Auto-tuned music playback sequence

[1589] Specific operation: The emotion engine adjusts the selection of the next song and the playback order, for example by changing the currently playing song to one with a relaxing effect.

[1590] Step 11:

[1591] Send Feedback

[1592] The device sends the monitoring results as feedback to the server, which receives them and stores them in a database to help improve the AI ​​model.

[1593] Input: Monitoring results

[1594] Output: The AI ​​model on the server is updated.

[1595] Specific operation: The server receives the monitoring results sent from the device and stores them in a database, which improves the accuracy of the next playlist generation.

[1596] (Application example 2)

[1597] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1598] Current music therapy systems have difficulty selecting appropriate music based on the individual preferences of dementia patients. Furthermore, there is no way to monitor the impact of the music being played on the patient's emotional state in real time and automatically adjust the music selection based on that information. This makes it difficult to maximize the effectiveness of music therapy.

[1599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1600] In this invention, the server includes means for inputting patient information, means for learning the patient's preferences based on the input patient information, means for listing appropriate music, means for playing the listed music, means for monitoring the patient's reaction during music playback, means for feeding back the monitored reaction, means for automatically adjusting the music based on the reaction data, and means for analyzing emotions in real time using a generative AI model, thereby making it possible to select music that matches the patient's preferences and automatically adjust the music in real time according to the patient's emotional state.

[1601] "Means for inputting patient information" refers to an interface that allows a user to input information such as the patient's name, date of birth, and preferred music genre.

[1602] "Means for learning patient preferences" refers to an artificial intelligence algorithm that learns the patient's preferred music and episodes based on input patient information.

[1603] "Means for listing appropriate music" refers to a mechanism for listing music suitable for a patient based on learned preferences and generating a playlist.

[1604] "Means for playing the listed music" refers to a function for playing the selected music using an audio device.

[1605] "Means for monitoring patient reactions while music is being played" refers to a system for monitoring the patient's facial expressions, voice, heart rate, etc. in real time while music is being played and analyzing their emotional state.

[1606] "Means for providing feedback on monitored responses" refers to a feedback function that records the monitored emotional state of the patient in a database and uses this information to improve the overall operation of the system and music selection.

[1607] "Means for automatically adjusting music based on reaction data" refers to a mechanism for changing and adjusting currently playing music in real time based on monitored emotional data.

[1608] "Generative AI model" refers to a trained artificial intelligence model used to make music recommendations and reaction analysis based on input data.

[1609] "Real-time emotion analysis means" refers to a system that analyzes a patient's facial expressions, voice, and heart rate in real time to identify their current emotional state.

[1610] This invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reactions during playback, a means for providing feedback, and a generative AI model for analyzing emotions in real time.

[1611] Server Processing

[1612] The server receives patient information and stores it in a database. When new patient information is received, the server activates an artificial intelligence (AI) algorithm to build a profile based on the patient's preferences and input episodes. Based on the profiled data, the server calls a music streaming service API to list appropriate songs. Finally, the generated playlist suggestions are sent to the user's device.

[1613] Terminal handling

[1614] The device displays the playlist candidates received from the server to the user. The user selects songs to play from the displayed playlist candidates. While the music is playing, the device uses a generative AI model to analyze the patient's facial expressions, voice, heart rate, etc. in real time to determine their emotional state. Based on the emotional state, the device automatically adjusts the current playlist to play songs that will help the patient maintain a relaxed state. The monitored emotional data is fed back to the server and used to improve the accuracy of the next playlist generation.

[1615] Hardware and Software Used

[1616] Hardware: A webcam and heart rate sensor are used to collect patient biometric information, as well as speakers and a headset for playing music.

[1617] Software: Python, OpenCV, TensorFlow, and DeepFace libraries are used. Data processing and AI operations are performed using Python programs, and face recognition and emotion analysis are performed using OpenCV and DeepFace libraries.

[1618] Specific examples

[1619] The server receives the patient's name ("Yamada Taro"), date of birth (July 15, 1940), and favorite music genre ("Showa pop"), and generates playlist candidates based on this. The device displays the playlist candidates, and songs like "Like the Flow of a River" and "Love Vacation" are selected. During playback, the generative AI model analyzes the patient's facial expressions and voice to detect whether they are relaxed. If it is determined that they are relaxed, it automatically changes to music with a similar effect.

[1620] Prompt Sentence Examples

[1621] Taro Yamada's favorite music genre is "Showa pop," and there is an episode related to a specific song, "Like the Flow of a River." Based on this, create an AI model that can recommend appropriate music. Also, include a system that monitors the subject's emotional state in real time and automatically adjusts the music list.

[1622] As described above, the present invention improves the quality of life of dementia patients by recommending music based on the patient's individual preferences and analyzing and adjusting their emotional state in real time.

[1623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1624] Step 1:

[1625] The user installs the app on their smartphone or device and launches it. The user enters the required information (e.g., username and password) on the login screen and logs in.

[1626] Input: User login information

[1627] Output: Login successful message, main menu displayed

[1628] Specific operation: When the user enters login information and presses the send button, the device sends an authentication request to the authentication server, and if authentication is successful, the main menu is displayed.

[1629] Step 2:

[1630] The user enters new patient information by clicking the "Register New Patient" button, entering basic information such as the patient's name, date of birth, and favorite music genre, and then clicking the "Next" button to proceed to the detailed information entry screen.

[1631] Input: Patient's name, date of birth, favorite music genre, detailed episode

[1632] Output: Patient information data

[1633] Specific operation: When the user enters patient information and presses the save button, the terminal sends and saves this information to the database.

[1634] Step 3:

[1635] The server receives the patient information sent from the device and stores it in a database. The reception of new patient information triggers the AI ​​algorithm.

[1636] Input: Patient information data

[1637] Output: AI algorithm execution flag, patient profile

[1638] How it works: When the server receives patient information, it writes it into a database and then runs an AI algorithm to generate a profile based on the patient's preferences.

[1639] Step 4:

[1640] Based on the generated patient profile, the server calls a music streaming service API to list relevant songs.

[1641] Input: Patient Profile

[1642] Output: Playlist suggestions

[1643] How it works: The AI ​​algorithm analyzes profile data, uses music streaming service APIs to collect related songs, and generates a list of playlist suggestions.

[1644] Step 5:

[1645] The server transmits the generated playlist candidates to the terminal, which receives them and displays them to the user.

[1646] Input: Playlist suggestions

[1647] Output: Playlist suggestions displayed on the device

[1648] Specific operation: The server transmits the playlist data to the terminal, and the terminal displays the received data on the user interface.

[1649] Step 6:

[1650] The user selects songs to play from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[1651] Input: User song selection

[1652] Output: Playback start signal, selected playlist

[1653] Specific operation: When the user selects a song and presses the start playback button, the data of the selected playlist is sent to the playback engine.

[1654] Step 7:

[1655] The device plays songs from the selected playlist, while the patient's facial expressions, voice, and heart rate are analyzed in real time using the HMD, camera, and heart rate sensor.

[1656] Input: Selected playlist, patient vitals

[1657] Output: Playing music, real-time analysis data

[1658] Specific operation: The device starts the playback engine and begins playing the music, and at the same time, it acquires data using a biometric information collection device and has it analyzed by the generative AI model.

[1659] Step 8:

[1660] Automatically adjusts songs in the current playlist based on real-time analyzed emotional data.

[1661] Input: Parsed emotion data

[1662] Output: Adjusted song list, next song to play

[1663] How it works: The generative AI model evaluates the currently playing song based on emotional data and automatically changes or adjusts the next song if necessary.

[1664] Step 9:

[1665] The monitoring results are fed back to the server to improve the accuracy of future playlist generation.

[1666] Input: Real-time analysis data and adjustment data

[1667] Output: Updated AI model, feedback data saved

[1668] Specific operation: The device sends emotion analysis data and adjustment data to the server, which stores it in a database and uses it to update the AI ​​model.

[1669] Through these steps, this music therapy support system provides music based on the individual preferences of dementia patients and analyzes and adjusts their emotions in real time, thereby improving their quality of life.

[1670] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1671] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1672] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1673] [Fourth embodiment]

[1674] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1675] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1676] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1677] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1678] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1679] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1680] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1681] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1682] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1683] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1684] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1685] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1686] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1687] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during the music playback, and a means for providing feedback.

[1688] User operations

[1689] 1. Install and launch the app

[1690] The user downloads, installs, and launches the app, then enters the required information on the login screen and logs in.

[1691] 2. Enter patient information

[1692] The user clicks the "Register New Patient" button and enters basic information such as the patient's name, date of birth, preferred music genre, etc. They then press the "Next" button to proceed to a screen where additional details can be entered.

[1693] 3. Add episodes

[1694] The user clicks the "Add Episode" button and inputs a song the patient liked, such as "Like the Flow of a River" by Hibari Misora, and an episode related to that song. The user then clicks the "Save" button to complete the input.

[1695] Server Processing

[1696] 1. Receiving and storing patient information

[1697] The server receives the patient information sent by the user and stores it in a database.

[1698] 2. Launching the AI ​​algorithm

[1699] When the server receives new patient information, it activates an AI algorithm that builds a profile based on the patient's preferences and anecdotes.

[1700] 3. Listing Music Candidates

[1701] The server calls a music streaming service API using keywords such as "Misora ​​Hibari" and "Showa pop songs," lists relevant songs, and generates playlist candidates.

[1702] 4. Accumulation of learning data

[1703] The server-generated playlists and associated patient information are stored in a database and used to improve the AI ​​model in the future.

[1704] Terminal handling

[1705] 1. Receiving and displaying playlists

[1706] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1707] 2. User selection of playlist

[1708] The user selects "Like the Flow of a River" and "Love's Vacation" from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[1709] 3. Music playback and monitoring

[1710] The device plays songs from the selected playlist, allowing the patient to listen to music. During playback, the device monitors the patient's facial expressions with a camera to detect smiling or relaxed expressions.

[1711] 4. Submitting Feedback

[1712] The monitoring results are fed back to the server, which then performs additional learning on the AI ​​model based on this feedback.

[1713] Specific examples

[1714] 1. Enter patient information

[1715] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[1716] 2. Server Processing

[1717] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[1718] 3. Generate a playlist

[1719] The device displays playlist suggestions to the user, who then selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device detects the patient's smile and sends the monitoring results back to the server, which uses this information to improve the AI ​​model.

[1720] This will enhance the effectiveness of music therapy for dementia patients and improve their relaxation and communication.

[1721] The processing flow will be explained below.

[1722] Step 1:

[1723] Installation and Login

[1724] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1725] Step 2:

[1726] Entering patient information

[1727] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1728] Step 3:

[1729] Sending patient information

[1730] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[1731] Step 4:

[1732] Receiving and storing information

[1733] The server receives the patient information sent from the terminal and stores the received information in a database.

[1734] Step 5:

[1735] Launching AI algorithms

[1736] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1737] Step 6:

[1738] Listing music candidates

[1739] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[1740] Step 7:

[1741] Sending a Playlist

[1742] The server transmits the generated playlist candidates to the terminal.

[1743] Step 8:

[1744] Receiving and displaying playlists

[1745] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[1746] Step 9:

[1747] User-selected playlist

[1748] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[1749] Step 10:

[1750] Playing music

[1751] The device will play the songs from the selected playlist, allowing the user to listen to music.

[1752] Step 11:

[1753] Reaction monitoring

[1754] While the device plays music, it monitors the patient's facial expressions and behavior with a camera and records the monitored information (e.g., smiling or relaxed facial expressions).

[1755] Step 12:

[1756] Send Feedback

[1757] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data and uses it to improve future playlist generation.

[1758] Example 1

[1759] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1760] Traditional music therapy has faced the challenge of selecting effective music based on the preferences of dementia patients. In particular, there has been a lack of systems that can analyze the preferences and reactions of dementia patients in real time and provide appropriate music based on that analysis. As a result, the effectiveness of music therapy in improving patients' quality of life has been limited.

[1761] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1762] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for creating a list of appropriate music, a means for monitoring the patient's reaction while music is being played, a means for providing feedback on the monitored reaction, and a means for storing the created playlist and related data for use in future improvements to the AI ​​model. This makes it possible to provide appropriate music according to the individual preferences of each patient, maximizing the effects of music therapy and improving the quality of life of dementia patients.

[1763] "Patient information" is personal data about a dementia patient, including basic information such as name, date of birth, and preferred music genre.

[1764] "Means for learning preferences" refers to technology that uses artificial intelligence algorithms to analyze a patient's preferences and tendencies based on input patient information and build a profile.

[1765] "Means for listing appropriate music" refers to technology that uses the API of a music streaming service to select relevant songs based on the patient's preferences and generate playlist candidates.

[1766] "Means for playing" refers to a device or software for playing the listed music.

[1767] "Monitoring means" refers to technology that uses cameras and sensors to observe the patient's facial expressions and behavior while music is playing, and collects data in real time.

[1768] "Feedback means" refers to a technology that sends monitored patient reaction data to a server and uses it for additional learning of the artificial intelligence model.

[1769] "Means for storing the generated playlists and associated data" refers to technology that stores playlist suggestions and patient response data in a database and uses them to improve future AI models.

[1770] "Artificial intelligence algorithms" refer to technologies that use machine learning and deep learning to find patterns in large amounts of data and make predictions based on patient preferences and responses.

[1771] A "music streaming service API" is a program interface provided by an external music distribution service, and refers to a mechanism for obtaining song information based on specific keywords.

[1772] This invention is a music therapy support system for improving the quality of life of dementia patients. This system involves a series of processes: inputting patient information, learning preferences, listing appropriate music, playing music, monitoring the patient's response, and providing feedback.

[1773] System configuration

[1774] Hardware and Software

[1775] 1. Server

[1776] Hardware: High-performance server machine (e.g., Dell PowerEdge server)

[1777] software:

[1778] Database: MySQL

[1779] AI algorithms: Python-based machine learning libraries (e.g., TensorFlow, PyTorch)

[1780] Music streaming service APIs: Spotify API and Apple Music API

[1781] 2. Terminal

[1782] Hardware: Smartphones and tablets (e.g. iPhone, iPad, Android devices)

[1783] software:

[1784] Application: Music therapy support app

[1785] Monitoring technology: camera modules (e.g., built-in device cameras), facial recognition software

[1786] System Operation

[1787] 1. Enter patient information

[1788] Users install the app on their device, launch it, and after logging in, enter basic information such as the patient's name, date of birth, and favorite music genre on the "New Patient Registration" screen.

[1789] Examples:

[1790] Enter the patient's name "Yamada Taro," date of birth "July 15, 1940," and favorite music genre "Showa era pop music."

[1791] The episode "Misora ​​Hibari's 'Like the Flow of a River'" has been added.

[1792] 2. Learning preferences

[1793] The server receives the patient information sent by the user, stores it in a database, and then activates an AI algorithm to build a profile based on the patient's preferences and anecdotes.

[1794] Examples:

[1795] The server runs an AI algorithm based on new patient information to obtain song information related to "Showa pop songs" and "Misora ​​Hibari."

[1796] 3. Listing Music Candidates

[1797] The server calls the music streaming service API, lists relevant songs, and generates playlist suggestions.

[1798] Examples:

[1799] The server calls the Spotify API to compile a list of songs related to "Misora ​​Hibari" and "Showa era pop songs" and generates playlist candidates.

[1800] 4. Playing Music

[1801] The device displays playlist candidates to the user, and the user selects the desired songs. When the user presses the "Start Play" button, the music begins to play.

[1802] Examples:

[1803] The user selects "Like the Flow of a River" and "Love Vacation" and presses the "Start Play" button.

[1804] 5. Monitoring patient response

[1805] The device uses a camera to monitor the patient's facial expressions during playback and detects smiling and relaxed expressions.

[1806] Examples:

[1807] The device's camera captures a picture of the patient's face, and facial recognition software detects a smile.

[1808] 6. Feedback

[1809] The device feeds back the monitoring results to the server, which then uses this information to perform additional learning on the AI ​​model.

[1810] Examples:

[1811] The device sends the detected smile data to the server, which then uses this data to retrain the AI ​​model.

[1812] Operating Procedure

[1813] Specific examples of operation procedures

[1814] 1. Install and launch the app

[1815] The user downloads and installs the app on their device.

[1816] Once the installation is complete, tap the app icon to launch it. Enter the required information on the login screen to complete the login process.

[1817] Specific prompt examples:

[1818] Download and install the app, then enter your details on the login screen to log in.

[1819] 2. Enter patient information

[1820] The user clicks the "Register New Patient" button, enters the required information, then clicks the "Add Episode" button, enters the specific episode, and saves it.

[1821] Specific prompt examples:

[1822] When entering basic patient information, click the "Register New Patient" button and enter the required information.

[1823] This will enable music therapy for dementia patients to be managed efficiently and provide music tailored to the individual preferences of each patient, thereby enhancing the effectiveness of the therapy.

[1824] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1825] Step 1: Install and launch the app

[1826] The user downloads and installs the app on their device.

[1827] Input: Download link

[1828] Output: Installed apps

[1829] Specific behavior:

[1830] The user searches for a "music therapy support app" in the app store, downloads and installs it, taps the app icon to launch the app, and enters their username and password on the login screen to complete the login process.

[1831] Step 2: Enter patient information

[1832] The user enters basic patient information on the "New Patient Registration" screen.

[1833] Input: Patient's name, date of birth, favorite music genre

[1834] Output: Entered patient information

[1835] Specific behavior:

[1836] The user clicks the "Register New Patient" button and enters his name "Yamada Taro," his date of birth "July 15, 1940," and his favorite music genre "Showa pop music."

[1837] Step 3: Add Episodes

[1838] The user enters and saves an episode related to the patient.

[1839] Input: Specific episode

[1840] Output: Added episode information

[1841] Specific behavior:

[1842] The user clicks the "Add Episode" button, enters the episode "Misora ​​Hibari's 'Like the Flow of a River'", and clicks the "Save" button to complete the process.

[1843] Step 4: Receiving and storing patient information (server)

[1844] The server stores the received patient information and episode information in a database.

[1845] Input: Patient and episode information

[1846] Output: Saved data

[1847] Specific behavior:

[1848] The server receives the patient information sent by the user and stores it in a MySQL database.

[1849] Step 5: Launching the AI ​​algorithm (server)

[1850] The server runs an artificial intelligence algorithm based on the patient information.

[1851] Input: Patient preference information

[1852] Output: Patient profile

[1853] Specific behavior:

[1854] The server runs a Python-based AI model that analyzes the patient's preferences and anecdotes to generate a profile.

[1855] Step 6: Listing music candidates (server)

[1856] The server uses the music streaming service API to retrieve related songs and generate playlist suggestions.

[1857] Input: Patient profile (music preferences and keywords)

[1858] Output: Music playlist suggestions

[1859] Specific behavior:

[1860] The server calls the Spotify API, retrieves related songs based on the keywords "Misora ​​Hibari" and "Showa pop songs," and generates playlist candidates.

[1861] Step 7: Receive and display the playlist (on your device)

[1862] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1863] Input: Playlist candidate data

[1864] Output: Playlist suggestions displayed

[1865] Specific behavior:

[1866] The device retrieves the playlist data from the server and displays it on the app screen.

[1867] Step 8: User selects playlist (User)

[1868] The user selects a song from the playlist candidates and starts playback.

[1869] Input: Selected song

[1870] Output: Finalized playlist

[1871] Specific behavior:

[1872] The user selects "Like the Flow of a River" and "Love Vacation" on the playlist screen and presses the "Start Play" button.

[1873] Step 9: Playing and Monitoring Music (Device)

[1874] The device plays the selected song and monitors the patient's response.

[1875] Input: Confirmed playlist

[1876] Output: Patient monitoring data

[1877] Specific behavior:

[1878] While the device plays music, it uses a camera module to monitor the patient's facial expressions and detect smiling or relaxed expressions.

[1879] Step 10: Send Feedback (Device)

[1880] The device sends the monitoring results to the server, which then uses them for further learning of the AI ​​model.

[1881] Input: Monitoring data

[1882] Output: Update data for the AI ​​model

[1883] Specific behavior:

[1884] The monitoring data detected by the device is sent to the server, which then uses this data to retrain the AI ​​model.

[1885] This enables the system to efficiently provide music tailored to the individual preferences of dementia patients, maximizing the effectiveness of music therapy.

[1886] (Application example 1)

[1887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1888] It is known that music therapy is effective in helping elderly people and dementia patients maintain a better mental state in their daily lives. However, there is no system yet that can provide optimal music for each patient and evaluate its effectiveness in real time. Furthermore, when elderly people listen to music, it is necessary not only to play the music but also to verify whether the music actually brings relaxation and a sense of happiness. To address this issue, we aim to provide a system that selects music based on individual patient information and evaluates its effectiveness in real time and provides feedback.

[1889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1890] In this invention, the server

[1891] a means for inputting patient information;

[1892] means for learning patient preferences based on input patient information;

[1893] A means of listing suitable music;

[1894] a means for playing the listed music;

[1895] a means for monitoring the patient's response while the music is being played;

[1896] a means for providing feedback on the monitored response;

[1897] A means of analyzing facial expressions in real time; and

[1898] A means of providing feedback using generative AI models; and

[1899] This makes it possible to provide optimal music for individual patients, evaluate its effectiveness in real time, and provide feedback.

[1900] The "means for inputting patient information" is an interface for inputting basic data such as the patient's name, date of birth, and musical preferences.

[1901] The "means for learning patient preferences based on input patient information" is a system that uses an artificial intelligence algorithm to learn a patient's musical preferences based on patient information.

[1902] The "means for listing appropriate music" is a system that suggests music that may be of interest in the form of a list based on learned preferences.

[1903] The "means for playing the listed music" is an audio playback system that actually plays the selected music so that the patient can listen to it.

[1904] The "means for monitoring the patient's reactions while music is being played" is a system that uses cameras and sensors to observe and record the patient's facial expressions and behavior while music is being played.

[1905] The "means for feeding back monitored responses" refers to a feedback mechanism that sends the monitoring results to a server or artificial intelligence system to help select the next piece of music or improve treatment methods.

[1906] The "means for analyzing facial expressions in real time" is a system for analyzing facial expressions from a patient's face photographed in real time using a camera and evaluating the results.

[1907] "Means for providing feedback using a generative AI model" refers to a mechanism for providing optimal feedback based on data analyzed using a generative AI model, thereby improving the system.

[1908] System Program

[1909] The invention can be implemented using a smartphone application with the following functions: The application includes inputting and learning patient information, listing and playing appropriate music, monitoring responses, and providing feedback.

[1910] Program processing

[1911] The application begins with a user entering patient information (such as name, date of birth, and favorite music genre) on a smartphone. This information is then sent to a server and stored in a database. The server then uses an artificial intelligence algorithm to learn the patient's preferences and presents a list of appropriate music based on that information.

[1912] Hardware and Software

[1913] Hardware: Smartphone's built-in camera

[1914] Software: OpenCV (image processing library), Keras (deep learning library), emotion_model.h5 (emotion recognition model)

[1915] Server processing: AI algorithms, databases

[1916] The server uses the generative AI model to generate a list of music based on the newly entered patient information and sends it back to the application, which then displays the list of music suggestions to the user and plays the music selected by the user.

[1917] Specific processing examples

[1918] While music is playing, the smartphone camera monitors the patient's reactions. Using OpenCV for real-time facial analysis, the captured facial image data is fed into an emotion recognition model. The model identifies facial expressions such as smiling or relaxed, and based on that, evaluates whether the currently playing music is appropriate for the patient.

[1919] The monitoring results are fed back to the server and used by the generative AI model to select music for future use and improve the system. For example, if a patient shows many relaxed facial expressions while listening to a particular piece of music, that piece of music will likely be recommended to similar patients.

[1920] Prompt Sentence Examples

[1921] "Capture a photo, analyze the emotion in real time, and display the results."

[1922] "Use an AI model to determine in real time whether an elderly person is relaxed and provide feedback on the results."

[1923] This invention will enhance the effectiveness of music therapy and improve the quality of daily life for elderly people and dementia patients.

[1924] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1925] Step 1:

[1926] The user launches the smartphone application and enters patient information. Specifically, a form is displayed in which the patient can enter their name, date of birth, and favorite music genre. This information is collected as input data.

[1927] Step 2:

[1928] The patient information entered by the user is sent from the terminal to the server, where it is saved in a database. The input data is sent in JSON format and stored in the database. The data processing performed here involves converting the input data into the correct format.

[1929] Step 3:

[1930] Based on the newly saved patient information, the server runs an artificial intelligence algorithm to learn the patient's preferences. The algorithm analyzes the input data and creates a profile of the patient's musical preferences based on the learning database. The output is a list of music genres and artists that are suitable for the patient.

[1931] Step 4:

[1932] The server uses the generative AI model to generate a list of suitable music. Specifically, it calls the API of a music streaming service based on the patient's preference profile to retrieve candidate songs. The output is a list of multiple songs suitable for the patient.

[1933] Step 5:

[1934] The terminal receives the music list sent from the server and displays it to the user. The user selects a song to play from the displayed list. The input here is the music list, and the output is the song selected by the user.

[1935] Step 6:

[1936] The device plays the selected music. During playback, the patient's facial expression is monitored in real time using the smartphone camera. The input data is a facial image captured by the camera, and the OpenCV library is used to detect the facial region and preprocess the image.

[1937] Step 7:

[1938] The device inputs the preprocessed facial image data into the emotion recognition model (emotion_model.h5) and analyzes facial expressions in real time. The output is the detected emotion (e.g., smiling, relaxed, angry, etc.).

[1939] Step 8:

[1940] The monitoring results are fed back to the server, which uses this feedback data to analyze the patient's reactions using a generative AI model to improve future music recommendation algorithms. The input is emotional data, and the output is updates to the learning database.

[1941] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1942] The present invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reaction during playback, a feedback means, and an emotion engine for recognizing the user's emotions.

[1943] User operations

[1944] 1. Install and launch the app

[1945] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1946] 2. Enter patient information

[1947] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1948] Server Processing

[1949] 1. Receiving and storing patient information

[1950] The server receives the patient information sent from the terminal and stores it in a database.

[1951] 2. Launching the AI ​​algorithm

[1952] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1953] 3. Listing Music Candidates

[1954] The server calls the music streaming service API based on the profile, lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop," and generates playlist candidates.

[1955] 4. Submit a playlist

[1956] The server transmits the generated playlist candidates to the terminal.

[1957] Terminal handling

[1958] 1. Receiving and displaying playlists

[1959] The terminal receives the playlist candidates sent from the server and displays them to the user.

[1960] 2. User selection of playlist

[1961] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[1962] 3. Playing music

[1963] The terminal plays the songs from the selected playlist, allowing the patient to listen to music.

[1964] 4. Monitoring and analysis using an emotion engine

[1965] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state. The emotion engine then classifies the monitored emotional data to identify emotional states such as relaxation, joy, and sadness.

[1966] 5. Automatic adjustment

[1967] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state, for example, if the patient is determined to be in a relaxed state, it will adjust the music played to continue playing songs that help maintain that state.

[1968] 6. Submitting Feedback

[1969] The device feeds back the monitoring results to the server, which receives this feedback data and updates the AI ​​model to improve the accuracy of future playlist generation.

[1970] Specific examples

[1971] 1. Enter patient information

[1972] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[1973] 2. Server Processing

[1974] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[1975] 3. Generate a playlist

[1976] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and detect whether they are relaxed.

[1977] 4. The Role of the Emotional Engine

[1978] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[1979] 5. Submitting Feedback

[1980] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[1981] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[1982] The processing flow will be explained below.

[1983] Step 1:

[1984] Installing the app and logging in

[1985] The user downloads the app, installs it on their device, and launches it for the first time. The user enters the required information (username, password, etc.) on the login screen and logs in.

[1986] Step 2:

[1987] Entering patient information

[1988] The user clicks the "Register New Patient" button. Next, the user enters basic information such as the patient's name, date of birth, and favorite music genre, and then presses the "Next" button to proceed to the detailed information entry screen. The user enters additional details (e.g., episodes related to a particular song) and presses the "Save" button to complete the process.

[1989] Step 3:

[1990] Sending patient information

[1991] The terminal sends the entered patient information to the server. The sent data includes basic information and detailed information.

[1992] Step 4:

[1993] Receiving and storing information

[1994] The server receives the patient information sent from the terminal and stores the received information in a database.

[1995] Step 5:

[1996] Launching AI algorithms

[1997] When the server receives new patient information, it triggers an AI algorithm that builds a profile based on the patient's preferences and input episodes.

[1998] Step 6:

[1999] Listing music candidates

[2000] The server calls the music streaming service API based on the profile it has created. It then lists related songs using keywords such as "Misora ​​Hibari" and "Showa pop music." It then generates playlist candidates based on the information about the songs listed.

[2001] Step 7:

[2002] Sending a Playlist

[2003] The server transmits the generated playlist candidates to the terminal.

[2004] Step 8:

[2005] Receiving and displaying playlists

[2006] The terminal receives the playlist candidates sent from the server, and displays the candidate list to the user.

[2007] Step 9:

[2008] User-selected playlist

[2009] The user selects a song to play from the list of candidates. The user presses the "Start Play" button to confirm the selected playlist.

[2010] Step 10:

[2011] Playing music

[2012] The device will play the songs from the selected playlist, allowing the user to listen to music.

[2013] Step 11:

[2014] Emotion engine monitoring and analysis

[2015] During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state, which is then classified as relaxation, joy, etc.

[2016] Step 12:

[2017] automatic adjustment

[2018] The emotion engine automatically adjusts the songs in the current playlist based on the determined emotional state: for example, if the patient is determined to be relaxed, songs that help maintain that state will be played next.

[2019] Step 13:

[2020] Send Feedback

[2021] The device feeds back the monitoring results to the server, which then updates the AI ​​model based on the feedback data to improve the accuracy of future playlist generation.

[2022] Specific examples

[2023] 1. Enter patient information

[2024] The user launches the app and enters the patient's name, "Yamada Taro," their date of birth, "July 15, 1940," and their favorite music, "Showa pop songs." They then add the episode "Misora ​​Hibari's 'Like the Flow of a River.'"

[2025] 2. Server Processing

[2026] The server receives and stores this information, then activates an AI algorithm to create a list of songs related to "Showa pop" and "Misora ​​Hibari," generate playlist candidates, and send them to the device.

[2027] 3. Generate a playlist

[2028] The device displays playlist suggestions to the user, and the user selects "Like the Flow of a River" and "Love Vacation" and begins playback. During playback, the device uses an emotion engine to analyze the patient's facial expressions and voice to determine whether they are relaxed.

[2029] 4. The Role of the Emotional Engine

[2030] If the patient is determined to be relaxed, the emotion engine automatically changes the next song to one with a similar relaxing effect to maintain that state.

[2031] 5. Submitting Feedback

[2032] The monitoring results are sent to a server, which then uses this information to update and improve the AI ​​model.

[2033] This system will maximize the effects of music therapy for dementia patients, helping them relax, stabilize their emotions, and improve their communication. Furthermore, the introduction of an emotion engine will enable the provision of even more highly customized music experiences.

[2034] Example 2

[2035] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2036] The challenge is to provide a system that maximizes the effectiveness of music therapy, which aims to improve the quality of life of dementia patients. In particular, because it is difficult to automatically select and adjust music according to the patient's emotional state, real-time analysis and automatic adjustment technology using an emotion engine are required to solve this problem. Another challenge is to build an efficient system using artificial intelligence algorithms to learn the patient's preferences.

[2037] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2038] In this invention, the server includes a means for inputting patient information, a means for learning the patient's preferences based on the input patient information, a means for listing appropriate music, a means for playing the listed music, a means for monitoring the patient's reaction during music playback, a means for providing feedback on the monitored reaction, a means for using an emotion engine to analyze the user's emotions in real time, and a means for automatically adjusting the music being played based on the analyzed emotional state. This makes it possible to maximize the effectiveness of music therapy for dementia patients and automatically adjust the music experience according to the patient's emotional state.

[2039] "Patient information" refers to basic data related to an individual patient, such as the patient's name, date of birth, and preferred music genre.

[2040] "Preference learning means" refers to the process or algorithm used to learn music that matches a patient's preferences based on information input by the patient.

[2041] "Music listing method" refers to the process of selecting appropriate music based on the patient's preferences and compiling it into a playlist.

[2042] "Means for playing music" refers to the ability to actually play songs from a selected music playlist and have the patient listen to them.

[2043] "Monitoring means" refers to the process and equipment used to monitor and collect data on the patient's responses while the music is playing.

[2044] "Feedback means" refers to the process of analyzing the patient's reaction data obtained through monitoring, feeding the results back into the system, and using them to determine the next music selection and playback method.

[2045] "Emotion engine" refers to software and algorithms that analyze a patient's facial expressions and voice in real time to identify their emotional state.

[2046] "Means for automatically adjusting" means a function for automatically changing the selection of music being played or the playback sequence based on the emotional state identified by the emotional engine.

[2047] This invention provides a music therapy support system aimed at improving the quality of life of dementia patients, and is implemented by combining the following means. Each means will be explained based on the role of the server, terminal, and user.

[2048] Server Roles

[2049] Receiving and storing patient information

[2050] The server receives the patient information sent from the device and stores it in a database, including the patient's name, date of birth, and preferred music genre.

[2051] Launching AI algorithms and building profiles

[2052] When the server receives new patient information, it invokes an AI algorithm using a Python machine learning library (e.g., scikit-learn) to build a profile for that patient based on their preferences and the anecdotes they provide.

[2053] Listing music candidates

[2054] Based on the constructed profile, the server calls the music streaming service API (e.g., Spotify API), lists related songs, and generates playlist suggestions.

[2055] Sending a Playlist

[2056] The generated playlist candidates are sent to the device in JSON format.

[2057] Device Role

[2058] Installing and launching the app

[2059] Users download and install a dedicated app on their smartphone or tablet, enter the required information on the login screen, and log in.

[2060] Entering patient information

[2061] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre, then enters more detailed information such as an episode related to a specific song, and taps the "Save" button to complete the process.

[2062] Receiving and displaying playlists

[2063] The device receives the playlist suggestions sent from the server and displays them to the user. The displayed information includes song titles, artist names, album art, and so on.

[2064] User selects and plays a playlist

[2065] The user selects a song to play from the list of suggestions and taps the "Start Play" button to confirm the playlist. Playback begins and the patient begins listening to the music.

[2066] Monitoring and analysis of patient responses

[2067] Real-time analysis by emotion engine

[2068] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to identify their emotional state. The emotion engine collects data on facial features and tone of voice to classify emotional states such as relaxation, joy, and sadness.

[2069] Auto-tuning and feedback

[2070] automatic adjustment

[2071] The emotion engine automatically adjusts the music being played based on the analysis results. For example, if it determines that the patient is relaxed, it will adjust the music to continue playing to help maintain that state.

[2072] Send Feedback

[2073] The device feeds back the monitoring results to the server in real time, and the server stores the feedback in a database and uses it as training data to help generate the next playlist.

[2074] Specific examples

[2075] Prompts for entering patient information and generating a playlist

[2076] The user starts the app and enters their name ("Yamada Taro"), their date of birth ("July 15, 1940"), their favorite music ("Showa pop"), and an episode related to a specific song by a certain famous artist.

[2077] Based on this information, the server creates a list of songs related to "Showa pop music" and "certain famous artists" and generates a playlist of candidates. The user selects a "specific song name" and "another song name" from the playlist sent to the device and begins playback.

[2078] Prompt Sentence Examples

[2079] "Please generate a relaxing playlist for a male dementia patient over 70 years old who likes Showa-era pop music. For example, please include a specific song by a certain famous artist or a hit song from the Showa era."

[2080] This system can maximize the effectiveness of music therapy for dementia patients and provide a music experience that suits their emotional state. Real-time analysis and automatic adjustment functions based on an emotion engine are expected to further enhance the effect.

[2081] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2082] Step 1:

[2083] Installing and launching the app

[2084] Users download and install a dedicated app on their smartphone or tablet, then launch the app and log in by entering their username and password on the login screen.

[2085] Input: None

[2086] Output: None

[2087] Specific operation: A user downloads an app from the Google Play Store or Apple App Store, launches it after installation, and then logs in by entering the required information on the login screen.

[2088] Step 2:

[2089] Entering patient information

[2090] The user taps the "Register New Patient" button and enters basic information such as the patient's name, date of birth, and favorite music genre. They then press the "Next" button to enter more detailed information, such as an episode related to a specific song, and tap the "Save" button to complete the process.

[2091] Input: Patient information (name, date of birth, music preferences, episodes)

[2092] Output: Patient information is saved on the device

[2093] Specific operation: The user enters the required information into the form, presses the button to proceed to the next screen, and when the series of information input is completed, presses the save button to save the information in the terminal.

[2094] Step 3:

[2095] Transmitting and storing patient information

[2096] The terminal transmits the stored patient information to a server, which receives the information and stores it in a database.

[2097] Input: Patient information stored on the device

[2098] Output: Patient information stored in the server database

[2099] Specific operation: The terminal sends patient information to the server via an HTTP request, and the server receives the information and stores it in the database.

[2100] Step 4:

[2101] Launching AI algorithms and building profiles

[2102] When the server receives new patient information, it activates an AI algorithm that builds a patient profile based on the patient's preferences and anecdotes.

[2103] Input: Patient information stored in the server database

[2104] Output: Patient preference profile

[2105] Specific operation: The server uses a Python machine learning library (e.g., scikit-learn) to generate a patient preference profile.

[2106] Step 5:

[2107] Listing music candidates

[2108] The server's AI model calls the music streaming service API based on the constructed profile, lists appropriate songs, and generates playlist suggestions.

[2109] Input: Patient Preference Profile

[2110] Output: Playlist suggestions

[2111] Specific operation: The server calls the Spotify API, searches for songs using keywords such as "Showa pop music," and compiles the generated song list in JSON format as playlist candidates.

[2112] Step 6:

[2113] Sending a Playlist

[2114] The server transmits the generated playlist candidates to the terminal.

[2115] Input: Playlist suggestions

[2116] Output: Playlist suggestions sent to device

[2117] Specific operation: The server sends playlist candidates to the terminal as an HTTP response.

[2118] Step 7:

[2119] Receiving and displaying playlists

[2120] The terminal receives the playlist candidates sent from the server and displays them to the user.

[2121] Input: Playlist suggestions from the server

[2122] Output: Playlist suggestions displayed to the user

[2123] Specific operation: The device displays the received playlist candidates to the user on a GUI, where the user can check information such as song title, artist name, and album art.

[2124] Step 8:

[2125] User selects and plays a playlist

[2126] The user selects songs to play from the displayed playlist candidates and taps the "Start Playing" button to confirm the playlist.

[2127] Input: Song selected by user

[2128] Output: Song that started playing

[2129] Specific behavior: The user selects a song from the list and presses the play button to play the music. The music will be played through the device's speakers or headphones.

[2130] Step 9:

[2131] Emotion engine monitoring and analysis

[2132] While music is playing, the device uses an emotion engine to analyze the patient's facial expressions and voice in real time to determine their emotional state.

[2133] Input: facial expression data and voice data of the patient being played back

[2134] Output: Patient's emotional state (relaxed, happy, sad, etc.)

[2135] Specific operation: The device uses the built-in camera and microphone to collect facial expression and voice data, which the emotion engine analyzes to identify the user's emotional state.

[2136] Step 10:

[2137] automatic adjustment

[2138] The emotion engine automatically adjusts songs in the currently playing playlist based on the determined emotional state.

[2139] Input: Emotional state determined by the emotion engine

[2140] Output: Auto-tuned music playback sequence

[2141] Specific operation: The emotion engine adjusts the selection of the next song and the playback order, for example by changing the currently playing song to one with a relaxing effect.

[2142] Step 11:

[2143] Send Feedback

[2144] The device sends the monitoring results as feedback to the server, which receives them and stores them in a database to help improve the AI ​​model.

[2145] Input: Monitoring results

[2146] Output: The AI ​​model on the server is updated.

[2147] Specific operation: The server receives the monitoring results sent from the device and stores them in a database, which improves the accuracy of the next playlist generation.

[2148] (Application example 2)

[2149] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2150] Current music therapy systems have difficulty selecting appropriate music based on the individual preferences of dementia patients. Furthermore, there is no way to monitor the impact of the music being played on the patient's emotional state in real time and automatically adjust the music selection based on that information. This makes it difficult to maximize the effectiveness of music therapy.

[2151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2152] In this invention, the server includes means for inputting patient information, means for learning the patient's preferences based on the input patient information, means for listing appropriate music, means for playing the listed music, means for monitoring the patient's reaction during music playback, means for feeding back the monitored reaction, means for automatically adjusting the music based on the reaction data, and means for analyzing emotions in real time using a generative AI model, thereby making it possible to select music that matches the patient's preferences and automatically adjust the music in real time according to the patient's emotional state.

[2153] "Means for inputting patient information" refers to an interface that allows a user to input information such as the patient's name, date of birth, and preferred music genre.

[2154] "Means for learning patient preferences" refers to an artificial intelligence algorithm that learns the patient's preferred music and episodes based on input patient information.

[2155] "Means for listing appropriate music" refers to a mechanism for listing music suitable for a patient based on learned preferences and generating a playlist.

[2156] "Means for playing the listed music" refers to a function for playing the selected music using an audio device.

[2157] "Means for monitoring patient reactions while music is being played" refers to a system for monitoring the patient's facial expressions, voice, heart rate, etc. in real time while music is being played and analyzing their emotional state.

[2158] "Means for providing feedback on monitored responses" refers to a feedback function that records the monitored emotional state of the patient in a database and uses this information to improve the overall operation of the system and music selection.

[2159] "Means for automatically adjusting music based on reaction data" refers to a mechanism for changing and adjusting currently playing music in real time based on monitored emotional data.

[2160] "Generative AI model" refers to a trained artificial intelligence model used to make music recommendations and reaction analysis based on input data.

[2161] "Real-time emotion analysis means" refers to a system that analyzes a patient's facial expressions, voice, and heart rate in real time to identify their current emotional state.

[2162] This invention is a music therapy support system aimed at improving the quality of life of dementia patients. This system includes a means for inputting patient information, a means for learning the patient's preferences, a means for listing appropriate music, a means for playing music, a means for monitoring the patient's reactions during playback, a means for providing feedback, and a generative AI model for analyzing emotions in real time.

[2163] Server Processing

[2164] The server receives patient information and stores it in a database. When new patient information is received, the server activates an artificial intelligence (AI) algorithm to build a profile based on the patient's preferences and input episodes. Based on the profiled data, the server calls a music streaming service API to list appropriate songs. Finally, the generated playlist suggestions are sent to the user's device.

[2165] Terminal handling

[2166] The device displays the playlist candidates received from the server to the user. The user selects songs to play from the displayed playlist candidates. While the music is playing, the device uses a generative AI model to analyze the patient's facial expressions, voice, heart rate, etc. in real time to determine their emotional state. Based on the emotional state, the device automatically adjusts the current playlist to play songs that will help the patient maintain a relaxed state. The monitored emotional data is fed back to the server and used to improve the accuracy of the next playlist generation.

[2167] Hardware and Software Used

[2168] Hardware: A webcam and heart rate sensor are used to collect patient biometric information, as well as speakers and a headset for playing music.

[2169] Software: Python, OpenCV, TensorFlow, and DeepFace libraries are used. Data processing and AI operations are performed using Python programs, and face recognition and emotion analysis are performed using OpenCV and DeepFace libraries.

[2170] Specific examples

[2171] The server receives the patient's name ("Yamada Taro"), date of birth (July 15, 1940), and favorite music genre ("Showa pop"), and generates playlist candidates based on this. The device displays the playlist candidates, and songs like "Like the Flow of a River" and "Love Vacation" are selected. During playback, the generative AI model analyzes the patient's facial expressions and voice to detect whether they are relaxed. If it is determined that they are relaxed, it automatically changes to music with a similar effect.

[2172] Prompt Sentence Examples

[2173] Taro Yamada's favorite music genre is "Showa pop," and there is an episode related to a specific song, "Like the Flow of a River." Based on this, create an AI model that can recommend appropriate music. Also, include a system that monitors the subject's emotional state in real time and automatically adjusts the music list.

[2174] As described above, the present invention improves the quality of life of dementia patients by recommending music based on the patient's individual preferences and analyzing and adjusting their emotional state in real time.

[2175] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2176] Step 1:

[2177] The user installs the app on their smartphone or device and launches it. The user enters the required information (e.g., username and password) on the login screen and logs in.

[2178] Input: User login information

[2179] Output: Login successful message, main menu displayed

[2180] Specific operation: When the user enters login information and presses the send button, the device sends an authentication request to the authentication server, and if authentication is successful, the main menu is displayed.

[2181] Step 2:

[2182] The user enters new patient information by clicking the "Register New Patient" button, entering basic information such as the patient's name, date of birth, and favorite music genre, and then clicking the "Next" button to proceed to the detailed information entry screen.

[2183] Input: Patient's name, date of birth, favorite music genre, detailed episode

[2184] Output: Patient information data

[2185] Specific operation: When the user enters patient information and presses the save button, the terminal sends and saves this information to the database.

[2186] Step 3:

[2187] The server receives the patient information sent from the device and stores it in a database. The reception of new patient information triggers the AI ​​algorithm.

[2188] Input: Patient information data

[2189] Output: AI algorithm execution flag, patient profile

[2190] How it works: When the server receives patient information, it writes it into a database and then runs an AI algorithm to generate a profile based on the patient's preferences.

[2191] Step 4:

[2192] Based on the generated patient profile, the server calls a music streaming service API to list relevant songs.

[2193] Input: Patient Profile

[2194] Output: Playlist suggestions

[2195] How it works: The AI ​​algorithm analyzes profile data, uses music streaming service APIs to collect related songs, and generates a list of playlist suggestions.

[2196] Step 5:

[2197] The server transmits the generated playlist candidates to the terminal, which receives them and displays them to the user.

[2198] Input: Playlist suggestions

[2199] Output: Playlist suggestions displayed on the device

[2200] Specific operation: The server transmits the playlist data to the terminal, and the terminal displays the received data on the user interface.

[2201] Step 6:

[2202] The user selects songs to play from the displayed playlist candidates, and presses the "Start Play" button to confirm the selected playlist.

[2203] Input: User song selection

[2204] Output: Playback start signal, selected playlist

[2205] Specific operation: When the user selects a song and presses the start playback button, the data of the selected playlist is sent to the playback engine.

[2206] Step 7:

[2207] The device plays songs from the selected playlist, while the patient's facial expressions, voice, and heart rate are analyzed in real time using the HMD, camera, and heart rate sensor.

[2208] Input: Selected playlist, patient vitals

[2209] Output: Playing music, real-time analysis data

[2210] Specific operation: The device starts the playback engine and begins playing the music, and at the same time, it acquires data using a biometric information collection device and has it analyzed by the generative AI model.

[2211] Step 8:

[2212] Automatically adjusts songs in the current playlist based on real-time analyzed emotional data.

[2213] Input: Parsed emotion data

[2214] Output: Adjusted song list, next song to play

[2215] How it works: The generative AI model evaluates the currently playing song based on emotional data and automatically changes or adjusts the next song if necessary.

[2216] Step 9:

[2217] The monitoring results are fed back to the server to improve the accuracy of future playlist generation.

[2218] Input: Real-time analysis data and adjustment data

[2219] Output: Updated AI model, feedback data saved

[2220] Specific operation: The device sends emotion analysis data and adjustment data to the server, which stores it in a database and uses it to update the AI ​​model.

[2221] Through these steps, this music therapy support system provides music based on the individual preferences of dementia patients and analyzes and adjusts their emotions in real time, thereby improving their quality of life.

[2222] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2223] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2224] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2225] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2226] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2227] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2228] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2229] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2230] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2231] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2232] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2233] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2234] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2235] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2236] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2237] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2238] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2239] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2240] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2241] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2242] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2243] The following is further disclosed regarding the above embodiment.

[2244] (Claim 1)

[2245] a means for inputting patient information;

[2246] means for learning patient preferences based on input patient information;

[2247] A means of listing suitable music;

[2248] a means for playing the listed music;

[2249] a means for monitoring the patient's response while the music is being played;

[2250] a means for providing feedback on the monitored response;

[2251] A system including:

[2252] (Claim 2)

[2253] 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

[2254] (Claim 3)

[2255] 10. The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm.

[2256] "Example 1"

[2257] (Claim 1)

[2258] a means for inputting patient information;

[2259] means for learning patient preferences based on input patient information;

[2260] A means of listing suitable music;

[2261] a means for playing the listed music;

[2262] a means for monitoring the patient's response while the music is being played;

[2263] a means for providing feedback on the monitored response;

[2264] A means for storing the generated playlists and associated data for use in future improvements to the artificial intelligence model; and

[2265] A system including:

[2266] (Claim 2)

[2267] 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

[2268] (Claim 3)

[2269] The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm to build a profile based on newly entered patient information and existing data and to create a list of music candidates.

[2270] "Application Example 1"

[2271] (Claim 1)

[2272] a means for inputting patient information;

[2273] means for learning patient preferences based on input patient information;

[2274] A means of listing suitable music;

[2275] a means for playing the listed music;

[2276] a means for monitoring the patient's response while the music is being played;

[2277] a means for providing feedback on the monitored response;

[2278] A means of analyzing facial expressions in real time; and

[2279] A means of providing feedback using generative AI models; and

[2280] A system including:

[2281] (Claim 2)

[2282] 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

[2283] (Claim 3)

[2284] 2. The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm and a generative AI model for analyzing facial expressions in real time.

[2285] "Example 2: Combining Emotion Engines"

[2286] (Claim 1)

[2287] a means for inputting patient information;

[2288] means for learning patient preferences based on input patient information;

[2289] A means of listing suitable music;

[2290] a means for playing the listed music;

[2291] a means for monitoring the patient's response while the music is being played;

[2292] a means for providing feedback on the monitored response;

[2293] A means using an emotion engine that analyzes user emotions in real time;

[2294] means for automatically adjusting the music being played based on the analyzed emotional state;

[2295] A system including:

[2296] (Claim 2)

[2297] 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

[2298] (Claim 3)

[2299] 10. The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm.

[2300] "Application example 2 when combining emotion engines"

[2301] (Claim 1)

[2302] a means for inputting patient information;

[2303] means for learning patient preferences based on input patient information;

[2304] A means of listing suitable music;

[2305] a means for playing the listed music;

[2306] a means for monitoring the patient's response while the music is being played;

[2307] a means for providing feedback on the monitored response;

[2308] A means for automatically adjusting the music according to the response data;

[2309] A means to analyze emotions in real time using generative AI models;

[2310] A system including:

[2311] (Claim 2)

[2312] 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

[2313] (Claim 3)

[2314] 10. The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm.

[2315] {keywords} [Explanation of symbols]

[2316] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting patient information; means for learning patient preferences based on input patient information; A means of listing suitable music; a means for playing the listed music; a means for monitoring the patient's response while the music is being played; a means for providing feedback on the monitored response; A system including:

2. 2. The system of claim 1, wherein the patient information includes name, date of birth, and preferred music genre.

3. 10. The system of claim 1, wherein the means for learning preferences utilizes an artificial intelligence algorithm.

Images