system

Abstract

A system is provided.SOLUTION: The system includes a means for collecting video data, a means for collecting audio data, a means for analyzing the collected video data and audio data, a means for generating an optimum BGM or noise based on data obtained from the analyzing means, a means for reproducing the generated BGM or noise, a means for inputting subjective evaluation by a user, and a means for optimizing the generating means based on the subjective evaluation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] ---

[0005] In modern society, ensuring quality sleep is an important issue for many people. In particular, with the increasing incidence of sleep disorders and stressful daily lives, effective methods for inducing sleep are needed. However, commercially available background music and noise for sleep induction are provided to the general public and are not optimized for individual sleep patterns or preferences. As a result, the effectiveness of these commercial products is limited, and actual evaluation is difficult. Furthermore, while customization based on individual preferences is desired, the technology to achieve this is insufficient. Therefore, the present invention aims to solve this problem by developing a system that provides an optimal sleep environment based on each individual's sleep data. [Means for solving the problem]

[0006] The present invention provides a system including a means for collecting video data, a means for collecting audio data, a means for analyzing the collected video data and audio data, a means for generating optimal background music or noise based on the data obtained from the analysis means, a means for playing the generated background music or noise, a means for a user to input a subjective evaluation, and a means for optimizing the generation means based on the subjective evaluation. In particular, the system further includes a means for transmitting video data and audio data, which transmits detailed data during sleep to a server, and a generation AI generates optimal background music or noise based on the analysis results. Furthermore, by providing a means for adjusting the parameters of the generation AI based on the user's subjective evaluation, the next music or noise can be further optimized for each individual user. In this way, the present invention can provide an optimal sleeping environment tailored to each individual's sleep pattern and improve the quality of their sleep.

[0007] ---

[0008] "Video data" refers to video information acquired by a visual sensor such as a camera.

[0009] "Audio data" is sound information acquired by an audio sensor such as a microphone.

[0010] The "analysis means" is a program or device for extracting and analyzing useful information from the collected video data and audio data.

[0011] The "generation means" is a program or device for generating optimal background music or noise based on the data obtained from the analysis means.

[0012] The "playback means" is a program or device that allows the user to hear the generated background music or noise.

[0013] "Subjective evaluation" is feedback information in which the user evaluates the quality of sleep and satisfaction with the music.

[0014] The "optimization means" is a program or device that adjusts the parameters of the generation means based on subjective evaluation to better suit the next background music or noise to the individual user.

[0015] ---

[0016] The above are definitions of important words included in the claims. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] ---

[0039] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0040] Data collection

[0041] Terminal

[0042] Before going to bed, the user launches the app and sets it to sleep mode.

[0043] The device's camera and microphone are used to collect video and audio data while you sleep.

[0044] The collected data is sent to a server at regular intervals.

[0045] Specific examples

[0046] 1. The user opens the app and selects sleep mode.

[0047] 2. The device activates the camera and microphone and begins collecting video and audio data.

[0048] 3. The collected data is sent to the server.

[0049] Data analysis

[0050] server

[0051] The server analyzes the received video and audio data.

[0052] The number of times the patient turns over and changes in position are extracted from the video data.

[0053] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[0054] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[0055] The analysis results are stored in a database.

[0056] Specific examples

[0057] 1. The server analyzes the video data and counts the number of times the person turns over.

[0058] 2. The server analyzes the audio data and detects the frequency of snoring.

[0059] 3. The server evaluates the sleep depth and stores it in a database.

[0060] BGM / noise generation and playback

[0061] server

[0062] The server uses a generative AI to generate optimal background music or noise based on the analysis results.

[0063] The generated sound data is transmitted to the terminal.

[0064] Terminal

[0065] The terminal plays the background music or noise received from the server.

[0066] Specific examples

[0067] 1. The server inputs the analysis results into the generation AI, which generates background music or noise.

[0068] 2. The generated sound data is sent to the device.

[0069] 3. The device plays the received sound data.

[0070] User Feedback

[0071] User

[0072] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[0073] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[0074] Terminal

[0075] The terminal transmits the subjective evaluation input by the user to the server.

[0076] Specific examples

[0077] 1. After waking up, the user opens the app and rates the quality of their sleep.

[0078] 2. The user enters a rating, and the device sends it to the server.

[0079] Next optimization

[0080] server

[0081] The server analyzes the received subjective evaluations and adjusts the parameters of the generation AI for generating the next background music or noise.

[0082] This allows the next sleep environment to be more tailored to the user.

[0083] Specific examples

[0084] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[0085] 2. Based on the analysis, generate optimized background music or noise for the next use.

[0086] ---

[0087] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[0088] The processing flow will be explained below.

[0089] ---

[0090] Step 1:

[0091] Terminal

[0092] The user launches the app and sets it to sleep mode.

[0093] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[0094] Step 2:

[0095] Terminal

[0096] The collected video and audio data is sent to the server at regular intervals.

[0097] Step 3:

[0098] server

[0099] The server receives the video data and audio data transmitted from the terminal.

[0100] Step 4:

[0101] server

[0102] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[0103] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[0104] Step 5:

[0105] server

[0106] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[0107] The analysis results are saved in a database.

[0108] Step 6:

[0109] server

[0110] The analysis results are input into a generation AI to generate the optimal background music or noise.

[0111] Step 7:

[0112] server

[0113] The generated sound data is sent to the terminal.

[0114] Step 8:

[0115] Terminal

[0116] The device plays background music or noise received from the server.

[0117] Step 9:

[0118] User

[0119] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[0120] Step 10:

[0121] Terminal

[0122] The subjective evaluation input by the user is sent to the server.

[0123] Step 11:

[0124] server

[0125] The server analyzes the received subjective evaluations.

[0126] Based on the feedback, adjust the parameters of the generated AI.

[0127] Step 12:

[0128] server

[0129] The next background music or noise is generated based on the adjusted parameters.

[0130] ---

[0131] The above is the specific flow of the program's processing, which allows the user to be provided with an optimal sleeping environment.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional sleep environment systems have struggled to provide optimal solutions for a user's individual sleep patterns or specific sleep problems. In particular, when it comes to generating background music or noise to improve sleep quality, they tend to use generic preset music or noises, and are unable to optimize them to meet the user's specific needs. Furthermore, there has been insufficient effort to effectively utilize subjective feedback from users to further optimize the next background music or noise.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes a means for periodically transmitting video and audio data, a means for analyzing the collected video and audio data to extract changes in the user's body position and audio events, and a means for evaluating the depth of sleep. This allows for a detailed analysis of the user's specific sleep patterns and problems, and based on that, optimal background music and noise can be generated. Furthermore, by adjusting the parameters of the generation AI model based on the user's subjective evaluation, the playback environment can be further optimized for subsequent playbacks.

[0137] "Video data" refers to all visual information acquired using a camera or other imaging device.

[0138] "Audio data" refers to all information relating to sound captured using a recording device such as a microphone.

[0139] "Transmitting means" refers to the technical methods or devices used to transmit collected data to another device or server at specified intervals.

[0140] "Means of analysis" refers to the programs or algorithms that process the collected data and extract useful information.

[0141] "Changes in body position" refers to changes in the posture or position of the user's body.

[0142] "Sleep depth" is an indicator of how deep a user's sleep state is, and is usually evaluated on a three-level scale: "light," "intermediate," and "deep."

[0143] "Generative AI model" refers to a general algorithm that uses artificial intelligence to generate new data or content.

[0144] "BGM" is short for background music and refers to music played to promote a particular atmosphere or emotion.

[0145] "Noise" refers to random or controlled sounds reproduced for a specific purpose.

[0146] "Subjective evaluation" refers to a personal evaluation made by a user based on their own experience and feelings.

[0147] "Parameters" refer to settings that adjust the behavior of a particular algorithm or model.

[0148] "Optimization" refers to the process of adjusting parameters or settings to achieve the most effective state or result for a particular purpose.

[0149] "Means for playback" refers to the technical methods and devices for actually outputting the generated sound data as sound.

[0150] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0151] Data collection

[0152] Terminal

[0153] Before going to bed, the user launches the app and sets it to sleep mode.

[0154] The device is equipped with a camera and microphone, which are used to collect video and audio data while the person is sleeping.

[0155] The collected data is sent to the server at regular intervals. For example, the data is stored in a buffer every hour, and when a certain amount of data has accumulated, it is sent.

[0156] Specific examples

[0157] 1. The user opens the app before going to bed and selects sleep mode.

[0158] 2. The device activates its camera and microphone. The camera captures the user's posture and facial orientation, while the microphone records audio data such as snoring, talking in your sleep, and teeth grinding.

[0159] 3. The collected data is stored in a buffer at regular intervals, and the accumulated data is sent to the server.

[0160] Data analysis

[0161] server

[0162] The received video and audio data is analyzed using an advanced data analysis algorithm installed on the server, which is used to extract changes in the user's body position and audio events.

[0163] The video data is used to count the number of times the patient turns over and changes in position, while the audio data is used to detect the frequency of snoring, talking in one's sleep, and teeth grinding.

[0164] The depth of sleep is assessed on a three-point scale: light, medium, or deep. This assessment is based on the extracted data.

[0165] Specific examples

[0166] 1. The server analyzes the received video data and counts the number of times the person turns over.

[0167] 2. The server analyzes the audio data and detects the frequency of snoring and the number of times the person talks in their sleep.

[0168] 3. Based on this information, the depth of sleep is evaluated and stored in a database.

[0169] BGM / Noise Generation

[0170] server

[0171] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4 (registered trademark)) to generate optimal background music or noise.

[0172] The generated sound data is sent to the terminal.

[0173] Specific examples

[0174] 1. The server inputs the following prompt to the generative AI model:

[0175] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[0176] 2. The generated sound data is sent to the device.

[0177] BGM / Noise playback

[0178] Terminal

[0179] The terminal plays the background music or noise received from the server.

[0180] The received sound data is played back at the appropriate time to optimize the user's sleep environment.

[0181] Specific examples

[0182] 1. The terminal prepares to play the sound data received from the server.

[0183] 2. Based on the collected data, background music or noise is played at the optimal timing.

[0184] Collecting user feedback

[0185] User

[0186] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[0187] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[0188] Terminal

[0189] The terminal transmits the subjective evaluation input by the user to the server.

[0190] Specific examples

[0191] 1. After waking up, the user opens the app and rates the quality of their sleep.

[0192] 2. The evaluation items are entered and the terminal sends them to the server.

[0193] Next optimization

[0194] server

[0195] The received subjective evaluations are analyzed and the parameters of the generative AI model for generating the next background music or noise are adjusted.

[0196] This allows background music or noise that is more suited to the user to be provided the next time the device is used.

[0197] Specific examples

[0198] 1. The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[0199] 2. Update the prompt text to generate optimized background music or noise the next time you use it.

[0200] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user in detail, it is possible to provide an individually optimized sleep environment.

[0201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0202] Step 1: Launch and configure the app

[0203] Terminal

[0204] The user launches the app by tapping the app icon on their device.

[0205] Select "Sleep Mode" from the in-app menu.

[0206] If a popup appears requesting permission to use the camera and microphone, the user selects Allow.

[0207] Input and Output

[0208] Input: User touch actions (tapping app icons, selecting sleep mode)

[0209] Output: Camera and microphone are activated, and sleep data collection is complete.

[0210] Specific actions

[0211] 1. When a user opens the app and selects sleep mode, the device activates the camera and microphone.

[0212] 2. The camera monitors the user's position and the microphone begins collecting sound from within the room.

[0213] Step 2: Data collection

[0214] Terminal

[0215] The device's camera captures the user's posture and facial orientation, while the microphone records snoring, talking in one's sleep, and environmental sounds.

[0216] The collected data is temporarily stored in a buffer.

[0217] Input and Output

[0218] Input: Video and audio data from the camera and microphone

[0219] Output: Buffered video and audio data

[0220] Specific actions

[0221] 1. The camera captures multiple frames of video per second, and the microphone records audio at regular intervals.

[0222] 2. The collected video and audio data is stored in a buffer.

[0223] Step 3: Sending data

[0224] Terminal

[0225] The data stored in the buffer is sent to the server at regular intervals.

[0226] Input and Output

[0227] Input: Video and audio data stored in the buffer

[0228] Output: Video and audio data sent to the server

[0229] Specific actions

[0230] 1. When a certain amount of data accumulates in the buffer, the terminal compresses the data and transmits it.

[0231] 2. The server receives the data.

[0232] Step 4: Analyze the data

[0233] server

[0234] The received video and audio data is analyzed. The number of times the patient turns over in their sleep and changes in posture are extracted from the video data, and the frequency of snoring, talking in their sleep, and teeth grinding is detected from the audio data.

[0235] Sleep depth is evaluated on three levels: "light," "intermediate," and "deep."

[0236] Input and Output

[0237] Input: Video and audio data sent to the server

[0238] Output: Extracted body position change data, audio event data, sleep depth assessment

[0239] Specific actions

[0240] 1. The server analyzes the video data frame by frame and counts the user's changes in position and the number of times they turn over in bed.

[0241] 2. Analyze audio data to detect the frequency of snoring, sleep talking, and teeth grinding.

[0242] 3. Based on the extracted data, sleep depth is evaluated on a three-point scale and the results are stored in a database.

[0243] Step 5: Generate background music / noise

[0244] server

[0245] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[0246] The generated sound data is sent to the terminal.

[0247] Input and Output

[0248] Input: Analysis result data, prompt statement

[0249] Output: Generated background music or noise

[0250] Specific actions

[0251] 1. The server inputs the following prompt to the generative AI model:

[0252] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[0253] 2. The generative AI model generates optimal sound data and sends it to the device.

[0254] Step 6: Play background music / noise

[0255] Terminal

[0256] The device plays background music or noise received from the server.

[0257] Input and Output

[0258] Input: BGM or noise sent from the server

[0259] Output: BGM or noise to be played

[0260] Specific actions

[0261] 1. The terminal prepares to play the sound data received from the server.

[0262] 2. Play background music or noise at specified times to optimize the user's sleep environment.

[0263] Step 7: Gather user feedback

[0264] User

[0265] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[0266] Input and Output

[0267] Input: User subjective evaluation data

[0268] Output: Subjective evaluation data sent to the server

[0269] Specific actions

[0270] 1. The user opens the app and enters information about their sleep quality (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[0271] 2. The device sends the entered evaluation data to the server.

[0272] Step 8: Next optimization

[0273] server

[0274] The received subjective evaluations are analyzed and the parameters of the generative AI model are adjusted.

[0275] Input and Output

[0276] Input: User subjective evaluation data

[0277] Output: Adjusted generative AI model parameters

[0278] Specific actions

[0279] 1. The server analyzes the received subjective evaluation data and identifies which elements were problematic.

[0280] 2. Adjust the parameters of the generative AI model for the next BGM or noise generation.

[0281] The above is the specific flow of the program processing of this system. Through each processing step, it is possible to effectively collect and analyze the user's sleep data and provide an individually optimized sleep environment.

[0282] (Application example 1)

[0283] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Conventional sleep improvement systems only collect users' sleep data and rarely provide personalized, optimized responses based on that data. Furthermore, they lack real-time feedback and improvement measures based on analyzed data, which means users' sleep quality is not fully improved. Furthermore, there are insufficient means for effectively utilizing subjective feedback.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0286] In this invention, the server includes means for collecting user sleep data, means for transmitting the collected video data and audio data, means for analyzing the transmitted data, a generative AI model that generates optimal background music or noise based on the data obtained from the analysis means, means for playing the generated background music or noise, means for the user to input a subjective evaluation of sleep quality, and means for adjusting the parameters of the generative AI model based on the subjective evaluation. This provides an optimal sleep environment tailored to each user's individual sleep pattern, and enables further optimization based on feedback.

[0287] "User's sleep data" refers to information including video data and audio data that indicate the user's sleep state.

[0288] "Video data" refers to video data that records the user's sleeping posture, the number of times they turn over in their sleep, and so on.

[0289] "Voice data" refers to audio data including the user's snoring, talking in their sleep, environmental sounds, and the like.

[0290] "Transmitting means" refers to the functions and processes for transferring collected video and audio data to a server.

[0291] "Means of analysis" refers to the process by which the server handles video and audio data and extracts and analyzes the number of times the patient turns over in bed and the frequency of snoring, etc.

[0292] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal background music or noise based on analytical results and subjective evaluation.

[0293] "BGM or Noise" refers to sound data such as music or environmental sounds created by a generative AI model that are played to improve a user's sleep.

[0294] "Means for playback" refers to the functions or devices for playing background music or noise generated on a device such as a smartphone.

[0295] "Subjective evaluation" refers to the user inputting their impression of the quality of sleep as a numerical value or a comment.

[0296] "Means for adjusting the parameters of the generative AI model" refers to the process of changing the settings of the generative AI model based on the user's subjective evaluation and reflecting this in the next background music or noise generation.

[0297] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0298] 1. Data Collection

[0299] Terminal

[0300] Before going to bed, the user launches the app and sets it to sleep mode.

[0301] The device's camera and microphone are used to collect video and audio data while you sleep.

[0302] The collected data is sent to a server at regular intervals.

[0303] Specific examples

[0304] The user opens the app and selects sleep mode.

[0305] The device will activate its camera and microphone and begin collecting video and audio data.

[0306] The collected data is sent to a server.

[0307] 2. Data Analysis

[0308] server

[0309] The server analyzes the received video and audio data.

[0310] The number of times the patient turns over and changes in position are extracted from the video data.

[0311] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[0312] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[0313] The analysis results are stored in a database.

[0314] Specific examples

[0315] The server analyzes the video data and counts the number of times the person turns over.

[0316] The server analyzes the audio data and detects the frequency of snoring.

[0317] The server evaluates the depth of sleep and stores it in a database.

[0318] 3. BGM / Noise Generation and Playback

[0319] server

[0320] The server uses a generative AI model to generate optimal background music or noise based on the analysis results.

[0321] The generated sound data is transmitted to the terminal.

[0322] Terminal

[0323] The terminal plays the background music or noise received from the server.

[0324] Specific examples

[0325] The server inputs the analysis results into a generative AI model to generate background music or noise.

[0326] The generated sound data is transmitted to the terminal.

[0327] The terminal plays back the received sound data.

[0328] 4. User Feedback

[0329] User

[0330] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[0331] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[0332] Terminal

[0333] The terminal transmits the subjective evaluation input by the user to the server.

[0334] Specific examples

[0335] After waking up, the user opens the app and rates the quality of their sleep.

[0336] The user enters a rating, which the terminal sends to the server.

[0337] 5. Next optimization

[0338] server

[0339] The server analyzes the received subjective evaluations and adjusts the parameters of the generative AI model for generating the next background music or noise.

[0340] This allows the next sleep environment to be more tailored to the user.

[0341] Specific examples

[0342] The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[0343] Based on the analysis, optimized background music or noise will be generated the next time you use it.

[0344] Hardware and software used

[0345] Hardware: Smartphone (camera, microphone, speaker)

[0346] Software: Python, OpenCV (cv2), Request, SoundDevice (sd), server backend (Flask or Django)

[0347] Data processing and calculation

[0348] 1. Data Collection:

[0349] Video and audio data is collected using the smartphone's camera and microphone.

[0350] The data is uploaded to the server as image or audio files at regular intervals.

[0351] 2. Data Analysis:

[0352] The server analyzes the received data and detects things like turning over in bed and snoring.

[0353] The data is stored in a database for evaluating the depth of sleep.

[0354] 3. BGM / Noise Generation and Playback:

[0355] The generative AI model is run on the server to generate optimal background music and noise.

[0356] The sound data is sent to a smartphone and played at the specified time.

[0357] 4. User Feedback:

[0358] Collect user evaluation data from the smartphone interface.

[0359] The feedback data is sent to the server and used for the next BGM generation.

[0360] Prompt Sentence Examples

[0361] Below are some examples of prompts to input to the generative AI model.

[0362] We analyzed the user's sleep data. The frequency of snoring was high, and the average number of turns was three. The user's sleep depth was shallow, so please generate alpha wave music that has a relaxing effect.

[0363] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[0364] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0365] Step 1:

[0366] Before going to bed, the user launches the app and sets it to sleep mode.

[0367] Input: User launches the app and selects sleep mode.

[0368] Output: The device is ready to start collecting data.

[0369] What happens: The user launches the app on their smartphone and selects "sleep mode," which causes the app to turn on the camera and microphone and prepare for data collection.

[0370] Step 2:

[0371] The device's camera and microphone are used to collect video and audio data while you sleep.

[0372] Input: The app activates the camera and microphone.

[0373] Output: Collected video and audio data.

[0374] How it works: The device's camera detects the user's posture and tossing and turning while sleeping, and the microphone detects snoring and talking in their sleep. This data is temporarily saved as image and audio files.

[0375] Step 3:

[0376] The collected data is sent to the server at regular intervals.

[0377] Input: Collected video and audio data.

[0378] Output: Data uploaded to the server.

[0379] How it works: Every 60 seconds, the device uploads the collected data to the server. Data transfer is performed via an internet connection and an API request.

[0380] Step 4:

[0381] The server parses the received data.

[0382] Input: Video and audio data uploaded to the server.

[0383] Output: Analysis results such as number of turns, frequency of snoring, and sleep depth.

[0384] How it works: The server analyzes the video data and counts the number of times the person turns over in bed and changes their position. Similarly, it extracts the frequency of snoring and talking in their sleep from the audio data. Based on this, it evaluates the depth of sleep and stores the results in a database.

[0385] Step 5:

[0386] Uses generative AI models to generate optimal background music or noise based on the analysis results.

[0387] Input: Analysis results (number of turns, frequency of snoring, depth of sleep).

[0388] Output: Generated background music or noise sound data.

[0389] Specific operation: The server inputs the analysis results into the generative AI model and generates a prompt. For example, it might input, "We analyzed the user's sleep data. We found that the user snores frequently and turns over an average of three times. The user's sleep depth is shallow, so please generate alpha wave music that has a relaxing effect." The generative AI model then generates music data based on the instructions.

[0390] Step 6:

[0391] The generated sound data is sent to the terminal and played back.

[0392] Input: Sound data for the generated background music or noise.

[0393] Output: Sound data played on the device.

[0394] Specific operation: The server sends the generated sound data to the device, which then plays the received sound data in real time to help the user sleep.

[0395] Step 7:

[0396] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[0397] Input: User's subjective evaluation (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[0398] Output: Subjective evaluation data sent to the server.

[0399] How it works: The user opens the app and inputs their sleep quality rating, including how easily they fell asleep, whether they woke up during the night, and how easily they woke up, using numerical values ​​and comments. The device then sends this data to the server.

[0400] Step 8:

[0401] The server analyzes the subjective evaluations it receives and adjusts the parameters of the generative AI model.

[0402] Input: Subjective evaluation data sent to the server.

[0403] Output: The adjusted parameters of the generative AI model.

[0404] How it works: The server analyzes the user's subjective evaluation data and adjusts the parameters of the generative AI model, which then generates more appropriate background music or noise for the next session, improving the user's sleep quality.

[0405] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0406] ---

[0407] This invention is a system that collects and analyzes a user's sleep data and combines it with an emotion engine that recognizes the user's emotions to generate optimal sleep-inducing background music and noises. It mainly functions in cooperation with the terminal, server, and user. The specific operation of each element is explained below.

[0408] Data collection

[0409] Terminal

[0410] Before going to bed, the user launches the app and sets it to sleep mode.

[0411] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[0412] The collected data is sent to a server at regular intervals.

[0413] Specific examples

[0414] 1. The user opens the app and selects sleep mode.

[0415] 2. The device activates the camera and microphone and begins collecting video and audio data.

[0416] 3. The collected data is sent to the server.

[0417] Data analysis

[0418] server

[0419] The server analyzes the received video and audio data.

[0420] The number of times the patient turns over and changes in position are extracted from the video data.

[0421] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[0422] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[0423] The analysis results are stored in a database.

[0424] Specific examples

[0425] 1. The server analyzes the video data and counts the number of times the person turns over.

[0426] 2. The server analyzes the audio data and detects the frequency of snoring.

[0427] 3. The server evaluates the sleep depth and stores it in a database.

[0428] emotion recognition

[0429] Terminal

[0430] The data collected by the device is sent to the emotion engine in real time.

[0431] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[0432] server

[0433] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[0434] Specific examples

[0435] 1. While the user is sleeping, the device sends facial expressions and voice to the emotion engine.

[0436] 2. The emotion engine analyzes and recognizes the user's emotional state, whether relaxed or anxious.

[0437] 3. The server receives the emotion information and integrates it into the analysis data.

[0438] BGM / noise generation and playback

[0439] server

[0440] The server uses generative AI to generate optimal background music or noise based on the analysis results and emotional information.

[0441] The generated sound data is transmitted to the terminal.

[0442] Terminal

[0443] The device plays the background music or noise received from the server.

[0444] Specific examples

[0445] 1. The server inputs the analysis results and emotional information into the generation AI, which then generates background music or noise.

[0446] 2. The generated sound data is sent to the device.

[0447] 3. The device plays the received sound data.

[0448] User Feedback

[0449] User

[0450] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[0451] Terminal

[0452] The subjective evaluation input by the user is sent to the server.

[0453] Specific examples

[0454] 1. After waking up, the user opens the app and rates their sleep quality and emotions.

[0455] 2. The user enters a rating, and the device sends it to the server.

[0456] Next optimization

[0457] server

[0458] The server analyzes the subjective evaluations received and adjusts the parameters of the generation AI for generating the next background music or noise.

[0459] This allows the next sleep environment to be more tailored to the user.

[0460] Specific examples

[0461] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[0462] 2. Based on the analysis, generate optimized background music or noise for the next use.

[0463] ---

[0464] The above is an embodiment of the present invention. By analyzing the sleep data and emotional state of the user, it is possible to provide an optimal sleeping environment for each individual.

[0465] The processing flow will be explained below.

[0466] ---

[0467] Step 1:

[0468] Terminal

[0469] The user launches the app and sets it to sleep mode.

[0470] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[0471] Step 2:

[0472] Terminal

[0473] The collected video and audio data is sent to the server at regular intervals.

[0474] Step 3:

[0475] server

[0476] The server receives the video data and audio data transmitted from the terminal.

[0477] Step 4:

[0478] server

[0479] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[0480] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[0481] Step 5:

[0482] server

[0483] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[0484] The analysis results are saved in a database.

[0485] Step 6:

[0486] Terminal

[0487] The collected data is sent to the emotion engine in real time.

[0488] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[0489] Step 7:

[0490] server

[0491] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[0492] Step 8:

[0493] server

[0494] The analysis results and emotional information are input into a generation AI to generate optimal background music or noise.

[0495] Step 9:

[0496] server

[0497] The generated sound data is sent to the terminal.

[0498] Step 10:

[0499] Terminal

[0500] The device plays background music or noise received from the server.

[0501] Step 11:

[0502] User

[0503] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[0504] Step 12:

[0505] Terminal

[0506] The subjective evaluation input by the user is sent to the server.

[0507] Step 13:

[0508] server

[0509] The server analyzes the received subjective evaluations.

[0510] Based on the feedback, adjust the parameters of the generated AI.

[0511] Step 14:

[0512] server

[0513] The next background music or noise is generated based on the adjusted parameters.

[0514] ---

[0515] The above is the specific processing flow of the invention combined with the emotion engine, which makes it possible to provide an optimal sleeping environment that reflects the user's emotional state.

[0516] Example 2

[0517] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0518] Conventional sleep environment improvement systems simply collect and analyze users' sleep data, and are limited in their ability to generate optimal sleep-inducing background music or noise that takes into account the user's emotional state. Furthermore, they lack the ability to optimize the next sound environment based on the user's subjective evaluation. This makes it difficult to provide a sleep environment that is suited to each individual user.

[0519] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing collected video data and audio data, a means for recognizing the user's emotional state, and a means for generating optimal background music or noise based on the data obtained from the analysis means and the emotion recognition means. This makes it possible to comprehensively analyze the user's sleep data and emotional state and provide individually optimized sleep-inducing background music or noise.

[0520] "Video data" refers to visual information that records the user's appearance and movements while sleeping.

[0521] "Audio data" refers to audio information that records sounds that occur during sleep, such as the user talking in their sleep or snoring.

[0522] "Means of collection" refers to the hardware and software used to acquire and record video and audio data.

[0523] "Means for analysis" refers to software that processes and analyzes the collected video and audio data to extract information such as the number of times the patient turns over in their sleep and the frequency of snoring.

[0524] "Means for recognizing emotional state" refers to software and hardware that analyzes the user's facial expressions and tone of voice to identify emotions such as relaxed or anxious.

[0525] "Means for generating optimal background music or noise" refers to a generative AI model that generates music or sounds that are most relaxing for the user based on analysis results and emotional state data.

[0526] "Means for playing" refers to devices and software for outputting the generated background music or noise so that the user can hear it.

[0527] "Means for inputting subjective evaluation" refers to an interface and software that allows the user to input feedback on sleep quality and emotional state after waking up.

[0528] The "means for optimizing the generation means" refers to algorithms and software for adjusting the next background music or noise to better suit the user based on the user's subjective evaluation.

[0529] The "transmitting means" refers to a communication interface and software for transmitting the video data and audio data collected by the terminal to the server in real time.

[0530] A "generative AI model" is an artificial intelligence model that generates optimal background music and noise based on the user's analytical data and emotional data.

[0531] A "prompt sentence" is an input sentence that instructs the generative AI model on what background music or noise to generate.

[0532] This invention is a system that collects a user's sleep data, analyzes that data, and combines it with an emotion engine that recognizes the user's emotional state to generate and play optimal sleep-inducing background music and noise. The system is primarily composed of a terminal, a server, and a user. The specific operation of each element is explained below.

[0533] Data collection

[0534] Terminal

[0535] Users use a smartphone with a dedicated sleep tracking app installed, launch the app before going to bed, and select sleep mode.

[0536] The device automatically activates the camera and microphone to collect video and audio data while the user sleeps, including recording the user's movements, changes in posture, snoring, and talking in their sleep.

[0537] The collected data is sent from the terminal to the server at regular intervals.

[0538] Examples:

[0539] 1. The user opens the app and selects "Sleep mode," which causes the device to activate the camera and microphone and begin collecting data.

[0540] 2. The collected video and audio data is sent to the server in real time.

[0541] Data analysis

[0542] server

[0543] The server uses a high-performance data analysis program to detect the number of times the patient turns over and changes in posture from the received video data, and extracts information such as snoring, talking in one's sleep, and teeth grinding from the audio data.

[0544] Based on the extracted data, the user's sleep depth is evaluated on a three-point scale: "light," "medium," or "deep." The evaluation results are stored in a database and used for subsequent processing.

[0545] Examples:

[0546] 1. The server processes the video data and identifies the number of times the user has turned over.

[0547] 2. Analyze audio data and record the frequency and volume of snoring.

[0548] 3. The sleep depth assessment results are stored in a database.

[0549] emotion recognition

[0550] Terminal

[0551] The device sends the collected video and audio data in real time to an emotion recognition engine, which analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[0552] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server.

[0553] server

[0554] The server integrates the emotional state data received from the emotion recognition engine into the analysis data to assess the user's overall sleep state.

[0555] Examples:

[0556] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[0557] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[0558] 3. Emotional state data is sent to the server and integrated into the analysis data.

[0559] BGM / noise generation and playback

[0560] server

[0561] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. The generative AI model receives a prompt and generates the optimal sound for the user.

[0562] The generated sound data is transmitted to the terminal.

[0563] Terminal

[0564] The device plays background music or noise received from the server to help the user sleep.

[0565] Examples:

[0566] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[0567] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[0568] 3. The server sends the sound data to the device, which then plays it.

[0569] User Feedback

[0570] User

[0571] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state, based on their sleep experience.

[0572] Terminal

[0573] The subjective evaluation input by the user is sent to the server.

[0574] Examples:

[0575] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[0576] 2. The device sends the evaluation data to the server.

[0577] Next optimization

[0578] server

[0579] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. This feedback optimizes the next BGM and noise to better suit the user.

[0580] Examples:

[0581] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[0582] 2. Provide that feedback to the generative AI model and use it for the next generation.

[0583] In this way, the present invention can comprehensively analyze the user's sleep data and emotional state, and provide sleep-inducing background music and noises that are optimized for each individual user.

[0584] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0585] Step 1: Data collection

[0586] The device installs a dedicated sleep tracking app on the user, and has them launch the app and select a sleep mode before going to bed. Input: The user operates the app. Output: The device turns on the camera and microphone.

[0587] The device uses a camera and microphone to collect video and audio data in real time while the user sleeps. Input: The user's movements, changes in posture, snoring, and sleep talking. Output: Collected video and audio data.

[0588] The collected data is sent to the server at regular intervals. Input: Collected video and audio data. Output: Data is sent to the server.

[0589] Specific behavior:

[0590] 1. The user opens the app and selects "Sleep Mode."

[0591] 2. The device activates the camera and microphone and begins collecting data while the user sleeps.

[0592] 3. Upload the collected data to the server at regular intervals.

[0593] Step 2: Data analysis

[0594] The server analyzes the received video data and detects the number of turns and changes in body position. Input: Video data. Output: Data on the number of turns and changes in body position.

[0595] The server analyzes the audio data and extracts information such as snoring frequency, sleep talking, teeth grinding, etc. Input: Audio data. Output: Snoring frequency and pattern data.

[0596] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep." Input: Analyzed data. Output: Sleep depth evaluation data.

[0597] The analysis results are saved in a database. Input: Sleep depth evaluation data. Output: Data saved in the database.

[0598] Specific behavior:

[0599] 1. The server analyzes the video data and identifies the number of times and duration that the user turned over in their sleep.

[0600] 2. The server analyzes the audio data and records the frequency and patterns of snoring.

[0601] 3. The server evaluates the sleep depth and stores the results in a database.

[0602] Step 3: Emotion Recognition

[0603] The device sends the collected video and audio data to the emotion recognition engine in real time. Input: Collected data. Output: Data sent to the emotion recognition engine.

[0604] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. Input: Video data and audio data. Output: Emotional state data.

[0605] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server. Input: Emotional state data. Output: Emotional information sent to the server.

[0606] Specific behavior:

[0607] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[0608] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[0609] 3. Emotional state data is sent to the server and integrated into the analysis data.

[0610] Step 4: Generate and play background music / noise

[0611] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. Input: Analysis results and emotional state data. Output: Generated background music or noise data.

[0612] The generative AI model generates the optimal sound by inputting an appropriate prompt sentence. For example, "Please generate music that will relax the user." Input: Prompt sentence. Output: Generated sound data.

[0613] The generated sound data is sent to the terminal. Input: Generated sound data. Output: Sound data sent to the terminal.

[0614] The device plays background music or noise received from the server. Input: Sound data from the server. Output: Played sound.

[0615] Specific behavior:

[0616] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[0617] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[0618] 3. The server sends the sound data to the device, which then plays it.

[0619] Step 5: User feedback

[0620] After waking up, the user opens the app and inputs their subjective assessment of their sleep quality and emotional state. Input: Input of subjective assessment. Output: Inputted assessment data.

[0621] The device sends the user's subjective evaluation to the server. Input: User's subjective evaluation data. Output: Evaluation data sent to the server.

[0622] Specific behavior:

[0623] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[0624] 2. The device sends the evaluation data to the server.

[0625] Step 6: Next optimization

[0626] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. Input: Subjective evaluation data. Output: Adjusted parameters of the generative AI model.

[0627] This will optimize the background music and noise for the next time to better suit the user. Input: Adjusted parameters. Output: Optimized sound environment for the next use.

[0628] Specific behavior:

[0629] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[0630] 2. Provide that feedback to the generative AI model, and adjust the parameters in the next generation to provide an optimized sound environment.

[0631] (Application example 2)

[0632] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0633] In modern society, many people find it difficult to get a good night's sleep due to stress and irregular living conditions. In this situation, there is a need for systems that support sleep. In particular, there is a need to provide sleep-inducing background music and noises that are optimized for each user's individual condition, thereby enabling users to sleep more comfortably.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and audio data, means for generating optimal background music or noise based on the data and emotional information obtained from the analysis means, and means for adjusting the prompts of the generative AI model based on the user's subjective evaluation. This makes it possible to analyze the user's sleep data and emotional state in detail and generate and provide optimal sleep-inducing background music or noise for each individual user based on the analysis results.

[0635] "Video data" refers to information in the form of moving images collected by a photographing device such as a camera.

[0636] "Audio data" refers to information in the form of audio collected by a sound collection device such as a microphone.

[0637] The "analysis means" is a method or device for extracting specific information based on the collected video and audio data.

[0638] "Emotion information" is data relating to the user's emotional state analyzed from facial expressions, tone of voice, and the like.

[0639] "Optimal background music or noise" refers to background music or noise that is generated to support the user's sleep and is most suitable for the user's state.

[0640] "Subjective evaluation" refers to evaluation data input by the user based on their own feelings, and includes information on the quality of sleep, emotional state, and the like.

[0641] A "generative AI model" is an artificial intelligence model that generates specific content based on input data.

[0642] A "prompt" is input data for a generative AI model, and includes instructions and conditions regarding the content to be generated.

[0643] "Collecting means" refers to a method or device for acquiring video data and audio data.

[0644] "Transmitting means" refers to a method or device for sending collected data to another device or server.

[0645] This invention is a system that collects and analyzes a user's sleep data and recognizes the user's emotions. It mainly functions in cooperation with a terminal, a server, and a user. The specific operation of each element is described below.

[0646] Data collection

[0647] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the camera and microphone and collects video and audio data while the device is sleeping. This collected data is then sent to a server at regular intervals.

[0648] The specific hardware used is the smartphone camera and microphone. Audio data is collected using the sounddevice library, and video data is collected using cv2 (OpenCV).

[0649] Data analysis

[0650] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. This information is used to evaluate the depth of sleep on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[0651] The analysis tools used are cv2 for video data analysis and numpy for audio data analysis, and MySQL (registered trademark) and PostgreSQL are used as databases.

[0652] emotion recognition

[0653] The device sends the collected data to the emotion engine in real time, and analyzes the user's facial expressions and tone of voice to recognize their emotional state. This emotional information is then sent to the server and integrated into the analyzed data.

[0654] For emotion recognition, a library called DeepFace is used, which allows the system to recognize the user's emotional state, such as whether they are relaxed or anxious.

[0655] BGM / noise generation and playback

[0656] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[0657] An example of a generative AI model is GPT-3 (registered trademark). An example of a prompt is as follows:

[0658] User Emotion: Relaxed, Sleep State: Deep

[0659] Please create the best background music for this situation.

[0660] User Feedback

[0661] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent to the server.

[0662] Next optimization

[0663] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model for generating the next background music or noise, so that the next sleep environment is more tailored to the user.

[0664] As a specific example, if a user evaluates that they are "in a relaxed state and in a deep sleep," the AI ​​model will be adjusted based on that data so that similar background music will be generated the next time.

[0665] In this way, the present invention can provide the user with an optimal sleeping environment.

[0666] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0667] Step 1:

[0668] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the smartphone's camera and microphone and starts collecting video and audio data. This collected data is sent to the server every five seconds.

[0669] Specific input: The user's action to set the application to sleep mode.

[0670] Specific operation: The camera and microphone are activated and video and audio data is collected.

[0671] Specific output: Video and audio data collected in real time.

[0672] Step 2:

[0673] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. Based on this information, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[0674] Specific input: video data, audio data.

[0675] Specific operations: Analyze body movements from video data using cv2, and detect snoring and sleep talking from audio data using numpy.

[0676] Specific output: Sleep depth assessment results and a database storing them.

[0677] Step 3:

[0678] The device collects data in real time and sends the user's facial expressions and tone of voice to the emotion engine to analyze the user's emotional state. The server receives the emotional information and integrates it into the analysis data.

[0679] Specific input: collected video and audio data.

[0680] Specific operation: Recognizes emotions by analyzing facial expressions and tone of voice using the DeepFace library.

[0681] Specific output: Recognized emotional information.

[0682] Step 4:

[0683] The server uses a generative AI model to generate the optimal background music or noise for the user based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[0684] Specific input: Sleep depth assessment results and emotional information.

[0685] Specific operation: A prompt sentence is input into the generative AI model to generate background music or noise. Example prompt sentence: "User's emotion: Relaxed, Sleep state: Deep. Please generate background music that is optimal for this state."

[0686] Specific output: Generated background music or noise data.

[0687] Step 5:

[0688] The device plays the generated background music or noise to help the user sleep.

[0689] Specific input: generated background music or noise data.

[0690] Specific operation: Plays sound using the device's audio function.

[0691] Specific output: background music or noise heard by the user.

[0692] Step 6:

[0693] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent from the device to the server.

[0694] Specific input: Subjective evaluation entered by the user.

[0695] Specific operation: Enter sleep quality and emotional state on the application's evaluation screen and send the data.

[0696] Specific output: Subjective evaluation data sent to the server.

[0697] Step 7:

[0698] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model to generate background music or noise for the next use, thereby providing a more suitable environment for the next use.

[0699] Specific input: Subjective evaluation data submitted by users.

[0700] Specific operation: Analyze subjective evaluations and adjust the prompt parameters of the generative AI model.

[0701] Specific output: The background music or noise data that will be generated next time based on the adjusted prompt.

[0702] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0703] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0704] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0705] [Second embodiment]

[0706] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0707] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0708] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0709] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0710] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0711] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0712] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0713] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0714] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0715] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0716] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0717] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0718] ---

[0719] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0720] Data collection

[0721] Terminal

[0722] Before going to bed, the user launches the app and sets it to sleep mode.

[0723] The device's camera and microphone are used to collect video and audio data while you sleep.

[0724] The collected data is sent to a server at regular intervals.

[0725] Specific examples

[0726] 1. The user opens the app and selects sleep mode.

[0727] 2. The device activates the camera and microphone and begins collecting video and audio data.

[0728] 3. The collected data is sent to the server.

[0729] Data analysis

[0730] server

[0731] The server analyzes the received video and audio data.

[0732] The number of times the patient turns over and changes in position are extracted from the video data.

[0733] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[0734] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[0735] The analysis results are stored in a database.

[0736] Specific examples

[0737] 1. The server analyzes the video data and counts the number of times the person turns over.

[0738] 2. The server analyzes the audio data and detects the frequency of snoring.

[0739] 3. The server evaluates the sleep depth and stores it in a database.

[0740] BGM / noise generation and playback

[0741] server

[0742] The server uses a generative AI to generate optimal background music or noise based on the analysis results.

[0743] The generated sound data is transmitted to the terminal.

[0744] Terminal

[0745] The terminal plays the background music or noise received from the server.

[0746] Specific examples

[0747] 1. The server inputs the analysis results into the generation AI, which generates background music or noise.

[0748] 2. The generated sound data is sent to the device.

[0749] 3. The device plays the received sound data.

[0750] User Feedback

[0751] User

[0752] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[0753] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[0754] Terminal

[0755] The terminal transmits the subjective evaluation input by the user to the server.

[0756] Specific examples

[0757] 1. After waking up, the user opens the app and rates the quality of their sleep.

[0758] 2. The user enters a rating, and the device sends it to the server.

[0759] Next optimization

[0760] server

[0761] The server analyzes the received subjective evaluations and adjusts the parameters of the generation AI for generating the next background music or noise.

[0762] This allows the next sleep environment to be more tailored to the user.

[0763] Specific examples

[0764] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[0765] 2. Based on the analysis, generate optimized background music or noise for the next use.

[0766] ---

[0767] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[0768] The processing flow will be explained below.

[0769] ---

[0770] Step 1:

[0771] Terminal

[0772] The user launches the app and sets it to sleep mode.

[0773] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[0774] Step 2:

[0775] Terminal

[0776] The collected video and audio data is sent to the server at regular intervals.

[0777] Step 3:

[0778] server

[0779] The server receives the video data and audio data transmitted from the terminal.

[0780] Step 4:

[0781] server

[0782] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[0783] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[0784] Step 5:

[0785] server

[0786] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[0787] The analysis results are saved in a database.

[0788] Step 6:

[0789] server

[0790] The analysis results are input into a generation AI to generate the optimal background music or noise.

[0791] Step 7:

[0792] server

[0793] The generated sound data is sent to the terminal.

[0794] Step 8:

[0795] Terminal

[0796] The device plays background music or noise received from the server.

[0797] Step 9:

[0798] User

[0799] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[0800] Step 10:

[0801] Terminal

[0802] The subjective evaluation input by the user is sent to the server.

[0803] Step 11:

[0804] server

[0805] The server analyzes the received subjective evaluations.

[0806] Based on the feedback, adjust the parameters of the generated AI.

[0807] Step 12:

[0808] server

[0809] The next background music or noise is generated based on the adjusted parameters.

[0810] ---

[0811] The above is the specific flow of the program's processing, which allows the user to be provided with an optimal sleeping environment.

[0812] Example 1

[0813] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0814] Conventional sleep environment systems have struggled to provide optimal solutions for a user's individual sleep patterns or specific sleep problems. In particular, when it comes to generating background music or noise to improve sleep quality, they tend to use generic preset music or noises, and are unable to optimize them to meet the user's specific needs. Furthermore, there has been insufficient effort to effectively utilize subjective feedback from users to further optimize the next background music or noise.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0816] In this invention, the server includes a means for periodically transmitting video and audio data, a means for analyzing the collected video and audio data to extract changes in the user's body position and audio events, and a means for evaluating the depth of sleep. This allows for a detailed analysis of the user's specific sleep patterns and problems, and based on that, optimal background music and noise can be generated. Furthermore, by adjusting the parameters of the generation AI model based on the user's subjective evaluation, the playback environment can be further optimized for subsequent playbacks.

[0817] "Video data" refers to all visual information acquired using a camera or other imaging device.

[0818] "Audio data" refers to all information relating to sound captured using a recording device such as a microphone.

[0819] "Transmitting means" refers to the technical methods or devices used to transmit collected data to another device or server at specified intervals.

[0820] "Means of analysis" refers to the programs or algorithms that process the collected data and extract useful information.

[0821] "Changes in body position" refers to changes in the posture or position of the user's body.

[0822] "Sleep depth" is an indicator of how deep a user's sleep state is, and is usually evaluated on a three-level scale: "light," "intermediate," and "deep."

[0823] "Generative AI model" refers to a general algorithm that uses artificial intelligence to generate new data or content.

[0824] "BGM" is short for background music and refers to music played to promote a particular atmosphere or emotion.

[0825] "Noise" refers to random or controlled sounds reproduced for a specific purpose.

[0826] "Subjective evaluation" refers to a personal evaluation made by a user based on their own experience and feelings.

[0827] "Parameters" refer to settings that adjust the behavior of a particular algorithm or model.

[0828] "Optimization" refers to the process of adjusting parameters or settings to achieve the most effective state or result for a particular purpose.

[0829] "Means for playback" refers to the technical methods and devices for actually outputting the generated sound data as sound.

[0830] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0831] Data collection

[0832] Terminal

[0833] Before going to bed, the user launches the app and sets it to sleep mode.

[0834] The device is equipped with a camera and microphone, which are used to collect video and audio data while the person is sleeping.

[0835] The collected data is sent to the server at regular intervals. For example, the data is stored in a buffer every hour, and when a certain amount of data has accumulated, it is sent.

[0836] Specific examples

[0837] 1. The user opens the app before going to bed and selects sleep mode.

[0838] 2. The device activates its camera and microphone. The camera captures the user's posture and facial orientation, while the microphone records audio data such as snoring, talking in your sleep, and teeth grinding.

[0839] 3. The collected data is stored in a buffer at regular intervals, and the accumulated data is sent to the server.

[0840] Data analysis

[0841] server

[0842] The received video and audio data is analyzed using an advanced data analysis algorithm installed on the server, which is used to extract changes in the user's body position and audio events.

[0843] The video data is used to count the number of times the patient turns over and changes in position, while the audio data is used to detect the frequency of snoring, talking in one's sleep, and teeth grinding.

[0844] The depth of sleep is assessed on a three-point scale: light, medium, or deep. This assessment is based on the extracted data.

[0845] Specific examples

[0846] 1. The server analyzes the received video data and counts the number of times the person turns over.

[0847] 2. The server analyzes the audio data and detects the frequency of snoring and the number of times the person talks in their sleep.

[0848] 3. Based on this information, the depth of sleep is evaluated and stored in a database.

[0849] BGM / Noise Generation

[0850] server

[0851] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[0852] The generated sound data is sent to the terminal.

[0853] Specific examples

[0854] 1. The server inputs the following prompt to the generative AI model:

[0855] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[0856] 2. The generated sound data is sent to the device.

[0857] BGM / Noise playback

[0858] Terminal

[0859] The terminal plays the background music or noise received from the server.

[0860] The received sound data is played back at the appropriate time to optimize the user's sleep environment.

[0861] Specific examples

[0862] 1. The terminal prepares to play the sound data received from the server.

[0863] 2. Based on the collected data, background music or noise is played at the optimal timing.

[0864] Collecting user feedback

[0865] User

[0866] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[0867] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[0868] Terminal

[0869] The terminal transmits the subjective evaluation input by the user to the server.

[0870] Specific examples

[0871] 1. After waking up, the user opens the app and rates the quality of their sleep.

[0872] 2. The evaluation items are entered and the terminal sends them to the server.

[0873] Next optimization

[0874] server

[0875] The received subjective evaluations are analyzed and the parameters of the generative AI model for generating the next background music or noise are adjusted.

[0876] This allows background music or noise that is more suited to the user to be provided the next time the device is used.

[0877] Specific examples

[0878] 1. The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[0879] 2. Update the prompt text to generate optimized background music or noise the next time you use it.

[0880] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user in detail, it is possible to provide an individually optimized sleep environment.

[0881] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0882] Step 1: Launch and configure the app

[0883] Terminal

[0884] The user launches the app by tapping the app icon on their device.

[0885] Select "Sleep Mode" from the in-app menu.

[0886] If a popup appears requesting permission to use the camera and microphone, the user selects Allow.

[0887] Input and Output

[0888] Input: User touch actions (tapping app icons, selecting sleep mode)

[0889] Output: Camera and microphone are activated, and sleep data collection is complete.

[0890] Specific actions

[0891] 1. When a user opens the app and selects sleep mode, the device activates the camera and microphone.

[0892] 2. The camera monitors the user's position and the microphone begins collecting sound from within the room.

[0893] Step 2: Data collection

[0894] Terminal

[0895] The device's camera captures the user's posture and facial orientation, while the microphone records snoring, talking in one's sleep, and environmental sounds.

[0896] The collected data is temporarily stored in a buffer.

[0897] Input and Output

[0898] Input: Video and audio data from the camera and microphone

[0899] Output: Buffered video and audio data

[0900] Specific actions

[0901] 1. The camera captures multiple frames of video per second, and the microphone records audio at regular intervals.

[0902] 2. The collected video and audio data is stored in a buffer.

[0903] Step 3: Sending data

[0904] Terminal

[0905] The data stored in the buffer is sent to the server at regular intervals.

[0906] Input and Output

[0907] Input: Video and audio data stored in the buffer

[0908] Output: Video and audio data sent to the server

[0909] Specific actions

[0910] 1. When a certain amount of data accumulates in the buffer, the terminal compresses the data and transmits it.

[0911] 2. The server receives the data.

[0912] Step 4: Analyze the data

[0913] server

[0914] The received video and audio data is analyzed. The number of times the patient turns over in their sleep and changes in posture are extracted from the video data, and the frequency of snoring, talking in their sleep, and teeth grinding is detected from the audio data.

[0915] Sleep depth is evaluated on three levels: "light," "intermediate," and "deep."

[0916] Input and Output

[0917] Input: Video and audio data sent to the server

[0918] Output: Extracted body position change data, audio event data, sleep depth assessment

[0919] Specific actions

[0920] 1. The server analyzes the video data frame by frame and counts the user's changes in position and the number of times they turn over in bed.

[0921] 2. Analyze audio data to detect the frequency of snoring, sleep talking, and teeth grinding.

[0922] 3. Based on the extracted data, sleep depth is evaluated on a three-point scale and the results are stored in a database.

[0923] Step 5: Generate background music / noise

[0924] server

[0925] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[0926] The generated sound data is sent to the terminal.

[0927] Input and Output

[0928] Input: Analysis result data, prompt statement

[0929] Output: Generated background music or noise

[0930] Specific actions

[0931] 1. The server inputs the following prompt to the generative AI model:

[0932] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[0933] 2. The generative AI model generates optimal sound data and sends it to the device.

[0934] Step 6: Play background music / noise

[0935] Terminal

[0936] The device plays background music or noise received from the server.

[0937] Input and Output

[0938] Input: BGM or noise sent from the server

[0939] Output: BGM or noise to be played

[0940] Specific actions

[0941] 1. The terminal prepares to play the sound data received from the server.

[0942] 2. Play background music or noise at specified times to optimize the user's sleep environment.

[0943] Step 7: Gather user feedback

[0944] User

[0945] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[0946] Input and Output

[0947] Input: User subjective evaluation data

[0948] Output: Subjective evaluation data sent to the server

[0949] Specific actions

[0950] 1. The user opens the app and enters information about their sleep quality (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[0951] 2. The device sends the entered evaluation data to the server.

[0952] Step 8: Next optimization

[0953] server

[0954] The received subjective evaluations are analyzed and the parameters of the generative AI model are adjusted.

[0955] Input and Output

[0956] Input: User subjective evaluation data

[0957] Output: Adjusted generative AI model parameters

[0958] Specific actions

[0959] 1. The server analyzes the received subjective evaluation data and identifies which elements were problematic.

[0960] 2. Adjust the parameters of the generative AI model for the next BGM or noise generation.

[0961] The above is the specific flow of the program processing of this system. Through each processing step, it is possible to effectively collect and analyze the user's sleep data and provide an individually optimized sleep environment.

[0962] (Application example 1)

[0963] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] Conventional sleep improvement systems only collect users' sleep data and rarely provide personalized, optimized responses based on that data. Furthermore, they lack real-time feedback and improvement measures based on analyzed data, which means users' sleep quality is not fully improved. Furthermore, there are insufficient means for effectively utilizing subjective feedback.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0966] In this invention, the server includes means for collecting user sleep data, means for transmitting the collected video data and audio data, means for analyzing the transmitted data, a generative AI model that generates optimal background music or noise based on the data obtained from the analysis means, means for playing the generated background music or noise, means for the user to input a subjective evaluation of sleep quality, and means for adjusting the parameters of the generative AI model based on the subjective evaluation. This provides an optimal sleep environment tailored to each user's individual sleep pattern, and enables further optimization based on feedback.

[0967] "User's sleep data" refers to information including video data and audio data that indicate the user's sleep state.

[0968] "Video data" refers to video data that records the user's sleeping posture, the number of times they turn over in their sleep, and so on.

[0969] "Voice data" refers to audio data including the user's snoring, talking in their sleep, environmental sounds, and the like.

[0970] "Transmitting means" refers to the functions and processes for transferring collected video and audio data to a server.

[0971] "Means of analysis" refers to the process by which the server handles video and audio data and extracts and analyzes the number of times the patient turns over in bed and the frequency of snoring, etc.

[0972] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal background music or noise based on analytical results and subjective evaluation.

[0973] "BGM or Noise" refers to sound data such as music or environmental sounds created by a generative AI model that are played to improve a user's sleep.

[0974] "Means for playback" refers to the functions or devices for playing background music or noise generated on a device such as a smartphone.

[0975] "Subjective evaluation" refers to the user inputting their impression of the quality of sleep as a numerical value or a comment.

[0976] "Means for adjusting the parameters of the generative AI model" refers to the process of changing the settings of the generative AI model based on the user's subjective evaluation and reflecting this in the next background music or noise generation.

[0977] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[0978] 1. Data Collection

[0979] Terminal

[0980] Before going to bed, the user launches the app and sets it to sleep mode.

[0981] The device's camera and microphone are used to collect video and audio data while you sleep.

[0982] The collected data is sent to a server at regular intervals.

[0983] Specific examples

[0984] The user opens the app and selects sleep mode.

[0985] The device will activate its camera and microphone and begin collecting video and audio data.

[0986] The collected data is sent to a server.

[0987] 2. Data Analysis

[0988] server

[0989] The server analyzes the received video and audio data.

[0990] The number of times the patient turns over and changes in position are extracted from the video data.

[0991] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[0992] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[0993] The analysis results are stored in a database.

[0994] Specific examples

[0995] The server analyzes the video data and counts the number of times the person turns over.

[0996] The server analyzes the audio data and detects the frequency of snoring.

[0997] The server evaluates the depth of sleep and stores it in a database.

[0998] 3. BGM / Noise Generation and Playback

[0999] server

[1000] The server uses a generative AI model to generate optimal background music or noise based on the analysis results.

[1001] The generated sound data is transmitted to the terminal.

[1002] Terminal

[1003] The terminal plays the background music or noise received from the server.

[1004] Specific examples

[1005] The server inputs the analysis results into a generative AI model to generate background music or noise.

[1006] The generated sound data is transmitted to the terminal.

[1007] The terminal plays back the received sound data.

[1008] 4. User Feedback

[1009] User

[1010] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[1011] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[1012] Terminal

[1013] The terminal transmits the subjective evaluation input by the user to the server.

[1014] Specific examples

[1015] After waking up, the user opens the app and rates the quality of their sleep.

[1016] The user enters a rating, which the terminal sends to the server.

[1017] 5. Next optimization

[1018] server

[1019] The server analyzes the received subjective evaluations and adjusts the parameters of the generative AI model for generating the next background music or noise.

[1020] This allows the next sleep environment to be more tailored to the user.

[1021] Specific examples

[1022] The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[1023] Based on the analysis, optimized background music or noise will be generated the next time you use it.

[1024] Hardware and software used

[1025] Hardware: Smartphone (camera, microphone, speaker)

[1026] Software: Python, OpenCV (cv2), Request, SoundDevice (sd), server backend (Flask or Django)

[1027] Data processing and calculation

[1028] 1. Data Collection:

[1029] Video and audio data is collected using the smartphone's camera and microphone.

[1030] The data is uploaded to the server as image or audio files at regular intervals.

[1031] 2. Data Analysis:

[1032] The server analyzes the received data and detects things like turning over in bed and snoring.

[1033] The data is stored in a database for evaluating the depth of sleep.

[1034] 3. BGM / Noise Generation and Playback:

[1035] The generative AI model is run on the server to generate optimal background music and noise.

[1036] The sound data is sent to a smartphone and played at the specified time.

[1037] 4. User Feedback:

[1038] Collect user evaluation data from the smartphone interface.

[1039] The feedback data is sent to the server and used for the next BGM generation.

[1040] Prompt Sentence Examples

[1041] Below are some examples of prompts to input to the generative AI model.

[1042] We analyzed the user's sleep data. The frequency of snoring was high, and the average number of turns was three. The user's sleep depth was shallow, so please generate alpha wave music that has a relaxing effect.

[1043] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[1044] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1045] Step 1:

[1046] Before going to bed, the user launches the app and sets it to sleep mode.

[1047] Input: User launches the app and selects sleep mode.

[1048] Output: The device is ready to start collecting data.

[1049] What happens: The user launches the app on their smartphone and selects "sleep mode," which causes the app to turn on the camera and microphone and prepare for data collection.

[1050] Step 2:

[1051] The device's camera and microphone are used to collect video and audio data while you sleep.

[1052] Input: The app activates the camera and microphone.

[1053] Output: Collected video and audio data.

[1054] How it works: The device's camera detects the user's posture and tossing and turning while sleeping, and the microphone detects snoring and talking in their sleep. This data is temporarily saved as image and audio files.

[1055] Step 3:

[1056] The collected data is sent to the server at regular intervals.

[1057] Input: Collected video and audio data.

[1058] Output: Data uploaded to the server.

[1059] How it works: Every 60 seconds, the device uploads the collected data to the server. Data transfer is performed via an internet connection and an API request.

[1060] Step 4:

[1061] The server parses the received data.

[1062] Input: Video and audio data uploaded to the server.

[1063] Output: Analysis results such as number of turns, frequency of snoring, and sleep depth.

[1064] How it works: The server analyzes the video data and counts the number of times the person turns over in bed and changes their position. Similarly, it extracts the frequency of snoring and talking in their sleep from the audio data. Based on this, it evaluates the depth of sleep and stores the results in a database.

[1065] Step 5:

[1066] Uses generative AI models to generate optimal background music or noise based on the analysis results.

[1067] Input: Analysis results (number of turns, frequency of snoring, depth of sleep).

[1068] Output: Generated background music or noise sound data.

[1069] Specific operation: The server inputs the analysis results into the generative AI model and generates a prompt. For example, it might input, "We analyzed the user's sleep data. We found that the user snores frequently and turns over an average of three times. The user's sleep depth is shallow, so please generate alpha wave music that has a relaxing effect." The generative AI model then generates music data based on the instructions.

[1070] Step 6:

[1071] The generated sound data is sent to the terminal and played back.

[1072] Input: Sound data for the generated background music or noise.

[1073] Output: Sound data played on the device.

[1074] Specific operation: The server sends the generated sound data to the device, which then plays the received sound data in real time to help the user sleep.

[1075] Step 7:

[1076] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[1077] Input: User's subjective evaluation (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[1078] Output: Subjective evaluation data sent to the server.

[1079] How it works: The user opens the app and inputs their sleep quality rating, including how easily they fell asleep, whether they woke up during the night, and how easily they woke up, using numerical values ​​and comments. The device then sends this data to the server.

[1080] Step 8:

[1081] The server analyzes the subjective evaluations it receives and adjusts the parameters of the generative AI model.

[1082] Input: Subjective evaluation data sent to the server.

[1083] Output: The adjusted parameters of the generative AI model.

[1084] How it works: The server analyzes the user's subjective evaluation data and adjusts the parameters of the generative AI model, which then generates more appropriate background music or noise for the next session, improving the user's sleep quality.

[1085] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1086] ---

[1087] This invention is a system that collects and analyzes a user's sleep data and combines it with an emotion engine that recognizes the user's emotions to generate optimal sleep-inducing background music and noises. It mainly functions in cooperation with the terminal, server, and user. The specific operation of each element is explained below.

[1088] Data collection

[1089] Terminal

[1090] Before going to bed, the user launches the app and sets it to sleep mode.

[1091] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[1092] The collected data is sent to a server at regular intervals.

[1093] Specific examples

[1094] 1. The user opens the app and selects sleep mode.

[1095] 2. The device activates the camera and microphone and begins collecting video and audio data.

[1096] 3. The collected data is sent to the server.

[1097] Data analysis

[1098] server

[1099] The server analyzes the received video and audio data.

[1100] The number of times the patient turns over and changes in position are extracted from the video data.

[1101] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[1102] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[1103] The analysis results are stored in a database.

[1104] Specific examples

[1105] 1. The server analyzes the video data and counts the number of times the person turns over.

[1106] 2. The server analyzes the audio data and detects the frequency of snoring.

[1107] 3. The server evaluates the sleep depth and stores it in a database.

[1108] emotion recognition

[1109] Terminal

[1110] The data collected by the device is sent to the emotion engine in real time.

[1111] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[1112] server

[1113] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[1114] Specific examples

[1115] 1. While the user is sleeping, the device sends facial expressions and voice to the emotion engine.

[1116] 2. The emotion engine analyzes and recognizes the user's emotional state, whether relaxed or anxious.

[1117] 3. The server receives the emotion information and integrates it into the analysis data.

[1118] BGM / noise generation and playback

[1119] server

[1120] The server uses generative AI to generate optimal background music or noise based on the analysis results and emotional information.

[1121] The generated sound data is transmitted to the terminal.

[1122] Terminal

[1123] The device plays the background music or noise received from the server.

[1124] Specific examples

[1125] 1. The server inputs the analysis results and emotional information into the generation AI, which then generates background music or noise.

[1126] 2. The generated sound data is sent to the device.

[1127] 3. The device plays the received sound data.

[1128] User Feedback

[1129] User

[1130] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[1131] Terminal

[1132] The subjective evaluation input by the user is sent to the server.

[1133] Specific examples

[1134] 1. After waking up, the user opens the app and rates their sleep quality and emotions.

[1135] 2. The user enters a rating, and the device sends it to the server.

[1136] Next optimization

[1137] server

[1138] The server analyzes the subjective evaluations received and adjusts the parameters of the generation AI for generating the next background music or noise.

[1139] This allows the next sleep environment to be more tailored to the user.

[1140] Specific examples

[1141] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[1142] 2. Based on the analysis, generate optimized background music or noise for the next use.

[1143] ---

[1144] The above is an embodiment of the present invention. By analyzing the sleep data and emotional state of the user, it is possible to provide an optimal sleeping environment for each individual.

[1145] The processing flow will be explained below.

[1146] ---

[1147] Step 1:

[1148] Terminal

[1149] The user launches the app and sets it to sleep mode.

[1150] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[1151] Step 2:

[1152] Terminal

[1153] The collected video and audio data is sent to the server at regular intervals.

[1154] Step 3:

[1155] server

[1156] The server receives the video data and audio data transmitted from the terminal.

[1157] Step 4:

[1158] server

[1159] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[1160] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[1161] Step 5:

[1162] server

[1163] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[1164] The analysis results are saved in a database.

[1165] Step 6:

[1166] Terminal

[1167] The collected data is sent to the emotion engine in real time.

[1168] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[1169] Step 7:

[1170] server

[1171] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[1172] Step 8:

[1173] server

[1174] The analysis results and emotional information are input into a generation AI to generate optimal background music or noise.

[1175] Step 9:

[1176] server

[1177] The generated sound data is sent to the terminal.

[1178] Step 10:

[1179] Terminal

[1180] The device plays background music or noise received from the server.

[1181] Step 11:

[1182] User

[1183] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[1184] Step 12:

[1185] Terminal

[1186] The subjective evaluation input by the user is sent to the server.

[1187] Step 13:

[1188] server

[1189] The server analyzes the received subjective evaluations.

[1190] Based on the feedback, adjust the parameters of the generated AI.

[1191] Step 14:

[1192] server

[1193] The next background music or noise is generated based on the adjusted parameters.

[1194] ---

[1195] The above is the specific processing flow of the invention combined with the emotion engine, which makes it possible to provide an optimal sleeping environment that reflects the user's emotional state.

[1196] Example 2

[1197] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1198] Conventional sleep environment improvement systems simply collect and analyze users' sleep data, and are limited in their ability to generate optimal sleep-inducing background music or noise that takes into account the user's emotional state. Furthermore, they lack the ability to optimize the next sound environment based on the user's subjective evaluation. This makes it difficult to provide a sleep environment that is suited to each individual user.

[1199] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing collected video data and audio data, a means for recognizing the user's emotional state, and a means for generating optimal background music or noise based on the data obtained from the analysis means and the emotion recognition means. This makes it possible to comprehensively analyze the user's sleep data and emotional state and provide individually optimized sleep-inducing background music or noise.

[1200] "Video data" refers to visual information that records the user's appearance and movements while sleeping.

[1201] "Audio data" refers to audio information that records sounds that occur during sleep, such as the user talking in their sleep or snoring.

[1202] "Means of collection" refers to the hardware and software used to acquire and record video and audio data.

[1203] "Means for analysis" refers to software that processes and analyzes the collected video and audio data to extract information such as the number of times the patient turns over in their sleep and the frequency of snoring.

[1204] "Means for recognizing emotional state" refers to software and hardware that analyzes the user's facial expressions and tone of voice to identify emotions such as relaxed or anxious.

[1205] "Means for generating optimal background music or noise" refers to a generative AI model that generates music or sounds that are most relaxing for the user based on analysis results and emotional state data.

[1206] "Means for playing" refers to devices and software for outputting the generated background music or noise so that the user can hear it.

[1207] "Means for inputting subjective evaluation" refers to an interface and software that allows the user to input feedback on sleep quality and emotional state after waking up.

[1208] The "means for optimizing the generation means" refers to algorithms and software for adjusting the next background music or noise to better suit the user based on the user's subjective evaluation.

[1209] The "transmitting means" refers to a communication interface and software for transmitting the video data and audio data collected by the terminal to the server in real time.

[1210] A "generative AI model" is an artificial intelligence model that generates optimal background music and noise based on the user's analytical data and emotional data.

[1211] A "prompt sentence" is an input sentence that instructs the generative AI model on what background music or noise to generate.

[1212] This invention is a system that collects a user's sleep data, analyzes that data, and combines it with an emotion engine that recognizes the user's emotional state to generate and play optimal sleep-inducing background music and noise. The system is primarily composed of a terminal, a server, and a user. The specific operation of each element is explained below.

[1213] Data collection

[1214] Terminal

[1215] Users use a smartphone with a dedicated sleep tracking app installed, launch the app before going to bed, and select sleep mode.

[1216] The device automatically activates the camera and microphone to collect video and audio data while the user sleeps, including recording the user's movements, changes in posture, snoring, and talking in their sleep.

[1217] The collected data is sent from the terminal to the server at regular intervals.

[1218] Examples:

[1219] 1. The user opens the app and selects "Sleep mode," which causes the device to activate the camera and microphone and begin collecting data.

[1220] 2. The collected video and audio data is sent to the server in real time.

[1221] Data analysis

[1222] server

[1223] The server uses a high-performance data analysis program to detect the number of times the patient turns over and changes in posture from the received video data, and extracts information such as snoring, talking in one's sleep, and teeth grinding from the audio data.

[1224] Based on the extracted data, the user's sleep depth is evaluated on a three-point scale: "light," "medium," or "deep." The evaluation results are stored in a database and used for subsequent processing.

[1225] Examples:

[1226] 1. The server processes the video data and identifies the number of times the user has turned over.

[1227] 2. Analyze audio data and record the frequency and volume of snoring.

[1228] 3. The sleep depth assessment results are stored in a database.

[1229] emotion recognition

[1230] Terminal

[1231] The device sends the collected video and audio data in real time to an emotion recognition engine, which analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[1232] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server.

[1233] server

[1234] The server integrates the emotional state data received from the emotion recognition engine into the analysis data to assess the user's overall sleep state.

[1235] Examples:

[1236] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[1237] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[1238] 3. Emotional state data is sent to the server and integrated into the analysis data.

[1239] BGM / noise generation and playback

[1240] server

[1241] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. The generative AI model receives a prompt and generates the optimal sound for the user.

[1242] The generated sound data is transmitted to the terminal.

[1243] Terminal

[1244] The device plays background music or noise received from the server to help the user sleep.

[1245] Examples:

[1246] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[1247] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[1248] 3. The server sends the sound data to the device, which then plays it.

[1249] User Feedback

[1250] User

[1251] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state, based on their sleep experience.

[1252] Terminal

[1253] The subjective evaluation input by the user is sent to the server.

[1254] Examples:

[1255] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[1256] 2. The device sends the evaluation data to the server.

[1257] Next optimization

[1258] server

[1259] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. This feedback optimizes the next BGM and noise to better suit the user.

[1260] Examples:

[1261] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[1262] 2. Provide that feedback to the generative AI model and use it for the next generation.

[1263] In this way, the present invention can comprehensively analyze the user's sleep data and emotional state, and provide sleep-inducing background music and noises that are optimized for each individual user.

[1264] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1265] Step 1: Data collection

[1266] The device installs a dedicated sleep tracking app on the user, and has them launch the app and select a sleep mode before going to bed. Input: The user operates the app. Output: The device turns on the camera and microphone.

[1267] The device uses a camera and microphone to collect video and audio data in real time while the user sleeps. Input: The user's movements, changes in posture, snoring, and sleep talking. Output: Collected video and audio data.

[1268] The collected data is sent to the server at regular intervals. Input: Collected video and audio data. Output: Data is sent to the server.

[1269] Specific behavior:

[1270] 1. The user opens the app and selects "Sleep Mode."

[1271] 2. The device activates the camera and microphone and begins collecting data while the user sleeps.

[1272] 3. Upload the collected data to the server at regular intervals.

[1273] Step 2: Data analysis

[1274] The server analyzes the received video data and detects the number of turns and changes in body position. Input: Video data. Output: Data on the number of turns and changes in body position.

[1275] The server analyzes the audio data and extracts information such as snoring frequency, sleep talking, teeth grinding, etc. Input: Audio data. Output: Snoring frequency and pattern data.

[1276] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep." Input: Analyzed data. Output: Sleep depth evaluation data.

[1277] The analysis results are saved in a database. Input: Sleep depth evaluation data. Output: Data saved in the database.

[1278] Specific behavior:

[1279] 1. The server analyzes the video data and identifies the number of times and duration that the user turned over in their sleep.

[1280] 2. The server analyzes the audio data and records the frequency and patterns of snoring.

[1281] 3. The server evaluates the sleep depth and stores the results in a database.

[1282] Step 3: Emotion Recognition

[1283] The device sends the collected video and audio data to the emotion recognition engine in real time. Input: Collected data. Output: Data sent to the emotion recognition engine.

[1284] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. Input: Video data and audio data. Output: Emotional state data.

[1285] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server. Input: Emotional state data. Output: Emotional information sent to the server.

[1286] Specific behavior:

[1287] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[1288] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[1289] 3. Emotional state data is sent to the server and integrated into the analysis data.

[1290] Step 4: Generate and play background music / noise

[1291] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. Input: Analysis results and emotional state data. Output: Generated background music or noise data.

[1292] The generative AI model generates the optimal sound by inputting an appropriate prompt sentence. For example, "Please generate music that will relax the user." Input: Prompt sentence. Output: Generated sound data.

[1293] The generated sound data is sent to the terminal. Input: Generated sound data. Output: Sound data sent to the terminal.

[1294] The device plays background music or noise received from the server. Input: Sound data from the server. Output: Played sound.

[1295] Specific behavior:

[1296] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[1297] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[1298] 3. The server sends the sound data to the device, which then plays it.

[1299] Step 5: User feedback

[1300] After waking up, the user opens the app and inputs their subjective assessment of their sleep quality and emotional state. Input: Input of subjective assessment. Output: Inputted assessment data.

[1301] The device sends the user's subjective evaluation to the server. Input: User's subjective evaluation data. Output: Evaluation data sent to the server.

[1302] Specific behavior:

[1303] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[1304] 2. The device sends the evaluation data to the server.

[1305] Step 6: Next optimization

[1306] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. Input: Subjective evaluation data. Output: Adjusted parameters of the generative AI model.

[1307] This will optimize the background music and noise for the next time to better suit the user. Input: Adjusted parameters. Output: Optimized sound environment for the next use.

[1308] Specific behavior:

[1309] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[1310] 2. Provide that feedback to the generative AI model, and adjust the parameters in the next generation to provide an optimized sound environment.

[1311] (Application example 2)

[1312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1313] In modern society, many people find it difficult to get a good night's sleep due to stress and irregular living conditions. In this situation, there is a need for systems that support sleep. In particular, there is a need to provide sleep-inducing background music and noises that are optimized for each user's individual condition, thereby enabling users to sleep more comfortably.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and audio data, means for generating optimal background music or noise based on the data and emotional information obtained from the analysis means, and means for adjusting the prompts of the generative AI model based on the user's subjective evaluation. This makes it possible to analyze the user's sleep data and emotional state in detail and generate and provide optimal sleep-inducing background music or noise for each individual user based on the analysis results.

[1315] "Video data" refers to information in the form of moving images collected by a photographing device such as a camera.

[1316] "Audio data" refers to information in the form of audio collected by a sound collection device such as a microphone.

[1317] The "analysis means" is a method or device for extracting specific information based on the collected video and audio data.

[1318] "Emotion information" is data relating to the user's emotional state analyzed from facial expressions, tone of voice, and the like.

[1319] "Optimal background music or noise" refers to background music or noise that is generated to support the user's sleep and is most suitable for the user's state.

[1320] "Subjective evaluation" refers to evaluation data input by the user based on their own feelings, and includes information on the quality of sleep, emotional state, and the like.

[1321] A "generative AI model" is an artificial intelligence model that generates specific content based on input data.

[1322] A "prompt" is input data for a generative AI model, and includes instructions and conditions regarding the content to be generated.

[1323] "Collecting means" refers to a method or device for acquiring video data and audio data.

[1324] "Transmitting means" refers to a method or device for sending collected data to another device or server.

[1325] This invention is a system that collects and analyzes a user's sleep data and recognizes the user's emotions. It mainly functions in cooperation with a terminal, a server, and a user. The specific operation of each element is described below.

[1326] Data collection

[1327] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the camera and microphone and collects video and audio data while the device is sleeping. This collected data is then sent to a server at regular intervals.

[1328] The specific hardware used is the smartphone camera and microphone. Audio data is collected using the sounddevice library, and video data is collected using cv2 (OpenCV).

[1329] Data analysis

[1330] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. This information is used to evaluate the depth of sleep on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[1331] The analysis tools used are cv2 for video data analysis and numpy for audio data analysis, while databases such as MySQL and PostgreSQL are used.

[1332] emotion recognition

[1333] The device sends the collected data to the emotion engine in real time, and analyzes the user's facial expressions and tone of voice to recognize their emotional state. This emotional information is then sent to the server and integrated into the analyzed data.

[1334] For emotion recognition, a library called DeepFace is used, which allows the system to recognize the user's emotional state, such as whether they are relaxed or anxious.

[1335] BGM / noise generation and playback

[1336] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[1337] An example of a generative AI model would be GPT-3. An example prompt would be:

[1338] User Emotion: Relaxed, Sleep State: Deep

[1339] Please create the best background music for this situation.

[1340] User Feedback

[1341] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent to the server.

[1342] Next optimization

[1343] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model for generating the next background music or noise, so that the next sleep environment is more tailored to the user.

[1344] As a specific example, if a user evaluates that they are "in a relaxed state and in a deep sleep," the AI ​​model will be adjusted based on that data so that similar background music will be generated the next time.

[1345] In this way, the present invention can provide the user with an optimal sleeping environment.

[1346] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1347] Step 1:

[1348] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the smartphone's camera and microphone and starts collecting video and audio data. This collected data is sent to the server every five seconds.

[1349] Specific input: The user's action to set the application to sleep mode.

[1350] Specific operation: The camera and microphone are activated and video and audio data is collected.

[1351] Specific output: Video and audio data collected in real time.

[1352] Step 2:

[1353] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. Based on this information, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[1354] Specific input: video data, audio data.

[1355] Specific operations: Analyze body movements from video data using cv2, and detect snoring and sleep talking from audio data using numpy.

[1356] Specific output: Sleep depth assessment results and a database storing them.

[1357] Step 3:

[1358] The device collects data in real time and sends the user's facial expressions and tone of voice to the emotion engine to analyze the user's emotional state. The server receives the emotional information and integrates it into the analysis data.

[1359] Specific input: collected video and audio data.

[1360] Specific operation: Recognizes emotions by analyzing facial expressions and tone of voice using the DeepFace library.

[1361] Specific output: Recognized emotional information.

[1362] Step 4:

[1363] The server uses a generative AI model to generate the optimal background music or noise for the user based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[1364] Specific input: Sleep depth assessment results and emotional information.

[1365] Specific operation: A prompt sentence is input into the generative AI model to generate background music or noise. Example prompt sentence: "User's emotion: Relaxed, Sleep state: Deep. Please generate background music that is optimal for this state."

[1366] Specific output: Generated background music or noise data.

[1367] Step 5:

[1368] The device plays the generated background music or noise to help the user sleep.

[1369] Specific input: generated background music or noise data.

[1370] Specific operation: Plays sound using the device's audio function.

[1371] Specific output: background music or noise heard by the user.

[1372] Step 6:

[1373] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent from the device to the server.

[1374] Specific input: Subjective evaluation entered by the user.

[1375] Specific operation: Enter sleep quality and emotional state on the application's evaluation screen and send the data.

[1376] Specific output: Subjective evaluation data sent to the server.

[1377] Step 7:

[1378] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model to generate background music or noise for the next use, thereby providing a more suitable environment for the next use.

[1379] Specific input: Subjective evaluation data submitted by users.

[1380] Specific operation: Analyze subjective evaluations and adjust the prompt parameters of the generative AI model.

[1381] Specific output: The background music or noise data that will be generated next time based on the adjusted prompt.

[1382] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1385] [Third embodiment]

[1386] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1387] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1389] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1393] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1396] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1397] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1398] ---

[1399] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[1400] Data collection

[1401] Terminal

[1402] Before going to bed, the user launches the app and sets it to sleep mode.

[1403] The device's camera and microphone are used to collect video and audio data while you sleep.

[1404] The collected data is sent to a server at regular intervals.

[1405] Specific examples

[1406] 1. The user opens the app and selects sleep mode.

[1407] 2. The device activates the camera and microphone and begins collecting video and audio data.

[1408] 3. The collected data is sent to the server.

[1409] Data analysis

[1410] server

[1411] The server analyzes the received video and audio data.

[1412] The number of times the patient turns over and changes in position are extracted from the video data.

[1413] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[1414] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[1415] The analysis results are stored in a database.

[1416] Specific examples

[1417] 1. The server analyzes the video data and counts the number of times the person turns over.

[1418] 2. The server analyzes the audio data and detects the frequency of snoring.

[1419] 3. The server evaluates the sleep depth and stores it in a database.

[1420] BGM / noise generation and playback

[1421] server

[1422] The server uses a generative AI to generate optimal background music or noise based on the analysis results.

[1423] The generated sound data is transmitted to the terminal.

[1424] Terminal

[1425] The terminal plays the background music or noise received from the server.

[1426] Specific examples

[1427] 1. The server inputs the analysis results into the generation AI, which generates background music or noise.

[1428] 2. The generated sound data is sent to the device.

[1429] 3. The device plays the received sound data.

[1430] User Feedback

[1431] User

[1432] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[1433] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[1434] Terminal

[1435] The terminal transmits the subjective evaluation input by the user to the server.

[1436] Specific examples

[1437] 1. After waking up, the user opens the app and rates the quality of their sleep.

[1438] 2. The user enters a rating, and the device sends it to the server.

[1439] Next optimization

[1440] server

[1441] The server analyzes the received subjective evaluations and adjusts the parameters of the generation AI for generating the next background music or noise.

[1442] This allows the next sleep environment to be more tailored to the user.

[1443] Specific examples

[1444] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[1445] 2. Based on the analysis, generate optimized background music or noise for the next use.

[1446] ---

[1447] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[1448] The processing flow will be explained below.

[1449] ---

[1450] Step 1:

[1451] Terminal

[1452] The user launches the app and sets it to sleep mode.

[1453] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[1454] Step 2:

[1455] Terminal

[1456] The collected video and audio data is sent to the server at regular intervals.

[1457] Step 3:

[1458] server

[1459] The server receives the video data and audio data transmitted from the terminal.

[1460] Step 4:

[1461] server

[1462] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[1463] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[1464] Step 5:

[1465] server

[1466] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[1467] The analysis results are saved in a database.

[1468] Step 6:

[1469] server

[1470] The analysis results are input into a generation AI to generate the optimal background music or noise.

[1471] Step 7:

[1472] server

[1473] The generated sound data is sent to the terminal.

[1474] Step 8:

[1475] Terminal

[1476] The device plays background music or noise received from the server.

[1477] Step 9:

[1478] User

[1479] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[1480] Step 10:

[1481] Terminal

[1482] The subjective evaluation input by the user is sent to the server.

[1483] Step 11:

[1484] server

[1485] The server analyzes the received subjective evaluations.

[1486] Based on the feedback, adjust the parameters of the generated AI.

[1487] Step 12:

[1488] server

[1489] The next background music or noise is generated based on the adjusted parameters.

[1490] ---

[1491] The above is the specific flow of the program's processing, which allows the user to be provided with an optimal sleeping environment.

[1492] Example 1

[1493] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1494] Conventional sleep environment systems have struggled to provide optimal solutions for a user's individual sleep patterns or specific sleep problems. In particular, when it comes to generating background music or noise to improve sleep quality, they tend to use generic preset music or noises, and are unable to optimize them to meet the user's specific needs. Furthermore, there has been insufficient effort to effectively utilize subjective feedback from users to further optimize the next background music or noise.

[1495] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1496] In this invention, the server includes a means for periodically transmitting video and audio data, a means for analyzing the collected video and audio data to extract changes in the user's body position and audio events, and a means for evaluating the depth of sleep. This allows for a detailed analysis of the user's specific sleep patterns and problems, and based on that, optimal background music and noise can be generated. Furthermore, by adjusting the parameters of the generation AI model based on the user's subjective evaluation, the playback environment can be further optimized for subsequent playbacks.

[1497] "Video data" refers to all visual information acquired using a camera or other imaging device.

[1498] "Audio data" refers to all information relating to sound captured using a recording device such as a microphone.

[1499] "Transmitting means" refers to the technical methods or devices used to transmit collected data to another device or server at specified intervals.

[1500] "Means of analysis" refers to the programs or algorithms that process the collected data and extract useful information.

[1501] "Changes in body position" refers to changes in the posture or position of the user's body.

[1502] "Sleep depth" is an indicator of how deep a user's sleep state is, and is usually evaluated on a three-level scale: "light," "intermediate," and "deep."

[1503] "Generative AI model" refers to a general algorithm that uses artificial intelligence to generate new data or content.

[1504] "BGM" is short for background music and refers to music played to promote a particular atmosphere or emotion.

[1505] "Noise" refers to random or controlled sounds reproduced for a specific purpose.

[1506] "Subjective evaluation" refers to a personal evaluation made by a user based on their own experience and feelings.

[1507] "Parameters" refer to settings that adjust the behavior of a particular algorithm or model.

[1508] "Optimization" refers to the process of adjusting parameters or settings to achieve the most effective state or result for a particular purpose.

[1509] "Means for playback" refers to the technical methods and devices for actually outputting the generated sound data as sound.

[1510] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[1511] Data collection

[1512] Terminal

[1513] Before going to bed, the user launches the app and sets it to sleep mode.

[1514] The device is equipped with a camera and microphone, which are used to collect video and audio data while the person is sleeping.

[1515] The collected data is sent to the server at regular intervals. For example, the data is stored in a buffer every hour, and when a certain amount of data has accumulated, it is sent.

[1516] Specific examples

[1517] 1. The user opens the app before going to bed and selects sleep mode.

[1518] 2. The device activates its camera and microphone. The camera captures the user's posture and facial orientation, while the microphone records audio data such as snoring, talking in your sleep, and teeth grinding.

[1519] 3. The collected data is stored in a buffer at regular intervals, and the accumulated data is sent to the server.

[1520] Data analysis

[1521] server

[1522] The received video and audio data is analyzed using an advanced data analysis algorithm installed on the server, which is used to extract changes in the user's body position and audio events.

[1523] The video data is used to count the number of times the patient turns over and changes in position, while the audio data is used to detect the frequency of snoring, talking in one's sleep, and teeth grinding.

[1524] The depth of sleep is assessed on a three-point scale: light, medium, or deep. This assessment is based on the extracted data.

[1525] Specific examples

[1526] 1. The server analyzes the received video data and counts the number of times the person turns over.

[1527] 2. The server analyzes the audio data and detects the frequency of snoring and the number of times the person talks in their sleep.

[1528] 3. Based on this information, the depth of sleep is evaluated and stored in a database.

[1529] BGM / Noise Generation

[1530] server

[1531] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[1532] The generated sound data is sent to the terminal.

[1533] Specific examples

[1534] 1. The server inputs the following prompt to the generative AI model:

[1535] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[1536] 2. The generated sound data is sent to the device.

[1537] BGM / Noise playback

[1538] Terminal

[1539] The terminal plays the background music or noise received from the server.

[1540] The received sound data is played back at the appropriate time to optimize the user's sleep environment.

[1541] Specific examples

[1542] 1. The terminal prepares to play the sound data received from the server.

[1543] 2. Based on the collected data, background music or noise is played at the optimal timing.

[1544] Collecting user feedback

[1545] User

[1546] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[1547] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[1548] Terminal

[1549] The terminal transmits the subjective evaluation input by the user to the server.

[1550] Specific examples

[1551] 1. After waking up, the user opens the app and rates the quality of their sleep.

[1552] 2. The evaluation items are entered and the terminal sends them to the server.

[1553] Next optimization

[1554] server

[1555] The received subjective evaluations are analyzed and the parameters of the generative AI model for generating the next background music or noise are adjusted.

[1556] This allows background music or noise that is more suited to the user to be provided the next time the device is used.

[1557] Specific examples

[1558] 1. The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[1559] 2. Update the prompt text to generate optimized background music or noise the next time you use it.

[1560] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user in detail, it is possible to provide an individually optimized sleep environment.

[1561] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1562] Step 1: Launch and configure the app

[1563] Terminal

[1564] The user launches the app by tapping the app icon on their device.

[1565] Select "Sleep Mode" from the in-app menu.

[1566] If a popup appears requesting permission to use the camera and microphone, the user selects Allow.

[1567] Input and Output

[1568] Input: User touch actions (tapping app icons, selecting sleep mode)

[1569] Output: Camera and microphone are activated, and sleep data collection is complete.

[1570] Specific actions

[1571] 1. When a user opens the app and selects sleep mode, the device activates the camera and microphone.

[1572] 2. The camera monitors the user's position and the microphone begins collecting sound from within the room.

[1573] Step 2: Data collection

[1574] Terminal

[1575] The device's camera captures the user's posture and facial orientation, while the microphone records snoring, talking in one's sleep, and environmental sounds.

[1576] The collected data is temporarily stored in a buffer.

[1577] Input and Output

[1578] Input: Video and audio data from the camera and microphone

[1579] Output: Buffered video and audio data

[1580] Specific actions

[1581] 1. The camera captures multiple frames of video per second, and the microphone records audio at regular intervals.

[1582] 2. The collected video and audio data is stored in a buffer.

[1583] Step 3: Sending data

[1584] Terminal

[1585] The data stored in the buffer is sent to the server at regular intervals.

[1586] Input and Output

[1587] Input: Video and audio data stored in the buffer

[1588] Output: Video and audio data sent to the server

[1589] Specific actions

[1590] 1. When a certain amount of data accumulates in the buffer, the terminal compresses the data and transmits it.

[1591] 2. The server receives the data.

[1592] Step 4: Analyze the data

[1593] server

[1594] The received video and audio data is analyzed. The number of times the patient turns over in their sleep and changes in posture are extracted from the video data, and the frequency of snoring, talking in their sleep, and teeth grinding is detected from the audio data.

[1595] Sleep depth is evaluated on three levels: "light," "intermediate," and "deep."

[1596] Input and Output

[1597] Input: Video and audio data sent to the server

[1598] Output: Extracted body position change data, audio event data, sleep depth assessment

[1599] Specific actions

[1600] 1. The server analyzes the video data frame by frame and counts the user's changes in position and the number of times they turn over in bed.

[1601] 2. Analyze audio data to detect the frequency of snoring, sleep talking, and teeth grinding.

[1602] 3. Based on the extracted data, sleep depth is evaluated on a three-point scale and the results are stored in a database.

[1603] Step 5: Generate background music / noise

[1604] server

[1605] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[1606] The generated sound data is sent to the terminal.

[1607] Input and Output

[1608] Input: Analysis result data, prompt statement

[1609] Output: Generated background music or noise

[1610] Specific actions

[1611] 1. The server inputs the following prompt to the generative AI model:

[1612] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[1613] 2. The generative AI model generates optimal sound data and sends it to the device.

[1614] Step 6: Play background music / noise

[1615] Terminal

[1616] The device plays background music or noise received from the server.

[1617] Input and Output

[1618] Input: BGM or noise sent from the server

[1619] Output: BGM or noise to be played

[1620] Specific actions

[1621] 1. The terminal prepares to play the sound data received from the server.

[1622] 2. Play background music or noise at specified times to optimize the user's sleep environment.

[1623] Step 7: Gather user feedback

[1624] User

[1625] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[1626] Input and Output

[1627] Input: User subjective evaluation data

[1628] Output: Subjective evaluation data sent to the server

[1629] Specific actions

[1630] 1. The user opens the app and enters information about their sleep quality (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[1631] 2. The device sends the entered evaluation data to the server.

[1632] Step 8: Next optimization

[1633] server

[1634] The received subjective evaluations are analyzed and the parameters of the generative AI model are adjusted.

[1635] Input and Output

[1636] Input: User subjective evaluation data

[1637] Output: Adjusted generative AI model parameters

[1638] Specific actions

[1639] 1. The server analyzes the received subjective evaluation data and identifies which elements were problematic.

[1640] 2. Adjust the parameters of the generative AI model for the next BGM or noise generation.

[1641] The above is the specific flow of the program processing of this system. Through each processing step, it is possible to effectively collect and analyze the user's sleep data and provide an individually optimized sleep environment.

[1642] (Application example 1)

[1643] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1644] Conventional sleep improvement systems only collect users' sleep data and rarely provide personalized, optimized responses based on that data. Furthermore, they lack real-time feedback and improvement measures based on analyzed data, which means users' sleep quality is not fully improved. Furthermore, there are insufficient means for effectively utilizing subjective feedback.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1646] In this invention, the server includes means for collecting user sleep data, means for transmitting the collected video data and audio data, means for analyzing the transmitted data, a generative AI model that generates optimal background music or noise based on the data obtained from the analysis means, means for playing the generated background music or noise, means for the user to input a subjective evaluation of sleep quality, and means for adjusting the parameters of the generative AI model based on the subjective evaluation. This provides an optimal sleep environment tailored to each user's individual sleep pattern, and enables further optimization based on feedback.

[1647] "User's sleep data" refers to information including video data and audio data that indicate the user's sleep state.

[1648] "Video data" refers to video data that records the user's sleeping posture, the number of times they turn over in their sleep, and so on.

[1649] "Voice data" refers to audio data including the user's snoring, talking in their sleep, environmental sounds, and the like.

[1650] "Transmitting means" refers to the functions and processes for transferring collected video and audio data to a server.

[1651] "Means of analysis" refers to the process by which the server handles video and audio data and extracts and analyzes the number of times the patient turns over in bed and the frequency of snoring, etc.

[1652] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal background music or noise based on analytical results and subjective evaluation.

[1653] "BGM or Noise" refers to sound data such as music or environmental sounds created by a generative AI model that are played to improve a user's sleep.

[1654] "Means for playback" refers to the functions or devices for playing background music or noise generated on a device such as a smartphone.

[1655] "Subjective evaluation" refers to the user inputting their impression of the quality of sleep as a numerical value or a comment.

[1656] "Means for adjusting the parameters of the generative AI model" refers to the process of changing the settings of the generative AI model based on the user's subjective evaluation and reflecting this in the next background music or noise generation.

[1657] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[1658] 1. Data Collection

[1659] Terminal

[1660] Before going to bed, the user launches the app and sets it to sleep mode.

[1661] The device's camera and microphone are used to collect video and audio data while you sleep.

[1662] The collected data is sent to a server at regular intervals.

[1663] Specific examples

[1664] The user opens the app and selects sleep mode.

[1665] The device will activate its camera and microphone and begin collecting video and audio data.

[1666] The collected data is sent to a server.

[1667] 2. Data Analysis

[1668] server

[1669] The server analyzes the received video and audio data.

[1670] The number of times the patient turns over and changes in position are extracted from the video data.

[1671] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[1672] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[1673] The analysis results are stored in a database.

[1674] Specific examples

[1675] The server analyzes the video data and counts the number of times the person turns over.

[1676] The server analyzes the audio data and detects the frequency of snoring.

[1677] The server evaluates the depth of sleep and stores it in a database.

[1678] 3. BGM / Noise Generation and Playback

[1679] server

[1680] The server uses a generative AI model to generate optimal background music or noise based on the analysis results.

[1681] The generated sound data is transmitted to the terminal.

[1682] Terminal

[1683] The terminal plays the background music or noise received from the server.

[1684] Specific examples

[1685] The server inputs the analysis results into a generative AI model to generate background music or noise.

[1686] The generated sound data is transmitted to the terminal.

[1687] The terminal plays back the received sound data.

[1688] 4. User Feedback

[1689] User

[1690] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[1691] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[1692] Terminal

[1693] The terminal transmits the subjective evaluation input by the user to the server.

[1694] Specific examples

[1695] After waking up, the user opens the app and rates the quality of their sleep.

[1696] The user enters a rating, which the terminal sends to the server.

[1697] 5. Next optimization

[1698] server

[1699] The server analyzes the received subjective evaluations and adjusts the parameters of the generative AI model for generating the next background music or noise.

[1700] This allows the next sleep environment to be more tailored to the user.

[1701] Specific examples

[1702] The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[1703] Based on the analysis, optimized background music or noise will be generated the next time you use it.

[1704] Hardware and software used

[1705] Hardware: Smartphone (camera, microphone, speaker)

[1706] Software: Python, OpenCV (cv2), Request, SoundDevice (sd), server backend (Flask or Django)

[1707] Data processing and calculation

[1708] 1. Data Collection:

[1709] Video and audio data is collected using the smartphone's camera and microphone.

[1710] The data is uploaded to the server as image or audio files at regular intervals.

[1711] 2. Data Analysis:

[1712] The server analyzes the received data and detects things like turning over in bed and snoring.

[1713] The data is stored in a database for evaluating the depth of sleep.

[1714] 3. BGM / Noise Generation and Playback:

[1715] The generative AI model is run on the server to generate optimal background music and noise.

[1716] The sound data is sent to a smartphone and played at the specified time.

[1717] 4. User Feedback:

[1718] Collect user evaluation data from the smartphone interface.

[1719] The feedback data is sent to the server and used for the next BGM generation.

[1720] Prompt Sentence Examples

[1721] Below are some examples of prompts to input to the generative AI model.

[1722] We analyzed the user's sleep data. The frequency of snoring was high, and the average number of turns was three. The user's sleep depth was shallow, so please generate alpha wave music that has a relaxing effect.

[1723] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[1724] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1725] Step 1:

[1726] Before going to bed, the user launches the app and sets it to sleep mode.

[1727] Input: User launches the app and selects sleep mode.

[1728] Output: The device is ready to start collecting data.

[1729] What happens: The user launches the app on their smartphone and selects "sleep mode," which causes the app to turn on the camera and microphone and prepare for data collection.

[1730] Step 2:

[1731] The device's camera and microphone are used to collect video and audio data while you sleep.

[1732] Input: The app activates the camera and microphone.

[1733] Output: Collected video and audio data.

[1734] How it works: The device's camera detects the user's posture and tossing and turning while sleeping, and the microphone detects snoring and talking in their sleep. This data is temporarily saved as image and audio files.

[1735] Step 3:

[1736] The collected data is sent to the server at regular intervals.

[1737] Input: Collected video and audio data.

[1738] Output: Data uploaded to the server.

[1739] How it works: Every 60 seconds, the device uploads the collected data to the server. Data transfer is performed via an internet connection and an API request.

[1740] Step 4:

[1741] The server parses the received data.

[1742] Input: Video and audio data uploaded to the server.

[1743] Output: Analysis results such as number of turns, frequency of snoring, and sleep depth.

[1744] How it works: The server analyzes the video data and counts the number of times the person turns over in bed and changes their position. Similarly, it extracts the frequency of snoring and talking in their sleep from the audio data. Based on this, it evaluates the depth of sleep and stores the results in a database.

[1745] Step 5:

[1746] Uses generative AI models to generate optimal background music or noise based on the analysis results.

[1747] Input: Analysis results (number of turns, frequency of snoring, depth of sleep).

[1748] Output: Generated background music or noise sound data.

[1749] Specific operation: The server inputs the analysis results into the generative AI model and generates a prompt. For example, it might input, "We analyzed the user's sleep data. We found that the user snores frequently and turns over an average of three times. The user's sleep depth is shallow, so please generate alpha wave music that has a relaxing effect." The generative AI model then generates music data based on the instructions.

[1750] Step 6:

[1751] The generated sound data is sent to the terminal and played back.

[1752] Input: Sound data for the generated background music or noise.

[1753] Output: Sound data played on the device.

[1754] Specific operation: The server sends the generated sound data to the device, which then plays the received sound data in real time to help the user sleep.

[1755] Step 7:

[1756] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[1757] Input: User's subjective evaluation (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[1758] Output: Subjective evaluation data sent to the server.

[1759] How it works: The user opens the app and inputs their sleep quality rating, including how easily they fell asleep, whether they woke up during the night, and how easily they woke up, using numerical values ​​and comments. The device then sends this data to the server.

[1760] Step 8:

[1761] The server analyzes the subjective evaluations it receives and adjusts the parameters of the generative AI model.

[1762] Input: Subjective evaluation data sent to the server.

[1763] Output: The adjusted parameters of the generative AI model.

[1764] How it works: The server analyzes the user's subjective evaluation data and adjusts the parameters of the generative AI model, which then generates more appropriate background music or noise for the next session, improving the user's sleep quality.

[1765] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1766] ---

[1767] This invention is a system that collects and analyzes a user's sleep data and combines it with an emotion engine that recognizes the user's emotions to generate optimal sleep-inducing background music and noises. It mainly functions in cooperation with the terminal, server, and user. The specific operation of each element is explained below.

[1768] Data collection

[1769] Terminal

[1770] Before going to bed, the user launches the app and sets it to sleep mode.

[1771] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[1772] The collected data is sent to a server at regular intervals.

[1773] Specific examples

[1774] 1. The user opens the app and selects sleep mode.

[1775] 2. The device activates the camera and microphone and begins collecting video and audio data.

[1776] 3. The collected data is sent to the server.

[1777] Data analysis

[1778] server

[1779] The server analyzes the received video and audio data.

[1780] The number of times the patient turns over and changes in position are extracted from the video data.

[1781] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[1782] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[1783] The analysis results are stored in a database.

[1784] Specific examples

[1785] 1. The server analyzes the video data and counts the number of times the person turns over.

[1786] 2. The server analyzes the audio data and detects the frequency of snoring.

[1787] 3. The server evaluates the sleep depth and stores it in a database.

[1788] emotion recognition

[1789] Terminal

[1790] The data collected by the device is sent to the emotion engine in real time.

[1791] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[1792] server

[1793] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[1794] Specific examples

[1795] 1. While the user is sleeping, the device sends facial expressions and voice to the emotion engine.

[1796] 2. The emotion engine analyzes and recognizes the user's emotional state, whether relaxed or anxious.

[1797] 3. The server receives the emotion information and integrates it into the analysis data.

[1798] BGM / noise generation and playback

[1799] server

[1800] The server uses generative AI to generate optimal background music or noise based on the analysis results and emotional information.

[1801] The generated sound data is transmitted to the terminal.

[1802] Terminal

[1803] The device plays the background music or noise received from the server.

[1804] Specific examples

[1805] 1. The server inputs the analysis results and emotional information into the generation AI, which then generates background music or noise.

[1806] 2. The generated sound data is sent to the device.

[1807] 3. The device plays the received sound data.

[1808] User Feedback

[1809] User

[1810] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[1811] Terminal

[1812] The subjective evaluation input by the user is sent to the server.

[1813] Specific examples

[1814] 1. After waking up, the user opens the app and rates their sleep quality and emotions.

[1815] 2. The user enters a rating, and the device sends it to the server.

[1816] Next optimization

[1817] server

[1818] The server analyzes the subjective evaluations received and adjusts the parameters of the generation AI for generating the next background music or noise.

[1819] This allows the next sleep environment to be more tailored to the user.

[1820] Specific examples

[1821] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[1822] 2. Based on the analysis, generate optimized background music or noise for the next use.

[1823] ---

[1824] The above is an embodiment of the present invention. By analyzing the sleep data and emotional state of the user, it is possible to provide an optimal sleeping environment for each individual.

[1825] The processing flow will be explained below.

[1826] ---

[1827] Step 1:

[1828] Terminal

[1829] The user launches the app and sets it to sleep mode.

[1830] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[1831] Step 2:

[1832] Terminal

[1833] The collected video and audio data is sent to the server at regular intervals.

[1834] Step 3:

[1835] server

[1836] The server receives the video data and audio data transmitted from the terminal.

[1837] Step 4:

[1838] server

[1839] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[1840] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[1841] Step 5:

[1842] server

[1843] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[1844] The analysis results are saved in a database.

[1845] Step 6:

[1846] Terminal

[1847] The collected data is sent to the emotion engine in real time.

[1848] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[1849] Step 7:

[1850] server

[1851] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[1852] Step 8:

[1853] server

[1854] The analysis results and emotional information are input into a generation AI to generate optimal background music or noise.

[1855] Step 9:

[1856] server

[1857] The generated sound data is sent to the terminal.

[1858] Step 10:

[1859] Terminal

[1860] The device plays background music or noise received from the server.

[1861] Step 11:

[1862] User

[1863] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[1864] Step 12:

[1865] Terminal

[1866] The subjective evaluation input by the user is sent to the server.

[1867] Step 13:

[1868] server

[1869] The server analyzes the received subjective evaluations.

[1870] Based on the feedback, adjust the parameters of the generated AI.

[1871] Step 14:

[1872] server

[1873] The next background music or noise is generated based on the adjusted parameters.

[1874] ---

[1875] The above is the specific processing flow of the invention combined with the emotion engine, which makes it possible to provide an optimal sleeping environment that reflects the user's emotional state.

[1876] Example 2

[1877] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1878] Conventional sleep environment improvement systems simply collect and analyze users' sleep data, and are limited in their ability to generate optimal sleep-inducing background music or noise that takes into account the user's emotional state. Furthermore, they lack the ability to optimize the next sound environment based on the user's subjective evaluation. This makes it difficult to provide a sleep environment that is suited to each individual user.

[1879] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing collected video data and audio data, a means for recognizing the user's emotional state, and a means for generating optimal background music or noise based on the data obtained from the analysis means and the emotion recognition means. This makes it possible to comprehensively analyze the user's sleep data and emotional state and provide individually optimized sleep-inducing background music or noise.

[1880] "Video data" refers to visual information that records the user's appearance and movements while sleeping.

[1881] "Audio data" refers to audio information that records sounds that occur during sleep, such as the user talking in their sleep or snoring.

[1882] "Means of collection" refers to the hardware and software used to acquire and record video and audio data.

[1883] "Means for analysis" refers to software that processes and analyzes the collected video and audio data to extract information such as the number of times the patient turns over in their sleep and the frequency of snoring.

[1884] "Means for recognizing emotional state" refers to software and hardware that analyzes the user's facial expressions and tone of voice to identify emotions such as relaxed or anxious.

[1885] "Means for generating optimal background music or noise" refers to a generative AI model that generates music or sounds that are most relaxing for the user based on analysis results and emotional state data.

[1886] "Means for playing" refers to devices and software for outputting the generated background music or noise so that the user can hear it.

[1887] "Means for inputting subjective evaluation" refers to an interface and software that allows the user to input feedback on sleep quality and emotional state after waking up.

[1888] The "means for optimizing the generation means" refers to algorithms and software for adjusting the next background music or noise to better suit the user based on the user's subjective evaluation.

[1889] The "transmitting means" refers to a communication interface and software for transmitting the video data and audio data collected by the terminal to the server in real time.

[1890] A "generative AI model" is an artificial intelligence model that generates optimal background music and noise based on the user's analytical data and emotional data.

[1891] A "prompt sentence" is an input sentence that instructs the generative AI model on what background music or noise to generate.

[1892] This invention is a system that collects a user's sleep data, analyzes that data, and combines it with an emotion engine that recognizes the user's emotional state to generate and play optimal sleep-inducing background music and noise. The system is primarily composed of a terminal, a server, and a user. The specific operation of each element is explained below.

[1893] Data collection

[1894] Terminal

[1895] Users use a smartphone with a dedicated sleep tracking app installed, launch the app before going to bed, and select sleep mode.

[1896] The device automatically activates the camera and microphone to collect video and audio data while the user sleeps, including recording the user's movements, changes in posture, snoring, and talking in their sleep.

[1897] The collected data is sent from the terminal to the server at regular intervals.

[1898] Examples:

[1899] 1. The user opens the app and selects "Sleep mode," which causes the device to activate the camera and microphone and begin collecting data.

[1900] 2. The collected video and audio data is sent to the server in real time.

[1901] Data analysis

[1902] server

[1903] The server uses a high-performance data analysis program to detect the number of times the patient turns over and changes in posture from the received video data, and extracts information such as snoring, talking in one's sleep, and teeth grinding from the audio data.

[1904] Based on the extracted data, the user's sleep depth is evaluated on a three-point scale: "light," "medium," or "deep." The evaluation results are stored in a database and used for subsequent processing.

[1905] Examples:

[1906] 1. The server processes the video data and identifies the number of times the user has turned over.

[1907] 2. Analyze audio data and record the frequency and volume of snoring.

[1908] 3. The sleep depth assessment results are stored in a database.

[1909] emotion recognition

[1910] Terminal

[1911] The device sends the collected video and audio data in real time to an emotion recognition engine, which analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[1912] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server.

[1913] server

[1914] The server integrates the emotional state data received from the emotion recognition engine into the analysis data to assess the user's overall sleep state.

[1915] Examples:

[1916] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[1917] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[1918] 3. Emotional state data is sent to the server and integrated into the analysis data.

[1919] BGM / noise generation and playback

[1920] server

[1921] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. The generative AI model receives a prompt and generates the optimal sound for the user.

[1922] The generated sound data is transmitted to the terminal.

[1923] Terminal

[1924] The device plays background music or noise received from the server to help the user sleep.

[1925] Examples:

[1926] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[1927] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[1928] 3. The server sends the sound data to the device, which then plays it.

[1929] User Feedback

[1930] User

[1931] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state, based on their sleep experience.

[1932] Terminal

[1933] The subjective evaluation input by the user is sent to the server.

[1934] Examples:

[1935] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[1936] 2. The device sends the evaluation data to the server.

[1937] Next optimization

[1938] server

[1939] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. This feedback optimizes the next BGM and noise to better suit the user.

[1940] Examples:

[1941] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[1942] 2. Provide that feedback to the generative AI model and use it for the next generation.

[1943] In this way, the present invention can comprehensively analyze the user's sleep data and emotional state, and provide sleep-inducing background music and noises that are optimized for each individual user.

[1944] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1945] Step 1: Data collection

[1946] The device installs a dedicated sleep tracking app on the user, and has them launch the app and select a sleep mode before going to bed. Input: The user operates the app. Output: The device turns on the camera and microphone.

[1947] The device uses a camera and microphone to collect video and audio data in real time while the user sleeps. Input: The user's movements, changes in posture, snoring, and sleep talking. Output: Collected video and audio data.

[1948] The collected data is sent to the server at regular intervals. Input: Collected video and audio data. Output: Data is sent to the server.

[1949] Specific behavior:

[1950] 1. The user opens the app and selects "Sleep Mode."

[1951] 2. The device activates the camera and microphone and begins collecting data while the user sleeps.

[1952] 3. Upload the collected data to the server at regular intervals.

[1953] Step 2: Data analysis

[1954] The server analyzes the received video data and detects the number of turns and changes in body position. Input: Video data. Output: Data on the number of turns and changes in body position.

[1955] The server analyzes the audio data and extracts information such as snoring frequency, sleep talking, teeth grinding, etc. Input: Audio data. Output: Snoring frequency and pattern data.

[1956] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep." Input: Analyzed data. Output: Sleep depth evaluation data.

[1957] The analysis results are saved in a database. Input: Sleep depth evaluation data. Output: Data saved in the database.

[1958] Specific behavior:

[1959] 1. The server analyzes the video data and identifies the number of times and duration that the user turned over in their sleep.

[1960] 2. The server analyzes the audio data and records the frequency and patterns of snoring.

[1961] 3. The server evaluates the sleep depth and stores the results in a database.

[1962] Step 3: Emotion Recognition

[1963] The device sends the collected video and audio data to the emotion recognition engine in real time. Input: Collected data. Output: Data sent to the emotion recognition engine.

[1964] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. Input: Video data and audio data. Output: Emotional state data.

[1965] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server. Input: Emotional state data. Output: Emotional information sent to the server.

[1966] Specific behavior:

[1967] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[1968] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[1969] 3. Emotional state data is sent to the server and integrated into the analysis data.

[1970] Step 4: Generate and play background music / noise

[1971] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. Input: Analysis results and emotional state data. Output: Generated background music or noise data.

[1972] The generative AI model generates the optimal sound by inputting an appropriate prompt sentence. For example, "Please generate music that will relax the user." Input: Prompt sentence. Output: Generated sound data.

[1973] The generated sound data is sent to the terminal. Input: Generated sound data. Output: Sound data sent to the terminal.

[1974] The device plays background music or noise received from the server. Input: Sound data from the server. Output: Played sound.

[1975] Specific behavior:

[1976] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[1977] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[1978] 3. The server sends the sound data to the device, which then plays it.

[1979] Step 5: User feedback

[1980] After waking up, the user opens the app and inputs their subjective assessment of their sleep quality and emotional state. Input: Input of subjective assessment. Output: Inputted assessment data.

[1981] The device sends the user's subjective evaluation to the server. Input: User's subjective evaluation data. Output: Evaluation data sent to the server.

[1982] Specific behavior:

[1983] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[1984] 2. The device sends the evaluation data to the server.

[1985] Step 6: Next optimization

[1986] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. Input: Subjective evaluation data. Output: Adjusted parameters of the generative AI model.

[1987] This will optimize the background music and noise for the next time to better suit the user. Input: Adjusted parameters. Output: Optimized sound environment for the next use.

[1988] Specific behavior:

[1989] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[1990] 2. Provide that feedback to the generative AI model, and adjust the parameters in the next generation to provide an optimized sound environment.

[1991] (Application example 2)

[1992] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1993] In modern society, many people find it difficult to get a good night's sleep due to stress and irregular living conditions. In this situation, there is a need for systems that support sleep. In particular, there is a need to provide sleep-inducing background music and noises that are optimized for each user's individual condition, thereby enabling users to sleep more comfortably.

[1994] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and audio data, means for generating optimal background music or noise based on the data and emotional information obtained from the analysis means, and means for adjusting the prompts of the generative AI model based on the user's subjective evaluation. This makes it possible to analyze the user's sleep data and emotional state in detail and generate and provide optimal sleep-inducing background music or noise for each individual user based on the analysis results.

[1995] "Video data" refers to information in the form of moving images collected by a photographing device such as a camera.

[1996] "Audio data" refers to information in the form of audio collected by a sound collection device such as a microphone.

[1997] The "analysis means" is a method or device for extracting specific information based on the collected video and audio data.

[1998] "Emotion information" is data relating to the user's emotional state analyzed from facial expressions, tone of voice, and the like.

[1999] "Optimal background music or noise" refers to background music or noise that is generated to support the user's sleep and is most suitable for the user's state.

[2000] "Subjective evaluation" refers to evaluation data input by the user based on their own feelings, and includes information on the quality of sleep, emotional state, and the like.

[2001] A "generative AI model" is an artificial intelligence model that generates specific content based on input data.

[2002] A "prompt" is input data for a generative AI model, and includes instructions and conditions regarding the content to be generated.

[2003] "Collecting means" refers to a method or device for acquiring video data and audio data.

[2004] "Transmitting means" refers to a method or device for sending collected data to another device or server.

[2005] This invention is a system that collects and analyzes a user's sleep data and recognizes the user's emotions. It mainly functions in cooperation with a terminal, a server, and a user. The specific operation of each element is described below.

[2006] Data collection

[2007] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the camera and microphone and collects video and audio data while the device is sleeping. This collected data is then sent to a server at regular intervals.

[2008] The specific hardware used is the smartphone camera and microphone. Audio data is collected using the sounddevice library, and video data is collected using cv2 (OpenCV).

[2009] Data analysis

[2010] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. This information is used to evaluate the depth of sleep on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[2011] The analysis tools used are cv2 for video data analysis and numpy for audio data analysis, while databases such as MySQL and PostgreSQL are used.

[2012] emotion recognition

[2013] The device sends the collected data to the emotion engine in real time, and analyzes the user's facial expressions and tone of voice to recognize their emotional state. This emotional information is then sent to the server and integrated into the analyzed data.

[2014] For emotion recognition, a library called DeepFace is used, which allows the system to recognize the user's emotional state, such as whether they are relaxed or anxious.

[2015] BGM / noise generation and playback

[2016] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[2017] An example of a generative AI model would be GPT-3. An example prompt would be:

[2018] User Emotion: Relaxed, Sleep State: Deep

[2019] Please create the best background music for this situation.

[2020] User Feedback

[2021] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent to the server.

[2022] Next optimization

[2023] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model for generating the next background music or noise, so that the next sleep environment is more tailored to the user.

[2024] As a specific example, if a user evaluates that they are "in a relaxed state and in a deep sleep," the AI ​​model will be adjusted based on that data so that similar background music will be generated the next time.

[2025] In this way, the present invention can provide the user with an optimal sleeping environment.

[2026] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2027] Step 1:

[2028] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the smartphone's camera and microphone and starts collecting video and audio data. This collected data is sent to the server every five seconds.

[2029] Specific input: The user's action to set the application to sleep mode.

[2030] Specific operation: The camera and microphone are activated and video and audio data is collected.

[2031] Specific output: Video and audio data collected in real time.

[2032] Step 2:

[2033] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. Based on this information, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[2034] Specific input: video data, audio data.

[2035] Specific operations: Analyze body movements from video data using cv2, and detect snoring and sleep talking from audio data using numpy.

[2036] Specific output: Sleep depth assessment results and a database storing them.

[2037] Step 3:

[2038] The device collects data in real time and sends the user's facial expressions and tone of voice to the emotion engine to analyze the user's emotional state. The server receives the emotional information and integrates it into the analysis data.

[2039] Specific input: collected video and audio data.

[2040] Specific operation: Recognizes emotions by analyzing facial expressions and tone of voice using the DeepFace library.

[2041] Specific output: Recognized emotional information.

[2042] Step 4:

[2043] The server uses a generative AI model to generate the optimal background music or noise for the user based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[2044] Specific input: Sleep depth assessment results and emotional information.

[2045] Specific operation: A prompt sentence is input into the generative AI model to generate background music or noise. Example prompt sentence: "User's emotion: Relaxed, Sleep state: Deep. Please generate background music that is optimal for this state."

[2046] Specific output: Generated background music or noise data.

[2047] Step 5:

[2048] The device plays the generated background music or noise to help the user sleep.

[2049] Specific input: generated background music or noise data.

[2050] Specific operation: Plays sound using the device's audio function.

[2051] Specific output: background music or noise heard by the user.

[2052] Step 6:

[2053] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent from the device to the server.

[2054] Specific input: Subjective evaluation entered by the user.

[2055] Specific operation: Enter sleep quality and emotional state on the application's evaluation screen and send the data.

[2056] Specific output: Subjective evaluation data sent to the server.

[2057] Step 7:

[2058] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model to generate background music or noise for the next use, thereby providing a more suitable environment for the next use.

[2059] Specific input: Subjective evaluation data submitted by users.

[2060] Specific operation: Analyze subjective evaluations and adjust the prompt parameters of the generative AI model.

[2061] Specific output: The background music or noise data that will be generated next time based on the adjusted prompt.

[2062] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2063] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2064] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2065] [Fourth embodiment]

[2066] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2067] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2068] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2069] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2070] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2071] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2072] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2073] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2074] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2075] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2076] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2077] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2078] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2079] ---

[2080] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[2081] Data collection

[2082] Terminal

[2083] Before going to bed, the user launches the app and sets it to sleep mode.

[2084] The device's camera and microphone are used to collect video and audio data while you sleep.

[2085] The collected data is sent to a server at regular intervals.

[2086] Specific examples

[2087] 1. The user opens the app and selects sleep mode.

[2088] 2. The device activates the camera and microphone and begins collecting video and audio data.

[2089] 3. The collected data is sent to the server.

[2090] Data analysis

[2091] server

[2092] The server analyzes the received video and audio data.

[2093] The number of times the patient turns over and changes in position are extracted from the video data.

[2094] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[2095] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[2096] The analysis results are stored in a database.

[2097] Specific examples

[2098] 1. The server analyzes the video data and counts the number of times the person turns over.

[2099] 2. The server analyzes the audio data and detects the frequency of snoring.

[2100] 3. The server evaluates the sleep depth and stores it in a database.

[2101] BGM / noise generation and playback

[2102] server

[2103] The server uses a generative AI to generate optimal background music or noise based on the analysis results.

[2104] The generated sound data is transmitted to the terminal.

[2105] Terminal

[2106] The terminal plays the background music or noise received from the server.

[2107] Specific examples

[2108] 1. The server inputs the analysis results into the generation AI, which generates background music or noise.

[2109] 2. The generated sound data is sent to the device.

[2110] 3. The device plays the received sound data.

[2111] User Feedback

[2112] User

[2113] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[2114] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[2115] Terminal

[2116] The terminal transmits the subjective evaluation input by the user to the server.

[2117] Specific examples

[2118] 1. After waking up, the user opens the app and rates the quality of their sleep.

[2119] 2. The user enters a rating, and the device sends it to the server.

[2120] Next optimization

[2121] server

[2122] The server analyzes the received subjective evaluations and adjusts the parameters of the generation AI for generating the next background music or noise.

[2123] This allows the next sleep environment to be more tailored to the user.

[2124] Specific examples

[2125] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[2126] 2. Based on the analysis, generate optimized background music or noise for the next use.

[2127] ---

[2128] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[2129] The processing flow will be explained below.

[2130] ---

[2131] Step 1:

[2132] Terminal

[2133] The user launches the app and sets it to sleep mode.

[2134] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[2135] Step 2:

[2136] Terminal

[2137] The collected video and audio data is sent to the server at regular intervals.

[2138] Step 3:

[2139] server

[2140] The server receives the video data and audio data transmitted from the terminal.

[2141] Step 4:

[2142] server

[2143] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[2144] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[2145] Step 5:

[2146] server

[2147] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[2148] The analysis results are saved in a database.

[2149] Step 6:

[2150] server

[2151] The analysis results are input into a generation AI to generate the optimal background music or noise.

[2152] Step 7:

[2153] server

[2154] The generated sound data is sent to the terminal.

[2155] Step 8:

[2156] Terminal

[2157] The device plays background music or noise received from the server.

[2158] Step 9:

[2159] User

[2160] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[2161] Step 10:

[2162] Terminal

[2163] The subjective evaluation input by the user is sent to the server.

[2164] Step 11:

[2165] server

[2166] The server analyzes the received subjective evaluations.

[2167] Based on the feedback, adjust the parameters of the generated AI.

[2168] Step 12:

[2169] server

[2170] The next background music or noise is generated based on the adjusted parameters.

[2171] ---

[2172] The above is the specific flow of the program's processing, which allows the user to be provided with an optimal sleeping environment.

[2173] Example 1

[2174] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2175] Conventional sleep environment systems have struggled to provide optimal solutions for a user's individual sleep patterns or specific sleep problems. In particular, when it comes to generating background music or noise to improve sleep quality, they tend to use generic preset music or noises, and are unable to optimize them to meet the user's specific needs. Furthermore, there has been insufficient effort to effectively utilize subjective feedback from users to further optimize the next background music or noise.

[2176] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2177] In this invention, the server includes a means for periodically transmitting video and audio data, a means for analyzing the collected video and audio data to extract changes in the user's body position and audio events, and a means for evaluating the depth of sleep. This allows for a detailed analysis of the user's specific sleep patterns and problems, and based on that, optimal background music and noise can be generated. Furthermore, by adjusting the parameters of the generation AI model based on the user's subjective evaluation, the playback environment can be further optimized for subsequent playbacks.

[2178] "Video data" refers to all visual information acquired using a camera or other imaging device.

[2179] "Audio data" refers to all information relating to sound captured using a recording device such as a microphone.

[2180] "Transmitting means" refers to the technical methods or devices used to transmit collected data to another device or server at specified intervals.

[2181] "Means of analysis" refers to the programs or algorithms that process the collected data and extract useful information.

[2182] "Changes in body position" refers to changes in the posture or position of the user's body.

[2183] "Sleep depth" is an indicator of how deep a user's sleep state is, and is usually evaluated on a three-level scale: "light," "intermediate," and "deep."

[2184] "Generative AI model" refers to a general algorithm that uses artificial intelligence to generate new data or content.

[2185] "BGM" is short for background music and refers to music played to promote a particular atmosphere or emotion.

[2186] "Noise" refers to random or controlled sounds reproduced for a specific purpose.

[2187] "Subjective evaluation" refers to a personal evaluation made by a user based on their own experience and feelings.

[2188] "Parameters" refer to settings that adjust the behavior of a particular algorithm or model.

[2189] "Optimization" refers to the process of adjusting parameters or settings to achieve the most effective state or result for a particular purpose.

[2190] "Means for playback" refers to the technical methods and devices for actually outputting the generated sound data as sound.

[2191] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[2192] Data collection

[2193] Terminal

[2194] Before going to bed, the user launches the app and sets it to sleep mode.

[2195] The device is equipped with a camera and microphone, which are used to collect video and audio data while the person is sleeping.

[2196] The collected data is sent to the server at regular intervals. For example, the data is stored in a buffer every hour, and when a certain amount of data has accumulated, it is sent.

[2197] Specific examples

[2198] 1. The user opens the app before going to bed and selects sleep mode.

[2199] 2. The device activates its camera and microphone. The camera captures the user's posture and facial orientation, while the microphone records audio data such as snoring, talking in your sleep, and teeth grinding.

[2200] 3. The collected data is stored in a buffer at regular intervals, and the accumulated data is sent to the server.

[2201] Data analysis

[2202] server

[2203] The received video and audio data is analyzed using an advanced data analysis algorithm installed on the server, which is used to extract changes in the user's body position and audio events.

[2204] The video data is used to count the number of times the patient turns over and changes in position, while the audio data is used to detect the frequency of snoring, talking in one's sleep, and teeth grinding.

[2205] The depth of sleep is assessed on a three-point scale: light, medium, or deep. This assessment is based on the extracted data.

[2206] Specific examples

[2207] 1. The server analyzes the received video data and counts the number of times the person turns over.

[2208] 2. The server analyzes the audio data and detects the frequency of snoring and the number of times the person talks in their sleep.

[2209] 3. Based on this information, the depth of sleep is evaluated and stored in a database.

[2210] BGM / Noise Generation

[2211] server

[2212] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[2213] The generated sound data is sent to the terminal.

[2214] Specific examples

[2215] 1. The server inputs the following prompt to the generative AI model:

[2216] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[2217] 2. The generated sound data is sent to the device.

[2218] BGM / Noise playback

[2219] Terminal

[2220] The terminal plays the background music or noise received from the server.

[2221] The received sound data is played back at the appropriate time to optimize the user's sleep environment.

[2222] Specific examples

[2223] 1. The terminal prepares to play the sound data received from the server.

[2224] 2. Based on the collected data, background music or noise is played at the optimal timing.

[2225] Collecting user feedback

[2226] User

[2227] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[2228] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[2229] Terminal

[2230] The terminal transmits the subjective evaluation input by the user to the server.

[2231] Specific examples

[2232] 1. After waking up, the user opens the app and rates the quality of their sleep.

[2233] 2. The evaluation items are entered and the terminal sends them to the server.

[2234] Next optimization

[2235] server

[2236] The received subjective evaluations are analyzed and the parameters of the generative AI model for generating the next background music or noise are adjusted.

[2237] This allows background music or noise that is more suited to the user to be provided the next time the device is used.

[2238] Specific examples

[2239] 1. The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[2240] 2. Update the prompt text to generate optimized background music or noise the next time you use it.

[2241] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user in detail, it is possible to provide an individually optimized sleep environment.

[2242] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2243] Step 1: Launch and configure the app

[2244] Terminal

[2245] The user launches the app by tapping the app icon on their device.

[2246] Select "Sleep Mode" from the in-app menu.

[2247] If a popup appears requesting permission to use the camera and microphone, the user selects Allow.

[2248] Input and Output

[2249] Input: User touch actions (tapping app icons, selecting sleep mode)

[2250] Output: Camera and microphone are activated, and sleep data collection is complete.

[2251] Specific actions

[2252] 1. When a user opens the app and selects sleep mode, the device activates the camera and microphone.

[2253] 2. The camera monitors the user's position and the microphone begins collecting sound from within the room.

[2254] Step 2: Data collection

[2255] Terminal

[2256] The device's camera captures the user's posture and facial orientation, while the microphone records snoring, talking in one's sleep, and environmental sounds.

[2257] The collected data is temporarily stored in a buffer.

[2258] Input and Output

[2259] Input: Video and audio data from the camera and microphone

[2260] Output: Buffered video and audio data

[2261] Specific actions

[2262] 1. The camera captures multiple frames of video per second, and the microphone records audio at regular intervals.

[2263] 2. The collected video and audio data is stored in a buffer.

[2264] Step 3: Sending data

[2265] Terminal

[2266] The data stored in the buffer is sent to the server at regular intervals.

[2267] Input and Output

[2268] Input: Video and audio data stored in the buffer

[2269] Output: Video and audio data sent to the server

[2270] Specific actions

[2271] 1. When a certain amount of data accumulates in the buffer, the terminal compresses the data and transmits it.

[2272] 2. The server receives the data.

[2273] Step 4: Analyze the data

[2274] server

[2275] The received video and audio data is analyzed. The number of times the patient turns over in their sleep and changes in posture are extracted from the video data, and the frequency of snoring, talking in their sleep, and teeth grinding is detected from the audio data.

[2276] Sleep depth is evaluated on three levels: "light," "intermediate," and "deep."

[2277] Input and Output

[2278] Input: Video and audio data sent to the server

[2279] Output: Extracted body position change data, audio event data, sleep depth assessment

[2280] Specific actions

[2281] 1. The server analyzes the video data frame by frame and counts the user's changes in position and the number of times they turn over in bed.

[2282] 2. Analyze audio data to detect the frequency of snoring, sleep talking, and teeth grinding.

[2283] 3. Based on the extracted data, sleep depth is evaluated on a three-point scale and the results are stored in a database.

[2284] Step 5: Generate background music / noise

[2285] server

[2286] Based on the analysis results, a prompt sentence is input into a generative AI model (e.g., GPT-4) to generate optimal background music or noise.

[2287] The generated sound data is sent to the terminal.

[2288] Input and Output

[2289] Input: Analysis result data, prompt statement

[2290] Output: Generated background music or noise

[2291] Specific actions

[2292] 1. The server inputs the following prompt to the generative AI model:

[2293] "Generate the optimal background music for the user's sleep state based on the following data: Number of turns: 12, Frequency of snoring: High, Sleep depth assessment: Light"

[2294] 2. The generative AI model generates optimal sound data and sends it to the device.

[2295] Step 6: Play background music / noise

[2296] Terminal

[2297] The device plays background music or noise received from the server.

[2298] Input and Output

[2299] Input: BGM or noise sent from the server

[2300] Output: BGM or noise to be played

[2301] Specific actions

[2302] 1. The terminal prepares to play the sound data received from the server.

[2303] 2. Play background music or noise at specified times to optimize the user's sleep environment.

[2304] Step 7: Gather user feedback

[2305] User

[2306] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[2307] Input and Output

[2308] Input: User subjective evaluation data

[2309] Output: Subjective evaluation data sent to the server

[2310] Specific actions

[2311] 1. The user opens the app and enters information about their sleep quality (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[2312] 2. The device sends the entered evaluation data to the server.

[2313] Step 8: Next optimization

[2314] server

[2315] The received subjective evaluations are analyzed and the parameters of the generative AI model are adjusted.

[2316] Input and Output

[2317] Input: User subjective evaluation data

[2318] Output: Adjusted generative AI model parameters

[2319] Specific actions

[2320] 1. The server analyzes the received subjective evaluation data and identifies which elements were problematic.

[2321] 2. Adjust the parameters of the generative AI model for the next BGM or noise generation.

[2322] The above is the specific flow of the program processing of this system. Through each processing step, it is possible to effectively collect and analyze the user's sleep data and provide an individually optimized sleep environment.

[2323] (Application example 1)

[2324] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2325] Conventional sleep improvement systems only collect users' sleep data and rarely provide personalized, optimized responses based on that data. Furthermore, they lack real-time feedback and improvement measures based on analyzed data, which means users' sleep quality is not fully improved. Furthermore, there are insufficient means for effectively utilizing subjective feedback.

[2326] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2327] In this invention, the server includes means for collecting user sleep data, means for transmitting the collected video data and audio data, means for analyzing the transmitted data, a generative AI model that generates optimal background music or noise based on the data obtained from the analysis means, means for playing the generated background music or noise, means for the user to input a subjective evaluation of sleep quality, and means for adjusting the parameters of the generative AI model based on the subjective evaluation. This provides an optimal sleep environment tailored to each user's individual sleep pattern, and enables further optimization based on feedback.

[2328] "User's sleep data" refers to information including video data and audio data that indicate the user's sleep state.

[2329] "Video data" refers to video data that records the user's sleeping posture, the number of times they turn over in their sleep, and so on.

[2330] "Voice data" refers to audio data including the user's snoring, talking in their sleep, environmental sounds, and the like.

[2331] "Transmitting means" refers to the functions and processes for transferring collected video and audio data to a server.

[2332] "Means of analysis" refers to the process by which the server handles video and audio data and extracts and analyzes the number of times the patient turns over in bed and the frequency of snoring, etc.

[2333] "Generative AI model" refers to an artificial intelligence algorithm that generates optimal background music or noise based on analytical results and subjective evaluation.

[2334] "BGM or Noise" refers to sound data such as music or environmental sounds created by a generative AI model that are played to improve a user's sleep.

[2335] "Means for playback" refers to the functions or devices for playing background music or noise generated on a device such as a smartphone.

[2336] "Subjective evaluation" refers to the user inputting their impression of the quality of sleep as a numerical value or a comment.

[2337] "Means for adjusting the parameters of the generative AI model" refers to the process of changing the settings of the generative AI model based on the user's subjective evaluation and reflecting this in the next background music or noise generation.

[2338] This invention is a system that collects and analyzes a user's sleep data and generates and plays optimal sleep-inducing background music and noise. It functions primarily through the cooperation of the device, server, and user. The specific operation of each element is explained below.

[2339] 1. Data Collection

[2340] Terminal

[2341] Before going to bed, the user launches the app and sets it to sleep mode.

[2342] The device's camera and microphone are used to collect video and audio data while you sleep.

[2343] The collected data is sent to a server at regular intervals.

[2344] Specific examples

[2345] The user opens the app and selects sleep mode.

[2346] The device will activate its camera and microphone and begin collecting video and audio data.

[2347] The collected data is sent to a server.

[2348] 2. Data Analysis

[2349] server

[2350] The server analyzes the received video and audio data.

[2351] The number of times the patient turns over and changes in position are extracted from the video data.

[2352] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[2353] Based on the extracted data, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep."

[2354] The analysis results are stored in a database.

[2355] Specific examples

[2356] The server analyzes the video data and counts the number of times the person turns over.

[2357] The server analyzes the audio data and detects the frequency of snoring.

[2358] The server evaluates the depth of sleep and stores it in a database.

[2359] 3. BGM / Noise Generation and Playback

[2360] server

[2361] The server uses a generative AI model to generate optimal background music or noise based on the analysis results.

[2362] The generated sound data is transmitted to the terminal.

[2363] Terminal

[2364] The terminal plays the background music or noise received from the server.

[2365] Specific examples

[2366] The server inputs the analysis results into a generative AI model to generate background music or noise.

[2367] The generated sound data is transmitted to the terminal.

[2368] The terminal plays back the received sound data.

[2369] 4. User Feedback

[2370] User

[2371] After waking up, users open the app and enter their subjective assessment of their sleep quality.

[2372] Subjective evaluations include "ease of falling asleep," "whether you woke up during the night," and "ease of waking up."

[2373] Terminal

[2374] The terminal transmits the subjective evaluation input by the user to the server.

[2375] Specific examples

[2376] After waking up, the user opens the app and rates the quality of their sleep.

[2377] The user enters a rating, which the terminal sends to the server.

[2378] 5. Next optimization

[2379] server

[2380] The server analyzes the received subjective evaluations and adjusts the parameters of the generative AI model for generating the next background music or noise.

[2381] This allows the next sleep environment to be more tailored to the user.

[2382] Specific examples

[2383] The server analyzes the subjective evaluations and adjusts the parameters of the generative AI model.

[2384] Based on the analysis, optimized background music or noise will be generated the next time you use it.

[2385] Hardware and software used

[2386] Hardware: Smartphone (camera, microphone, speaker)

[2387] Software: Python, OpenCV (cv2), Request, SoundDevice (sd), server backend (Flask or Django)

[2388] Data processing and calculation

[2389] 1. Data Collection:

[2390] Video and audio data is collected using the smartphone's camera and microphone.

[2391] The data is uploaded to the server as image or audio files at regular intervals.

[2392] 2. Data Analysis:

[2393] The server analyzes the received data and detects things like turning over in bed and snoring.

[2394] The data is stored in a database for evaluating the depth of sleep.

[2395] 3. BGM / Noise Generation and Playback:

[2396] The generative AI model is run on the server to generate optimal background music and noise.

[2397] The sound data is sent to a smartphone and played at the specified time.

[2398] 4. User Feedback:

[2399] Collect user evaluation data from the smartphone interface.

[2400] The feedback data is sent to the server and used for the next BGM generation.

[2401] Prompt Sentence Examples

[2402] Below are some examples of prompts to input to the generative AI model.

[2403] We analyzed the user's sleep data. The frequency of snoring was high, and the average number of turns was three. The user's sleep depth was shallow, so please generate alpha wave music that has a relaxing effect.

[2404] The above is an embodiment of the present invention. By collecting and analyzing the sleep data of a user, it is possible to provide an optimal sleep environment for each individual.

[2405] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2406] Step 1:

[2407] Before going to bed, the user launches the app and sets it to sleep mode.

[2408] Input: User launches the app and selects sleep mode.

[2409] Output: The device is ready to start collecting data.

[2410] What happens: The user launches the app on their smartphone and selects "sleep mode," which causes the app to turn on the camera and microphone and prepare for data collection.

[2411] Step 2:

[2412] The device's camera and microphone are used to collect video and audio data while you sleep.

[2413] Input: The app activates the camera and microphone.

[2414] Output: Collected video and audio data.

[2415] How it works: The device's camera detects the user's posture and tossing and turning while sleeping, and the microphone detects snoring and talking in their sleep. This data is temporarily saved as image and audio files.

[2416] Step 3:

[2417] The collected data is sent to the server at regular intervals.

[2418] Input: Collected video and audio data.

[2419] Output: Data uploaded to the server.

[2420] How it works: Every 60 seconds, the device uploads the collected data to the server. Data transfer is performed via an internet connection and an API request.

[2421] Step 4:

[2422] The server parses the received data.

[2423] Input: Video and audio data uploaded to the server.

[2424] Output: Analysis results such as number of turns, frequency of snoring, and sleep depth.

[2425] How it works: The server analyzes the video data and counts the number of times the person turns over in bed and changes their position. Similarly, it extracts the frequency of snoring and talking in their sleep from the audio data. Based on this, it evaluates the depth of sleep and stores the results in a database.

[2426] Step 5:

[2427] Uses generative AI models to generate optimal background music or noise based on the analysis results.

[2428] Input: Analysis results (number of turns, frequency of snoring, depth of sleep).

[2429] Output: Generated background music or noise sound data.

[2430] Specific operation: The server inputs the analysis results into the generative AI model and generates a prompt. For example, it might input, "We analyzed the user's sleep data. We found that the user snores frequently and turns over an average of three times. The user's sleep depth is shallow, so please generate alpha wave music that has a relaxing effect." The generative AI model then generates music data based on the instructions.

[2431] Step 6:

[2432] The generated sound data is sent to the terminal and played back.

[2433] Input: Sound data for the generated background music or noise.

[2434] Output: Sound data played on the device.

[2435] Specific operation: The server sends the generated sound data to the device, which then plays the received sound data in real time to help the user sleep.

[2436] Step 7:

[2437] After waking up, the user opens the app and enters a subjective assessment of the quality of their sleep.

[2438] Input: User's subjective evaluation (e.g., how easily they fell asleep, whether they woke up during the night, how easily they woke up, etc.).

[2439] Output: Subjective evaluation data sent to the server.

[2440] How it works: The user opens the app and inputs their sleep quality rating, including how easily they fell asleep, whether they woke up during the night, and how easily they woke up, using numerical values ​​and comments. The device then sends this data to the server.

[2441] Step 8:

[2442] The server analyzes the subjective evaluations it receives and adjusts the parameters of the generative AI model.

[2443] Input: Subjective evaluation data sent to the server.

[2444] Output: The adjusted parameters of the generative AI model.

[2445] How it works: The server analyzes the user's subjective evaluation data and adjusts the parameters of the generative AI model, which then generates more appropriate background music or noise for the next session, improving the user's sleep quality.

[2446] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2447] ---

[2448] This invention is a system that collects and analyzes a user's sleep data and combines it with an emotion engine that recognizes the user's emotions to generate optimal sleep-inducing background music and noises. It mainly functions in cooperation with the terminal, server, and user. The specific operation of each element is explained below.

[2449] Data collection

[2450] Terminal

[2451] Before going to bed, the user launches the app and sets it to sleep mode.

[2452] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[2453] The collected data is sent to a server at regular intervals.

[2454] Specific examples

[2455] 1. The user opens the app and selects sleep mode.

[2456] 2. The device activates the camera and microphone and begins collecting video and audio data.

[2457] 3. The collected data is sent to the server.

[2458] Data analysis

[2459] server

[2460] The server analyzes the received video and audio data.

[2461] The number of times the patient turns over and changes in position are extracted from the video data.

[2462] Information such as snoring, talking in your sleep, and teeth grinding is extracted from the audio data.

[2463] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[2464] The analysis results are stored in a database.

[2465] Specific examples

[2466] 1. The server analyzes the video data and counts the number of times the person turns over.

[2467] 2. The server analyzes the audio data and detects the frequency of snoring.

[2468] 3. The server evaluates the sleep depth and stores it in a database.

[2469] emotion recognition

[2470] Terminal

[2471] The data collected by the device is sent to the emotion engine in real time.

[2472] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[2473] server

[2474] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[2475] Specific examples

[2476] 1. While the user is sleeping, the device sends facial expressions and voice to the emotion engine.

[2477] 2. The emotion engine analyzes and recognizes the user's emotional state, whether relaxed or anxious.

[2478] 3. The server receives the emotion information and integrates it into the analysis data.

[2479] BGM / noise generation and playback

[2480] server

[2481] The server uses generative AI to generate optimal background music or noise based on the analysis results and emotional information.

[2482] The generated sound data is transmitted to the terminal.

[2483] Terminal

[2484] The device plays the background music or noise received from the server.

[2485] Specific examples

[2486] 1. The server inputs the analysis results and emotional information into the generation AI, which then generates background music or noise.

[2487] 2. The generated sound data is sent to the device.

[2488] 3. The device plays the received sound data.

[2489] User Feedback

[2490] User

[2491] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[2492] Terminal

[2493] The subjective evaluation input by the user is sent to the server.

[2494] Specific examples

[2495] 1. After waking up, the user opens the app and rates their sleep quality and emotions.

[2496] 2. The user enters a rating, and the device sends it to the server.

[2497] Next optimization

[2498] server

[2499] The server analyzes the subjective evaluations received and adjusts the parameters of the generation AI for generating the next background music or noise.

[2500] This allows the next sleep environment to be more tailored to the user.

[2501] Specific examples

[2502] 1. The server analyzes the subjective evaluation and adjusts the parameters of the generated AI.

[2503] 2. Based on the analysis, generate optimized background music or noise for the next use.

[2504] ---

[2505] The above is an embodiment of the present invention. By analyzing the sleep data and emotional state of the user, it is possible to provide an optimal sleeping environment for each individual.

[2506] The processing flow will be explained below.

[2507] ---

[2508] Step 1:

[2509] Terminal

[2510] The user launches the app and sets it to sleep mode.

[2511] The camera and microphone will turn on and begin collecting video and audio data while you sleep.

[2512] Step 2:

[2513] Terminal

[2514] The collected video and audio data is sent to the server at regular intervals.

[2515] Step 3:

[2516] server

[2517] The server receives the video data and audio data transmitted from the terminal.

[2518] Step 4:

[2519] server

[2520] Using a video analysis algorithm, the number of times the patient turns over and changes in position are extracted.

[2521] Using a voice analysis algorithm, information such as snoring, talking in your sleep, and teeth grinding is extracted.

[2522] Step 5:

[2523] server

[2524] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep."

[2525] The analysis results are saved in a database.

[2526] Step 6:

[2527] Terminal

[2528] The collected data is sent to the emotion engine in real time.

[2529] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their current emotional state.

[2530] Step 7:

[2531] server

[2532] The server receives the emotion information obtained from the emotion engine and integrates it into the analysis data.

[2533] Step 8:

[2534] server

[2535] The analysis results and emotional information are input into a generation AI to generate optimal background music or noise.

[2536] Step 9:

[2537] server

[2538] The generated sound data is sent to the terminal.

[2539] Step 10:

[2540] Terminal

[2541] The device plays background music or noise received from the server.

[2542] Step 11:

[2543] User

[2544] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state.

[2545] Step 12:

[2546] Terminal

[2547] The subjective evaluation input by the user is sent to the server.

[2548] Step 13:

[2549] server

[2550] The server analyzes the received subjective evaluations.

[2551] Based on the feedback, adjust the parameters of the generated AI.

[2552] Step 14:

[2553] server

[2554] The next background music or noise is generated based on the adjusted parameters.

[2555] ---

[2556] The above is the specific processing flow of the invention combined with the emotion engine, which makes it possible to provide an optimal sleeping environment that reflects the user's emotional state.

[2557] Example 2

[2558] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2559] Conventional sleep environment improvement systems simply collect and analyze users' sleep data, and are limited in their ability to generate optimal sleep-inducing background music or noise that takes into account the user's emotional state. Furthermore, they lack the ability to optimize the next sound environment based on the user's subjective evaluation. This makes it difficult to provide a sleep environment that is suited to each individual user.

[2560] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing collected video data and audio data, a means for recognizing the user's emotional state, and a means for generating optimal background music or noise based on the data obtained from the analysis means and the emotion recognition means. This makes it possible to comprehensively analyze the user's sleep data and emotional state and provide individually optimized sleep-inducing background music or noise.

[2561] "Video data" refers to visual information that records the user's appearance and movements while sleeping.

[2562] "Audio data" refers to audio information that records sounds that occur during sleep, such as the user talking in their sleep or snoring.

[2563] "Means of collection" refers to the hardware and software used to acquire and record video and audio data.

[2564] "Means for analysis" refers to software that processes and analyzes the collected video and audio data to extract information such as the number of times the patient turns over in their sleep and the frequency of snoring.

[2565] "Means for recognizing emotional state" refers to software and hardware that analyzes the user's facial expressions and tone of voice to identify emotions such as relaxed or anxious.

[2566] "Means for generating optimal background music or noise" refers to a generative AI model that generates music or sounds that are most relaxing for the user based on analysis results and emotional state data.

[2567] "Means for playing" refers to devices and software for outputting the generated background music or noise so that the user can hear it.

[2568] "Means for inputting subjective evaluation" refers to an interface and software that allows the user to input feedback on sleep quality and emotional state after waking up.

[2569] The "means for optimizing the generation means" refers to algorithms and software for adjusting the next background music or noise to better suit the user based on the user's subjective evaluation.

[2570] The "transmitting means" refers to a communication interface and software for transmitting the video data and audio data collected by the terminal to the server in real time.

[2571] A "generative AI model" is an artificial intelligence model that generates optimal background music and noise based on the user's analytical data and emotional data.

[2572] A "prompt sentence" is an input sentence that instructs the generative AI model on what background music or noise to generate.

[2573] This invention is a system that collects a user's sleep data, analyzes that data, and combines it with an emotion engine that recognizes the user's emotional state to generate and play optimal sleep-inducing background music and noise. The system is primarily composed of a terminal, a server, and a user. The specific operation of each element is explained below.

[2574] Data collection

[2575] Terminal

[2576] Users use a smartphone with a dedicated sleep tracking app installed, launch the app before going to bed, and select sleep mode.

[2577] The device automatically activates the camera and microphone to collect video and audio data while the user sleeps, including recording the user's movements, changes in posture, snoring, and talking in their sleep.

[2578] The collected data is sent from the terminal to the server at regular intervals.

[2579] Examples:

[2580] 1. The user opens the app and selects "Sleep mode," which causes the device to activate the camera and microphone and begin collecting data.

[2581] 2. The collected video and audio data is sent to the server in real time.

[2582] Data analysis

[2583] server

[2584] The server uses a high-performance data analysis program to detect the number of times the patient turns over and changes in posture from the received video data, and extracts information such as snoring, talking in one's sleep, and teeth grinding from the audio data.

[2585] Based on the extracted data, the user's sleep depth is evaluated on a three-point scale: "light," "medium," or "deep." The evaluation results are stored in a database and used for subsequent processing.

[2586] Examples:

[2587] 1. The server processes the video data and identifies the number of times the user has turned over.

[2588] 2. Analyze audio data and record the frequency and volume of snoring.

[2589] 3. The sleep depth assessment results are stored in a database.

[2590] emotion recognition

[2591] Terminal

[2592] The device sends the collected video and audio data in real time to an emotion recognition engine, which analyzes the user's facial expressions and tone of voice to recognize their emotional state.

[2593] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server.

[2594] server

[2595] The server integrates the emotional state data received from the emotion recognition engine into the analysis data to assess the user's overall sleep state.

[2596] Examples:

[2597] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[2598] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[2599] 3. Emotional state data is sent to the server and integrated into the analysis data.

[2600] BGM / noise generation and playback

[2601] server

[2602] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. The generative AI model receives a prompt and generates the optimal sound for the user.

[2603] The generated sound data is transmitted to the terminal.

[2604] Terminal

[2605] The device plays background music or noise received from the server to help the user sleep.

[2606] Examples:

[2607] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[2608] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[2609] 3. The server sends the sound data to the device, which then plays it.

[2610] User Feedback

[2611] User

[2612] After waking up, users open the app and enter their subjective assessment of their sleep quality and emotional state, based on their sleep experience.

[2613] Terminal

[2614] The subjective evaluation input by the user is sent to the server.

[2615] Examples:

[2616] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[2617] 2. The device sends the evaluation data to the server.

[2618] Next optimization

[2619] server

[2620] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. This feedback optimizes the next BGM and noise to better suit the user.

[2621] Examples:

[2622] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[2623] 2. Provide that feedback to the generative AI model and use it for the next generation.

[2624] In this way, the present invention can comprehensively analyze the user's sleep data and emotional state, and provide sleep-inducing background music and noises that are optimized for each individual user.

[2625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2626] Step 1: Data collection

[2627] The device installs a dedicated sleep tracking app on the user, and has them launch the app and select a sleep mode before going to bed. Input: The user operates the app. Output: The device turns on the camera and microphone.

[2628] The device uses a camera and microphone to collect video and audio data in real time while the user sleeps. Input: The user's movements, changes in posture, snoring, and sleep talking. Output: Collected video and audio data.

[2629] The collected data is sent to the server at regular intervals. Input: Collected video and audio data. Output: Data is sent to the server.

[2630] Specific behavior:

[2631] 1. The user opens the app and selects "Sleep Mode."

[2632] 2. The device activates the camera and microphone and begins collecting data while the user sleeps.

[2633] 3. Upload the collected data to the server at regular intervals.

[2634] Step 2: Data analysis

[2635] The server analyzes the received video data and detects the number of turns and changes in body position. Input: Video data. Output: Data on the number of turns and changes in body position.

[2636] The server analyzes the audio data and extracts information such as snoring frequency, sleep talking, teeth grinding, etc. Input: Audio data. Output: Snoring frequency and pattern data.

[2637] Based on the extracted data, sleep depth is evaluated on a three-point scale: "light," "medium," and "deep." Input: Analyzed data. Output: Sleep depth evaluation data.

[2638] The analysis results are saved in a database. Input: Sleep depth evaluation data. Output: Data saved in the database.

[2639] Specific behavior:

[2640] 1. The server analyzes the video data and identifies the number of times and duration that the user turned over in their sleep.

[2641] 2. The server analyzes the audio data and records the frequency and patterns of snoring.

[2642] 3. The server evaluates the sleep depth and stores the results in a database.

[2643] Step 3: Emotion Recognition

[2644] The device sends the collected video and audio data to the emotion recognition engine in real time. Input: Collected data. Output: Data sent to the emotion recognition engine.

[2645] The emotion recognition engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. Input: Video data and audio data. Output: Emotional state data.

[2646] The analysis results from the emotion recognition engine are fed back to the device, which then sends them to the server. Input: Emotional state data. Output: Emotional information sent to the server.

[2647] Specific behavior:

[2648] 1. The device sends the user's facial expression and voice data to the emotion recognition engine.

[2649] 2. An emotion recognition engine analyzes and identifies the user's emotional state, whether relaxed or anxious.

[2650] 3. Emotional state data is sent to the server and integrated into the analysis data.

[2651] Step 4: Generate and play background music / noise

[2652] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional state data. Input: Analysis results and emotional state data. Output: Generated background music or noise data.

[2653] The generative AI model generates the optimal sound by inputting an appropriate prompt sentence. For example, "Please generate music that will relax the user." Input: Prompt sentence. Output: Generated sound data.

[2654] The generated sound data is sent to the terminal. Input: Generated sound data. Output: Sound data sent to the terminal.

[2655] The device plays background music or noise received from the server. Input: Sound data from the server. Output: Played sound.

[2656] Specific behavior:

[2657] 1. Based on the analysis results, the server inputs a prompt to the generation AI, such as "Please generate music that will relax the user."

[2658] 2. The generative AI model creates appropriate background music or noise and returns the sound data to the server.

[2659] 3. The server sends the sound data to the device, which then plays it.

[2660] Step 5: User feedback

[2661] After waking up, the user opens the app and inputs their subjective assessment of their sleep quality and emotional state. Input: Input of subjective assessment. Output: Inputted assessment data.

[2662] The device sends the user's subjective evaluation to the server. Input: User's subjective evaluation data. Output: Evaluation data sent to the server.

[2663] Specific behavior:

[2664] 1. The user wakes up, opens the app, and enters a rating in response to the question, "How did you feel about your sleep last night?"

[2665] 2. The device sends the evaluation data to the server.

[2666] Step 6: Next optimization

[2667] The server adjusts the parameters of the generative AI model based on the user's subjective evaluation. Input: Subjective evaluation data. Output: Adjusted parameters of the generative AI model.

[2668] This will optimize the background music and noise for the next time to better suit the user. Input: Adjusted parameters. Output: Optimized sound environment for the next use.

[2669] Specific behavior:

[2670] 1. The server analyzes the subjective evaluation and records specific feedback such as "The user found last night's background music relaxing, but the volume was too loud."

[2671] 2. Provide that feedback to the generative AI model, and adjust the parameters in the next generation to provide an optimized sound environment.

[2672] (Application example 2)

[2673] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2674] In modern society, many people find it difficult to get a good night's sleep due to stress and irregular living conditions. In this situation, there is a need for systems that support sleep. In particular, there is a need to provide sleep-inducing background music and noises that are optimized for each user's individual condition, thereby enabling users to sleep more comfortably.

[2675] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing video data and audio data, means for generating optimal background music or noise based on the data and emotional information obtained from the analysis means, and means for adjusting the prompts of the generative AI model based on the user's subjective evaluation. This makes it possible to analyze the user's sleep data and emotional state in detail and generate and provide optimal sleep-inducing background music or noise for each individual user based on the analysis results.

[2676] "Video data" refers to information in the form of moving images collected by a photographing device such as a camera.

[2677] "Audio data" refers to information in the form of audio collected by a sound collection device such as a microphone.

[2678] The "analysis means" is a method or device for extracting specific information based on the collected video and audio data.

[2679] "Emotion information" is data relating to the user's emotional state analyzed from facial expressions, tone of voice, and the like.

[2680] "Optimal background music or noise" refers to background music or noise that is generated to support the user's sleep and is most suitable for the user's state.

[2681] "Subjective evaluation" refers to evaluation data input by the user based on their own feelings, and includes information on the quality of sleep, emotional state, and the like.

[2682] A "generative AI model" is an artificial intelligence model that generates specific content based on input data.

[2683] A "prompt" is input data for a generative AI model, and includes instructions and conditions regarding the content to be generated.

[2684] "Collecting means" refers to a method or device for acquiring video data and audio data.

[2685] "Transmitting means" refers to a method or device for sending collected data to another device or server.

[2686] This invention is a system that collects and analyzes a user's sleep data and recognizes the user's emotions. It mainly functions in cooperation with a terminal, a server, and a user. The specific operation of each element is described below.

[2687] Data collection

[2688] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the camera and microphone and collects video and audio data while the device is sleeping. This collected data is then sent to a server at regular intervals.

[2689] The specific hardware used is the smartphone camera and microphone. Audio data is collected using the sounddevice library, and video data is collected using cv2 (OpenCV).

[2690] Data analysis

[2691] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. This information is used to evaluate the depth of sleep on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[2692] The analysis tools used are cv2 for video data analysis and numpy for audio data analysis, while databases such as MySQL and PostgreSQL are used.

[2693] emotion recognition

[2694] The device sends the collected data to the emotion engine in real time, and analyzes the user's facial expressions and tone of voice to recognize their emotional state. This emotional information is then sent to the server and integrated into the analyzed data.

[2695] For emotion recognition, a library called DeepFace is used, which allows the system to recognize the user's emotional state, such as whether they are relaxed or anxious.

[2696] BGM / noise generation and playback

[2697] The server uses a generative AI model to generate optimal background music or noise based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[2698] An example of a generative AI model would be GPT-3. An example prompt would be:

[2699] User Emotion: Relaxed, Sleep State: Deep

[2700] Please create the best background music for this situation.

[2701] User Feedback

[2702] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent to the server.

[2703] Next optimization

[2704] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model for generating the next background music or noise, so that the next sleep environment is more tailored to the user.

[2705] As a specific example, if a user evaluates that they are "in a relaxed state and in a deep sleep," the AI ​​model will be adjusted based on that data so that similar background music will be generated the next time.

[2706] In this way, the present invention can provide the user with an optimal sleeping environment.

[2707] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2708] Step 1:

[2709] Before going to bed, the user launches the application and sets the device to sleep mode, which activates the smartphone's camera and microphone and starts collecting video and audio data. This collected data is sent to the server every five seconds.

[2710] Specific input: The user's action to set the application to sleep mode.

[2711] Specific operation: The camera and microphone are activated and video and audio data is collected.

[2712] Specific output: Video and audio data collected in real time.

[2713] Step 2:

[2714] The server analyzes the received video and audio data and extracts information such as the number of times the person turns over in their sleep, changes in posture, snoring, talking in their sleep, teeth grinding, etc. Based on this information, the depth of sleep is evaluated on a three-point scale: "light," "medium," or "deep," and the results are stored in a database.

[2715] Specific input: video data, audio data.

[2716] Specific operations: Analyze body movements from video data using cv2, and detect snoring and sleep talking from audio data using numpy.

[2717] Specific output: Sleep depth assessment results and a database storing them.

[2718] Step 3:

[2719] The device collects data in real time and sends the user's facial expressions and tone of voice to the emotion engine to analyze the user's emotional state. The server receives the emotional information and integrates it into the analysis data.

[2720] Specific input: collected video and audio data.

[2721] Specific operation: Recognizes emotions by analyzing facial expressions and tone of voice using the DeepFace library.

[2722] Specific output: Recognized emotional information.

[2723] Step 4:

[2724] The server uses a generative AI model to generate the optimal background music or noise for the user based on the analysis results and emotional information. The generated sound data is sent to the device and played in real time.

[2725] Specific input: Sleep depth assessment results and emotional information.

[2726] Specific operation: A prompt sentence is input into the generative AI model to generate background music or noise. Example prompt sentence: "User's emotion: Relaxed, Sleep state: Deep. Please generate background music that is optimal for this state."

[2727] Specific output: Generated background music or noise data.

[2728] Step 5:

[2729] The device plays the generated background music or noise to help the user sleep.

[2730] Specific input: generated background music or noise data.

[2731] Specific operation: Plays sound using the device's audio function.

[2732] Specific output: background music or noise heard by the user.

[2733] Step 6:

[2734] After waking up, the user opens the application and inputs their subjective assessment of their sleep quality and emotional state, which is then sent from the device to the server.

[2735] Specific input: Subjective evaluation entered by the user.

[2736] Specific operation: Enter sleep quality and emotional state on the application's evaluation screen and send the data.

[2737] Specific output: Subjective evaluation data sent to the server.

[2738] Step 7:

[2739] The server analyzes the received subjective evaluations and adjusts the prompts of the generative AI model to generate background music or noise for the next use, thereby providing a more suitable environment for the next use.

[2740] Specific input: Subjective evaluation data submitted by users.

[2741] Specific operation: Analyze subjective evaluations and adjust the prompt parameters of the generative AI model.

[2742] Specific output: The background music or noise data that will be generated next time based on the adjusted prompt.

[2743] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2744] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2745] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2746] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2747] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2748] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2749] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2750] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2751] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2752] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2753] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according t...

Claims

1. a means for collecting video data; means for collecting audio data; means for analyzing the collected video and audio data; means for generating optimal background music or noise based on the data obtained from the analysis means; a means for playing the generated background music or noise; A means for a user to input a subjective evaluation; and means for optimizing the generating means based on said subjective evaluation.

2. 10. The system of claim 1, further comprising means for transmitting video and audio data.

3. The system according to claim 1 , further comprising means for adjusting parameters of the generating AI based on a subjective evaluation by a user.

Images