system

Abstract

Provide a system. A means for acquiring a user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the means to a server; a means for analyzing the acquired data in the server and identifying the user's current situation and environment; means for generating music based on the situation or environment; means for transmitting the generated music to a user terminal; a means for playing the music transmitted from the server in the terminal; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For music lovers and those who often listen to background music, choosing music that suits the situation and environment can be a hassle. This invention aims to solve this problem by automatically generating music that is perfect for the user's situation and environment, thereby providing an experience that solves the problem of making the user's daily life a little more enjoyable. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system as follows. Specifically, the system includes a means for acquiring a user's current location, schedule, weather, and camera footage, a means for transmitting the acquired data to a server, and a means for analyzing the data in the server and identifying the user's current situation and environment. The system also includes a means for generating music based on the situation and environment, a means for transmitting the generated music to the user's device, and a means for playing the music transmitted from the server on the device. The system also includes a function for acquiring location information and movement speed to identify the user's activity status, and a function for determining music parameters based on the analysis results, taking into account the weather and time of day.

[0006] "User's current location" is information indicating the user's current physical location, which is usually obtained using GPS data.

[0007] "Schedule" refers to information indicating a user's plans and schedules, and is obtained from a calendar application or a schedule function within a device.

[0008] "Weather" refers to information showing the current weather conditions in the user's location, obtained from weather information services or APIs.

[0009] "Camera footage" refers to images and video data captured by a camera installed on a user's device.

[0010] A "server" is a computer system that receives, analyzes, generates music, and transmits data.

[0011] "Data analysis" is the process of analyzing various acquired data to identify the user's current situation and environment.

[0012] "Music generation" is the process of using algorithms and AI to create music that is appropriate for a situation or environment based on analyzed data.

[0013] A "terminal" is a device used by a user that collects, transmits, and receives data and plays music.

[0014] "Location information" is geographical data that indicates the user's current location.

[0015] "Movement speed" is data indicating the speed at which a user's device moves from one point to another.

[0016] "Musical parameters" are elements that make up music, and include information such as tempo, key, instrument composition, melody, and the like.

[0017] The "analysis results" are information about the user's current situation and environment obtained as a result of data analysis. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information. This system is composed of multiple components and can provide music that matches the user's situation in real time. An embodiment of the present invention will be described in detail below.

[0040] Server-side processing

[0041] 1. Data Reception

[0042] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, weather data, etc.

[0043] 2. Data Analysis

[0044] The server analyzes the received data and identifies the user's current situation and environment. Based on location information and movement speed data, it determines where the user is currently located and what activities they are engaged in. Weather information and schedule information are also used to obtain a more detailed understanding of the situation.

[0045] 3. Music Generation

[0046] The server's music generation AI uses the analyzed data to generate music suited to the situation. For example, if you're driving along the coast, refreshing, rhythmic music will be generated. Parameters such as tempo, key, and instrumentation are adjusted according to the user's situation.

[0047] 4. Sending music

[0048] The server transmits the generated music data to the user's terminal.

[0049] Terminal side processing

[0050] 1. Data Collection

[0051] The device collects the user's location using a GPS component, measures the speed of movement using an accelerometer, captures camera footage, retrieves schedules from a schedule app, and collects weather information from a weather app.

[0052] 2. Data Transmission

[0053] The terminal transmits the collected data to the server.

[0054] 3. Music reception and playback

[0055] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0056] User operations

[0057] 1. Start the service

[0058] The service starts when the user launches a dedicated app on the device.

[0059] After the initial setup, data collection, transmission, and music playback are performed automatically, so no special operations are required by the user.

[0060] 2. Music experience

[0061] Users can enjoy music that is suited to the situation in real time. For example, refreshing rhythmic music will be automatically played while driving on the beach.

[0062] Example: Coastal driving

[0063] 1. Data Collection

[0064] The device uses GPS to determine when the user is traveling along the coast, an accelerometer to measure the speed of the vehicle, and a camera to capture the coastal scenery.

[0065] 2. Data Transmission

[0066] The device transmits the collected location information, movement speed, and camera image data to the server.

[0067] 3. Data Analysis

[0068] The server analyzes the data and determines that the user is driving along the coast. It also checks the weather and sees that it is a clear day.

[0069] 4. Music Generation

[0070] The server generates music suitable for a drive to the coast, such as refreshing rhythmic music or music that includes the sound of waves.

[0071] 5. Sending and playing music

[0072] The server transmits the generated music data to the user's terminal, where it is played in real time.

[0073] In this way, the system of the present invention can generate and play music that is optimal for the user's current situation, providing a comfortable musical experience.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[0077] Step 2:

[0078] The device captures the surroundings using the built-in camera. The captured image is sent to the image analysis module, which analyzes the scene characteristics (e.g., beach, park, city, etc.). The analysis results are also temporarily stored in storage.

[0079] Step 3:

[0080] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[0081] Step 4:

[0082] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[0083] Step 5:

[0084] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet sent includes location information, movement speed, analysis results of camera footage, schedule information, and weather information.

[0085] Step 6:

[0086] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[0087] Step 7:

[0088] The server's data analysis module analyzes the data to determine the user's current situation and environment. Based on location information and movement speed data, it determines where the user is and what they are doing. The analysis of camera footage determines the surrounding environment (e.g., beach, park, etc.), and schedule information determines the user's current schedule. Weather information is also used as a reference.

[0089] Step 8:

[0090] The server's music generation AI uses the analysis results to generate music that is optimal for the user's current situation and environment. The music generation algorithm determines musical parameters such as tempo, key, and instrument composition, and generates the song based on them.

[0091] Step 9:

[0092] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[0093] Step 10:

[0094] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[0095] Step 11:

[0096] The device's music playback module plays the received music data, allowing users to enjoy music that is best suited to their situation and environment in real time.

[0097] In this way, the system of the present invention uses a variety of data, such as location information, camera footage, schedules, and weather information, to automatically generate music that matches the user's current situation and provide it in real time.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Conventional music playback systems have been unable to provide music that is suited to a user's individual state or environment in real time. As a result, it has been difficult to provide a music experience that matches the user's activity and emotions. It has been particularly challenging to provide appropriate music to users who are on the move or attending a specific event.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes means for acquiring user location information, schedule, weather, and video data, means for transmitting the data acquired by the above means to the server, means for analyzing the acquired data in the server and identifying the user's current state and environment, means for generating music based on the state and environment, means for transmitting the generated music to the user's device, and means for playing the music transmitted from the server in the device, thereby making it possible to provide music that is optimal for the user's real-time situation and environment.

[0103] "Location information" is geographical data for identifying the user's current location.

[0104] "Schedule" is data that includes information about a user's plans and events.

[0105] "Weather" is data about weather conditions in a particular area.

[0106] "Video data" is visual information captured by a device such as a camera.

[0107] A "server" is a computer system for processing and managing data.

[0108] "Device" refers to a terminal or device used by a user, including a smartphone, tablet, etc.

[0109] "Analysis" is the process of interpreting acquired data and identifying the user's condition and environment.

[0110] "Music generation" is the process of creating music based on specific conditions or circumstances.

[0111] "Transmission" is the process of transferring data or information from one point to another.

[0112] "Playback" is the process of decoding received music data and playing the music through an audio output device.

[0113] The present invention is a system that acquires a user's current location, schedule, weather, and video data, and generates and plays optimal music based on that information. This system is composed of a server and a user's terminal, and specific embodiments are shown below.

[0114] Server-side processing

[0115] The server receives location information, weather information, schedule data, and video data sent from the user's device via an Internet connection.

[0116] The server analyzes the various data it receives. It analyzes location information and movement speed data to determine the user's current location and movement status. By combining this with weather information and schedule information, it can grasp the user's detailed situation. For example, if the user is driving along the coast, the received location information and weather information can determine that the coast is sunny and clear.

[0117] Based on this analysis data, the music generation AI installed on the server generates appropriate music. A prompt sentence is used to generate the music. The following is an example:

[0118] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0119] The server transmits the generated music data to the user's device in real time, where it is decoded and played back through a playback device (speakers or headphones).

[0120] Terminal side processing

[0121] The device acquires the user's location information using the built-in GPS module, measures the speed of movement using the accelerometer, captures surrounding images using the camera, obtains schedules from a schedule app, and collects current weather information from a weather app.

[0122] The collected data is sent in real time to a server via an internet connection, making it immediately available for analysis.

[0123] The device receives the music data sent from the server, decodes it, and plays it back, allowing users to enjoy a real-time music experience.

[0124] User operations

[0125] Users simply launch the dedicated app on their device to start the service. Once the initial setup is complete, all processes, including data collection, transmission, and music playback, are carried out automatically, with no special operations required from the user.

[0126] Example: Coastal driving

[0127] The device uses GPS to determine that the user is traveling along the coast. The acceleration sensor measures the vehicle's speed and the camera captures the coastal scenery. This data is sent to a server, which analyzes the data and determines that the user is driving along the coast. Weather information is used to confirm that the weather is clear, and music appropriate to the conditions is generated. For example, refreshing, rhythmic music or music including the sound of waves is generated. This generated music data is sent to the user's device and played in real time.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Server-side processing

[0130] Step 1: Receiving data

[0131] Input: Location, speed, camera footage, schedule, and weather data sent from the user's device.

[0132] Detailed operation: The server receives this data via HTTP requests or WebSocket communication.

[0133] Output: The received data is stored on the server.

[0134] Step 2: Data analysis

[0135] Input: Location, speed, camera footage, schedule, and weather data received in step 1.

[0136] Detailed operation: The server combines location information and movement speed to identify the user's specific activity status (e.g., driving, jogging, etc.). It also analyzes camera footage using image recognition technology to understand the user's surrounding environment. It checks the user's current activity and upcoming plans from schedule information and identifies the local weather using weather data.

[0137] Output: Analysis results that reflect the user's current state and environment.

[0138] Step 3: Music Generation

[0139] Input: Analysis results obtained in step 2.

[0140] Detailed operation: The music generation AI installed on the server creates a prompt based on the analysis results, and generates music according to the prompt according to the situation.

[0141] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0142] Output: The generated music data.

[0143] Step 4: Send your music

[0144] Input: The music data generated in step 3.

[0145] Detailed operation: The server transmits the generated music data to the user's device in real time via HTTP streaming or WebSocket connection.

[0146] Output: The transmitted music data.

[0147] Terminal side processing

[0148] Step 1: Data collection

[0149] Input: User behavior, surrounding environment information, and weather and schedule information.

[0150] Detailed operation: The device collects location information using the built-in GPS module, measures movement speed using the accelerometer, captures surrounding images using the camera, obtains schedule data from the schedule app, and collects weather information from the weather app.

[0151] Output: Collected location information, movement speed, camera footage, schedule, and weather data.

[0152] Step 2: Send data

[0153] Input: Location, speed, camera footage, schedule, and weather data collected in Step 1.

[0154] How it works in detail: The device sends this data to a server in real time via an internet connection.

[0155] Output: The data sent to the server.

[0156] Step 3: Receive and play music

[0157] Input: Music data sent from the server.

[0158] Detailed operation: The device receives music data in real time, decodes it, and plays it through speakers or headphones.

[0159] Output: Music played for the user to listen to.

[0160] User operations

[0161] Step 1: Start the service

[0162] Detailed operation: The user launches the dedicated app on the device to start the service. After the initial setup is complete, the process of data collection, transmission, and music playback is automatic.

[0163] Step 2: Music experience

[0164] Detailed operation: Users can enjoy music that suits the situation in real time. This process does not require any special operation, as the system automatically generates and plays the music.

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] In modern life, there are many systems that provide optimal music based on the user's current location, schedule, weather, etc., but there is no established method for automatically generating music suitable for relaxation situations at home and providing it in real time. In particular, when a user wants to relax in a quiet environment at home, providing music that is appropriate for that situation is a challenge.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for acquiring the user's current location, schedule, weather, and camera footage, means for transmitting the data acquired by the above means to the server, and means for generating relaxing music based on the quiet conditions in the home acquired from the smartphone or home security camera, thereby enabling music appropriate for the conditions to be automatically generated and played when the user wants to relax at home.

[0170] "User's current location" refers to the physical location where the user is currently located.

[0171] "Schedule" refers to data in which a user records future plans and events.

[0172] "Weather" refers to data indicating current weather conditions.

[0173] "Camera Footage" refers to real-time video data captured by a camera.

[0174] A "smartphone" is a type of mobile phone, a small device with advanced computing capabilities.

[0175] "Home security camera" refers to a camera device installed to monitor the safety of a home.

[0176] A "server" refers to a computer system that performs processes such as sending and receiving data and analyzing data over a network.

[0177] "Relaxing music" refers to music designed to help a user relax.

[0178] "User's terminal" refers to an electronic device such as a computer or smartphone used by a user.

[0179] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on this data. This system can provide music that suits the situation in the home in real time. An embodiment of the present invention will be described in detail below.

[0180] System Configuration

[0181] 1. Hardware:

[0182] Smartphone (iOS or ANDROID (registered trademark))

[0183] Home security cameras or doorbell cameras (e.g., regular home security cameras)

[0184] Network Connection

[0185] 2. Software:

[0186] Custom application (app for home music assistant)

[0187] Cloud server (e.g., Amazon Web Services, Google Cloud)

[0188] Music generation AI (e.g., general generative AI models)

[0189] Program processing

[0190] 1. Data collection methods:

[0191] The system obtains the user's current location, schedule, weather information, and camera footage from smartphones and home security cameras, using the smartphone's GPS, calendar app, weather app, and camera footage.

[0192] 2. Means of data transmission:

[0193] This data is sent to a cloud server, where it is securely transmitted over the network.

[0194] 3. Data analysis methods:

[0195] The cloud server analyzes the transmitted data and determines the user's current situation, such as whether they are in a quiet living room, by using location data, camera footage, weather information, and schedule information.

[0196] 4. Music Generation Means:

[0197] Based on the analysis results, a generative AI model is used to generate relaxing music. For example, if you want to spend some quiet time at home, music using piano or ambient sounds will be generated.

[0198] 5. Music transmission method:

[0199] The generated music data is sent from the cloud server to the user's device.

[0200] 6. Music playback method:

[0201] The user's device (smartphone) decodes the received music data in real time and plays it back, allowing the user to listen to the music through a dedicated app.

[0202] Specific examples

[0203] A situation where the user wants to relax at home would occur if the camera footage detected a quiet living room and the user had no scheduled events on their calendar. In this situation, the user launches a dedicated app on their smartphone, which starts data collection. The data is sent to a cloud server and analyzed. The server determines from the quiet home environment that the user wants to relax, and uses a generative AI model to generate relaxing music. This music data is then sent to the user's smartphone and played in real time.

[0204] Prompt Sentence Examples

[0205] "I'm at home, my camera is showing a quiet living room, and I have nothing planned on my calendar. Please generate some relaxing music that's appropriate for this situation."

[0206] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0207] Step 1:

[0208] A user launches a dedicated application on their smartphone. The application obtains the user's current location using GPS and retrieves their schedule from a schedule application. It also collects weather information from a weather application on the smartphone. It also captures real-time video from a home security camera. The application collects and integrates this data.

[0209] Input: Launching a smartphone app without any operations

[0210] Output: Current location data, schedule data, weather information, camera footage

[0211] Step 2:

[0212] The device sends the collected data to the cloud server, where the network module inside the device ensures efficient and secure transmission. The data is encrypted before transmission.

[0213] Input: Current location data, schedule data, weather information, camera footage

[0214] Output: Consolidated data sent to the cloud server

[0215] Step 3:

[0216] The server analyzes the received data and identifies the user's current situation and environment. It verifies the user's current location based on GPS data and the user's current schedule based on schedule data. It checks local weather conditions based on weather information and analyzes camera footage to understand the situation within the home (for example, the quietness of the living room).

[0217] Input: Integrated data (current location data, schedule data, weather information, camera footage)

[0218] Output: Analysis of the user's current situation and environment

[0219] Step 4:

[0220] Based on the analysis results, the server uses a generative AI model to generate music that is optimal for the current situation. To generate this music, the AI ​​is input with prompts that correspond to the user's situation and environment, and the resulting music data is obtained. For example, a prompt might be, "I'm currently at home, the camera is showing a quiet living room, and there are no particular events on my calendar. Please generate relaxing music that is appropriate for this situation."

[0221] Input: Analysis of the user's current situation and environment

[0222] Output: Generated music data

[0223] Step 5:

[0224] The generated music data is sent from the cloud server to the user's device, where it is compressed and securely transmitted to the device.

[0225] Input: Generated music data

[0226] Output: Music data sent to the user's device

[0227] Step 6:

[0228] The device decodes the received music data and plays it in real time through a dedicated application, allowing users to relax by listening to the provided music.

[0229] Input: Music data sent from the cloud server

[0230] Output: Music played in real time

[0231] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0232] This system combines a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information, with an emotion engine that recognizes the user's emotions. The introduction of the emotion engine makes it possible to generate music according to the user's emotional state, providing a more personalized music experience.

[0233] Server-side processing

[0234] 1. Data Reception

[0235] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, and weather data.

[0236] 2. Data Analysis

[0237] The server analyzes the received data to identify the user's current situation and environment. It also uses an emotion engine to recognize the user's emotions from the camera footage and audio data. This allows the server to identify the user's "emotional state" in addition to their "location" and "activity status."

[0238] 3. Music Generation

[0239] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. For example, if the user is driving along the coast and in a relaxed emotional state, refreshing, rhythmic music will be generated. On the other hand, if the user is feeling stressed, calm music for relaxation will be generated.

[0240] 4. Sending music

[0241] The server transmits the generated music data to the user's terminal.

[0242] Terminal side processing

[0243] 1. Data Collection

[0244] The device collects user location information using a GPS component, measures movement speed using an accelerometer, captures camera footage, and collects schedule and weather information.

[0245] An emotion engine is used to analyze the user's emotions from camera footage and audio data.

[0246] 2. Data Transmission

[0247] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[0248] 3. Music reception and playback

[0249] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0250] User operations

[0251] 1. Start the service

[0252] The service starts when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback will be carried out automatically.

[0253] 2. Music experience

[0254] Users can enjoy music that matches their mood and situation in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play.

[0255] Example: When driving along the coast

[0256] 1. Data Collection

[0257] The device uses GPS to determine that the user is traveling along the coast, an accelerometer to measure the vehicle's speed, a camera to capture the coastal scenery, and an emotion engine to analyze the user's relaxed facial expression.

[0258] 2. Data Transmission

[0259] The device sends the collected location information, movement speed, camera image data, and emotion analysis results to the server.

[0260] 3. Data Analysis

[0261] The server analyzes the transmitted data and determines that the user is driving along the coast, and at the same time, determines from emotion analysis that the user is relaxing.

[0262] 4. Music Generation

[0263] The server generates music suitable for a coastal drive and a relaxed emotional state, for example, refreshing and rhythmic music.

[0264] 5. Sending and playing music

[0265] The server transmits the generated music data to the user's terminal, where it is played in real time.

[0266] In this way, the system of the present invention provides the user with a more comfortable music experience by taking into account not only the user's current situation but also their emotional state and providing optimal music in real time.

[0267] The processing flow will be explained below.

[0268] Step 1:

[0269] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[0270] Step 2:

[0271] The device uses a built-in camera to capture images of the user's face and surroundings. The captured images are sent to an emotion engine, which analyzes emotions from the user's facial expressions and voice data. The camera images are also used to identify the environment (e.g., beach, park, city, etc.).

[0272] Step 3:

[0273] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[0274] Step 4:

[0275] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[0276] Step 5:

[0277] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet includes location information, movement speed, camera image analysis results (environment identification and emotion analysis), schedule information, and weather information.

[0278] Step 6:

[0279] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[0280] Step 7:

[0281] The server's data analysis module analyzes the data to determine the user's current situation, environment, and emotional state. Location and movement speed data are used to determine where the user is and what they are doing. Analysis of the camera footage determines the surrounding environment (e.g., beach, park, etc.), and data from the emotion engine is used to determine the user's emotional state. Schedule information and weather information are also taken into account in the analysis.

[0282] Step 8:

[0283] The server's music generation AI generates music that best suits the situation based on the analysis results and the user's recognized emotional state. The music generation algorithm determines musical parameters such as tempo, key, and instrumentation, and generates the song based on them.

[0284] Step 9:

[0285] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[0286] Step 10:

[0287] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[0288] Step 11:

[0289] The device's music playback module plays the received music data, allowing users to enjoy music that best suits their situation, environment, and emotions in real time.

[0290] In this way, the system of the present invention utilizes a variety of data, such as location information, camera footage, schedules, weather information, and emotional data, to automatically generate music that matches the user's current situation and emotional state and provide it in real time.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] If the music provided is not suited to the user's real-time situation or emotional state, the music experience will be unsatisfying for the user, which may reduce the user's experience.

[0294] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring the user's current location, schedule, weather, and video data, means for transmitting the data acquired by the above means to a data center, means for analyzing the acquired data at the data center and identifying the user's current situation and environment, and emotion analysis means for recognizing the user's emotional state from the video data and audio data. This makes it possible to provide personalized music in real time based on the user's situation and emotional state.

[0295] "Current location of user" refers to the physical location information of the user in real time.

[0296] "Schedule" refers to information about activities or events that a user has set or planned in advance.

[0297] "Weather" is information about the current weather conditions, including temperature, humidity, wind speed, and precipitation.

[0298] "Video data" is visual data that records the situation and scenery around the user.

[0299] A "data center" is a facility where servers and storage devices are installed and where data is processed and stored.

[0300] "Emotion analysis means" is a technology for recognizing the emotional state of a user using video data and audio data.

[0301] "Location Information" means a user's real-time physical coordinate data obtained using GPS or other means.

[0302] "Moving speed" is data that measures the speed at which the user is moving.

[0303] "Activity status" refers to information about the user's current activities and means of transportation.

[0304] An "information terminal" is an electronic device that is capable of data communication and running applications, such as a smartphone, tablet, or PC.

[0305] "Musical parameters" refer to elements such as tempo, rhythm, and tone that are set when generating music.

[0306] This invention is a system that acquires and analyzes a user's current location, schedule, weather, and video data to understand the user's situation and emotional state, and generates and plays personalized music based on that information.By introducing an emotion analysis means, this invention achieves more advanced personalization according to the user's emotional state.

[0307] Server-side example

[0308] The server receives data sent from the user's device. This data includes the user's current location, schedule, weather, and video data. The data is acquired in real time using cloud services for rapid processing. For example, commonly used cloud services can be used for database storage and streaming services.

[0309] The received data is analyzed on the server. This analysis includes algorithms to understand the user's current situation and environment, as well as emotion analysis to recognize the user's emotional state from video and audio data. For example, image processing and machine learning frameworks such as OpenCV and TENSORFLOW® are used to analyze video data.

[0310] Based on the analysis results, the server-side music generation AI generates music that best suits the user's situation and emotional state. This music generation uses the machine learning library Magenta and other technologies. The generated music is then sent to the user's device via a streaming protocol.

[0311] Example of terminal side

[0312] The device collects data from multiple sensors and applications. Specifically, it uses a GPS module to obtain the user's location and an accelerometer to measure their speed. It also captures camera footage and collects audio data from a microphone. This data is preprocessed by the device's applications and then sent to a data center.

[0313] The acquired data is sent to a server via an Internet connection, usually using the HTTP / HTTPS protocol. The music data sent from the server is decoded on the device and played in real time. The device has a music playback application that runs in the background, so no user interaction is required.

[0314] User operations

[0315] Users first install the dedicated app on their device and then start the service by launching the app. During the initial setup, the app is granted the necessary permissions (location information, camera, microphone, etc.). This allows data collection, transmission, and music playback to occur automatically.

[0316] Users can enjoy music that adapts to their situation and emotions in real time. For example, if you feel relaxed while driving along the coast, your device will automatically play refreshing, rhythmic music. This makes the user's music experience more personalized and improves their satisfaction.

[0317] Specific operation example

[0318] For example, if the user is driving along the coast, the following steps are performed:

[0319] 1. The device's GPS determines that the user is traveling along the coast, and the accelerometer measures that the car is moving at a constant speed.

[0320] 2. The camera captures the coastal scenery, and the emotion engine analyzes the user's relaxed facial expression.

[0321] 3. The device sends location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data to the server.

[0322] 4. The server analyzes this data and determines that the user is driving along the coast, and at the same time, determines from the emotion analysis results that the user is relaxing.

[0323] 5. The server's music generation AI generates music that best suits the user's situation and emotions, such as refreshing and rhythmic music.

[0324] 6. The generated music data is sent to the user's device using a streaming protocol and played in real time.

[0325] Prompt Sentence Examples

[0326] Examples of prompts that users can enter into a generative AI model:

[0327] "The user is driving along the coast and feeling relaxed. Generate refreshing, rhythmic music that suits this situation."

[0328] The system provides a personalized music experience by comprehensively considering the user's current situation and emotional state. The music generated in real time is designed to provide a more comfortable and satisfying experience for the user.

[0329] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0330] Server-side processing

[0331] Step 1: Receiving data

[0332] The server receives location, schedule, weather, and video data sent from the user's device. The server uses a stable Internet connection to receive the data quickly and accurately. The input data includes the user's real-time information, which is stored in the server's storage. The output data is stored in a database.

[0333] Step 2: Data analysis

[0334] The server analyzes the received data. Specifically, it uses location information to identify the user's current location and determines the user's mode of transportation (car, walking, etc.) from their movement speed. It then uses emotion analysis to recognize the user's emotional state (e.g., relaxed, stressed, etc.) from the video and audio data. This analysis uses image processing and machine learning libraries such as OpenCV and TensorFlow. The inputs are the user's location information, movement speed, video data, and audio data, and the output is the analysis results of the user's current location, mode of transportation, and emotional state.

[0335] Step 3: Music Generation

[0336] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. This music generation uses machine learning libraries such as Magenta. For example, if the user is in a relaxed emotional state while driving along the coast, refreshing, rhythmic music will be generated. The input is the analysis results, and the output is the generated music data.

[0337] Step 4: Send your music

[0338] The server sends the generated music data to the user's device. A streaming protocol (e.g., HTTP Live Streaming (HLS)) is used for transmission, allowing the user to enjoy music in real time. The input is the generated music data, and the output is the music data sent to the user's device.

[0339] Terminal side processing

[0340] Step 1: Data collection

[0341] The device collects the user's location information using a GPS module and measures their speed using an acceleration sensor. It also captures camera footage and obtains audio data from a microphone. These data are preprocessed by the device's application. The input is data from the sensors and application, and the output is processed location information, speed, camera footage, and audio data.

[0342] Step 2: Send data

[0343] The device sends the acquired data (location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data) to a server. A stable Internet connection is required for transmission, and the HTTP / HTTPS protocol is usually used. The input is the processed data, and the output is the data sent to the server.

[0344] Step 3: Receive and play music

[0345] The device receives the music data sent from the server, decodes it, and plays it. The device's music playback application runs in the background, playing music continuously even while the user is using other apps. The input is the music data sent from the server, and the output is music played in real time.

[0346] User operations

[0347] Step 1: Start the service

[0348] The service begins when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback are performed automatically. The only specific operations the user must perform are installing the app and configuring the initial setup. The input is installing and configuring the app, and the output is a system that is ready to use.

[0349] Step 2: Music experience

[0350] Users can enjoy music that matches their situation and emotions in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play from your device. The input is the user's real-time situation and emotional data, and the output is a music experience that matches that.

[0351] (Application example 2)

[0352] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0353] In modern food delivery services, delivery workers face a lot of stress and fatigue while performing their duties. This can lead to reduced delivery efficiency and a poor user experience. Furthermore, there is a lack of systems that can grasp the delivery worker's emotional state in real time and provide appropriate support. Therefore, there is a need for a system that provides appropriate music and notifications in real time according to the delivery worker's emotional state and environment.

[0354] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring the user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the above means to the server; means in the server for analyzing the acquired data and identifying the user's current situation and environment; an emotion engine that includes an emotion engine for analyzing the user's emotional state and for recognizing the user's emotional state using the emotion engine; means for generating music based on the situation, environment, and the user's emotional state; means for transmitting the generated music to the user's terminal; and means for playing the music transmitted from the server on the terminal. This reduces stress for delivery personnel by providing appropriate music and notifications in real time according to the delivery personnel's emotional state and environment, thereby enabling them to perform their work more efficiently.

[0355] "User's current location" refers to the location where a person is currently located, obtained using a location information system such as GPS.

[0356] A "schedule" is information that indicates a user's dates and plans, and is data that is managed through a calendar application or online service.

[0357] "Weather" refers to weather information for the user's location, including meteorological data such as temperature, precipitation, and wind speed.

[0358] "Camera footage" refers to video footage and photographic data captured through a digital camera.

[0359] "Means" refers to methods, equipment, or systems used to achieve a certain purpose.

[0360] A "server" refers to a computer system that processes data and provides services over a network.

[0361] "Conditions and environment" refers to the specific situation or surrounding environment in which the user is placed, and is information that includes the state at that time and external factors.

[0362] An "emotion engine" refers to software or algorithms that analyze and recognize a person's emotional state from camera footage and audio data.

[0363] "Generating music" refers to creating new music data using specific algorithms or AI.

[0364] "User's device" refers to electronic devices that are directly used by the user, such as smartphones, tablets, and personal computers.

[0365] "Analysis" refers to the process of carefully analyzing acquired data to derive specific information or conditions.

[0366] "Music playback" refers to the audible reproduction of generated music data through audio equipment such as speakers or headphones.

[0367] The following describes an embodiment of the present invention. First, the overall system configuration will be described, and then the role and processing of each component will be described in detail.

[0368] System configuration

[0369] The system mainly consists of a server, a user's device, and the user. The server analyzes data and generates music, while the user's device collects and transmits data, and receives and plays music. The detailed processing flow is as follows:

[0370] Server Processing

[0371] 1. Data Reception

[0372] The server receives various data sent from the user's device, including the user's current location using GPS, schedule, weather, camera footage, and emotional state analyzed by the emotion engine.

[0373] 2. Data Analysis

[0374] The server uses data analysis software and an emotion engine to analyze the received data, thereby determining the user's current situation, environment, and emotional state.

[0375] 3. Music Generation

[0376] The server's music generation AI generates optimal music based on the analyzed data. For example, if a delivery person is feeling stressed, it will generate relaxing music. The software used is a music generation AI model.

[0377] 4. Sending music

[0378] The server transmits the generated music data to the user's terminal.

[0379] Terminal handling

[0380] 1. Data Collection

[0381] The device uses a GPS component to collect user location information and an acceleration sensor to measure movement speed. It also acquires camera footage and collects schedule and weather information. The emotion engine analyzes the user's emotions from the camera footage and audio data.

[0382] 2. Data Transmission

[0383] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[0384] 3. Music reception and playback

[0385] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0386] User operations

[0387] 1. Start the service

[0388] The user simply launches the dedicated app on the device and performs the initial setup. Data collection, transmission, and music playback then begin automatically, allowing the user to enjoy music tailored to their situation and emotions without any special operations.

[0389] Example: Delivery person

[0390] 1. Data Collection

[0391] The device uses GPS to identify the delivery person's current location, an accelerometer to measure their speed, a camera to capture video footage, and an emotion engine to analyze the delivery person's emotional state.

[0392] 2. Data Transmission

[0393] The collected data is sent to a server, which analyzes it.

[0394] 3. Music Generation

[0395] If the delivery person is feeling stressed, the server generates relaxing music and sends it to the terminal.

[0396] 4. Playing Music

[0397] The music sent is played on the device, allowing delivery personnel to enjoy music that suits their situation and emotions in real time.

[0398] Examples of prompt statements

[0399] Design an application that generates optimal music and notifications in real time based on the user's current location, weather, delivery route, camera footage, and emotional state. The music will help delivery drivers relieve stress and deliver efficiently, while the notifications will guide them to rest stops and generate thank-you messages for the delivery recipients.

[0400] In this way, the system of the present invention can provide a more comfortable working environment for delivery personnel by taking into account the user's current situation and emotional state and providing optimal music and notifications in real time.

[0401] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0402] Step 1: Data collection

[0403] The device uses a GPS component to collect the user's location information. It also measures the user's speed using an acceleration sensor. It also captures camera footage and obtains weather and schedule information from a weather API and schedule service. It uses an emotion engine to analyze the user's emotions from the camera footage and audio data. This data is used for subsequent processing, so it undergoes any necessary data processing (e.g., coordinate normalization, speed filtering, video data analysis, etc.).

[0404] Input: location information, movement speed, camera footage, weather information, schedule information, emotional data

[0405] Output: Analyzed location information, speed data, emotion data, weather data, schedule data

[0406] Step 2: Send data

[0407] The device sends the collected and analyzed data to a server. This data transmission is carried out via a network (Internet or mobile data communication). The server receives various data.

[0408] Input: Analyzed location information, speed data, emotion data, weather data, schedule data

[0409] Output: Send data to the server

[0410] Step 3: Receiving data

[0411] The server receives the data sent from the device, temporarily stores the data, and uses it for subsequent analysis.

[0412] Input: Data sent from the terminal

[0413] Output: Data stored on the server

[0414] Step 4: Data analysis

[0415] The server analyzes the received data and identifies the user's current situation and environment. This analysis process uses data analysis software and an emotion engine. For example, it determines the user's location from location information and recognizes their current emotional state from emotion data. Based on this, it sets the parameters necessary for music generation.

[0416] Input: Data stored on the server

[0417] Output: Analysis results (user situation, environment, emotional state)

[0418] Step 5: Music Generation

[0419] The server uses a music generation AI model to generate optimal music based on the analyzed data. For example, if the user is stressed, it generates relaxing music, and if the user is relaxed, it generates lively music. The generated music data has a style and tempo that suits the user's emotions and environment.

[0420] Input: Analysis results

[0421] Output: Generated music data

[0422] Step 6: Send music

[0423] The server transmits the generated music data to the user's terminal via a network.

[0424] Input: Generated music data

[0425] Output: Sending music data to the device

[0426] Step 7: Receive and play music

[0427] The device receives the music data sent from the server, decodes it, and plays it. The music is provided to the user in real time, providing an appropriate experience based on the user's current emotions and environment.

[0428] Input: Music data sent from the server

[0429] Output: Music being played

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0433] [Second embodiment]

[0434] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0435] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0441] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0445] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0446] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information. This system is composed of multiple components and can provide music that matches the user's situation in real time. An embodiment of the present invention will be described in detail below.

[0447] Server-side processing

[0448] 1. Data Reception

[0449] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, weather data, etc.

[0450] 2. Data Analysis

[0451] The server analyzes the received data and identifies the user's current situation and environment. Based on location information and movement speed data, it determines where the user is currently located and what activities they are engaged in. Weather information and schedule information are also used to obtain a more detailed understanding of the situation.

[0452] 3. Music Generation

[0453] The server's music generation AI uses the analyzed data to generate music suited to the situation. For example, if you're driving along the coast, refreshing, rhythmic music will be generated. Parameters such as tempo, key, and instrumentation are adjusted according to the user's situation.

[0454] 4. Sending music

[0455] The server transmits the generated music data to the user's terminal.

[0456] Terminal side processing

[0457] 1. Data Collection

[0458] The device collects the user's location using a GPS component, measures the speed of movement using an accelerometer, captures camera footage, retrieves schedules from a schedule app, and collects weather information from a weather app.

[0459] 2. Data Transmission

[0460] The terminal transmits the collected data to the server.

[0461] 3. Music reception and playback

[0462] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0463] User operations

[0464] 1. Start the service

[0465] The service starts when the user launches a dedicated app on the device.

[0466] After the initial setup, data collection, transmission, and music playback are performed automatically, so no special operations are required by the user.

[0467] 2. Music experience

[0468] Users can enjoy music that is suited to the situation in real time. For example, refreshing rhythmic music will be automatically played while driving on the beach.

[0469] Example: Coastal driving

[0470] 1. Data Collection

[0471] The device uses GPS to determine when the user is traveling along the coast, an accelerometer to measure the speed of the vehicle, and a camera to capture the coastal scenery.

[0472] 2. Data Transmission

[0473] The device transmits the collected location information, movement speed, and camera image data to the server.

[0474] 3. Data Analysis

[0475] The server analyzes the data and determines that the user is driving along the coast. It also checks the weather and sees that it is a clear day.

[0476] 4. Music Generation

[0477] The server generates music suitable for a drive to the coast, such as refreshing rhythmic music or music that includes the sound of waves.

[0478] 5. Sending and playing music

[0479] The server transmits the generated music data to the user's terminal, where it is played in real time.

[0480] In this way, the system of the present invention can generate and play music that is optimal for the user's current situation, providing a comfortable musical experience.

[0481] The processing flow will be explained below.

[0482] Step 1:

[0483] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[0484] Step 2:

[0485] The device captures the surroundings using the built-in camera. The captured image is sent to the image analysis module, which analyzes the scene characteristics (e.g., beach, park, city, etc.). The analysis results are also temporarily stored in storage.

[0486] Step 3:

[0487] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[0488] Step 4:

[0489] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[0490] Step 5:

[0491] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet sent includes location information, movement speed, analysis results of camera footage, schedule information, and weather information.

[0492] Step 6:

[0493] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[0494] Step 7:

[0495] The server's data analysis module analyzes the data to determine the user's current situation and environment. Based on location information and movement speed data, it determines where the user is and what they are doing. The analysis of camera footage determines the surrounding environment (e.g., beach, park, etc.), and schedule information determines the user's current schedule. Weather information is also used as a reference.

[0496] Step 8:

[0497] The server's music generation AI uses the analysis results to generate music that is optimal for the user's current situation and environment. The music generation algorithm determines musical parameters such as tempo, key, and instrument composition, and generates the song based on them.

[0498] Step 9:

[0499] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[0500] Step 10:

[0501] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[0502] Step 11:

[0503] The device's music playback module plays the received music data, allowing users to enjoy music that is best suited to their situation and environment in real time.

[0504] In this way, the system of the present invention uses a variety of data, such as location information, camera footage, schedules, and weather information, to automatically generate music that matches the user's current situation and provide it in real time.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Conventional music playback systems have been unable to provide music that is suited to a user's individual state or environment in real time. As a result, it has been difficult to provide a music experience that matches the user's activity and emotions. It has been particularly challenging to provide appropriate music to users who are on the move or attending a specific event.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes means for acquiring user location information, schedule, weather, and video data, means for transmitting the data acquired by the above means to the server, means for analyzing the acquired data in the server and identifying the user's current state and environment, means for generating music based on the state and environment, means for transmitting the generated music to the user's device, and means for playing the music transmitted from the server in the device, thereby making it possible to provide music that is optimal for the user's real-time situation and environment.

[0510] "Location information" is geographical data for identifying the user's current location.

[0511] "Schedule" is data that includes information about a user's plans and events.

[0512] "Weather" is data about weather conditions in a particular area.

[0513] "Video data" is visual information captured by a device such as a camera.

[0514] A "server" is a computer system for processing and managing data.

[0515] "Device" refers to a terminal or device used by a user, including a smartphone, tablet, etc.

[0516] "Analysis" is the process of interpreting acquired data and identifying the user's condition and environment.

[0517] "Music generation" is the process of creating music based on specific conditions or circumstances.

[0518] "Transmission" is the process of transferring data or information from one point to another.

[0519] "Playback" is the process of decoding received music data and playing the music through an audio output device.

[0520] The present invention is a system that acquires a user's current location, schedule, weather, and video data, and generates and plays optimal music based on that information. This system is composed of a server and a user's terminal, and specific embodiments are shown below.

[0521] Server-side processing

[0522] The server receives location information, weather information, schedule data, and video data sent from the user's device via an Internet connection.

[0523] The server analyzes the various data it receives. It analyzes location information and movement speed data to determine the user's current location and movement status. By combining this with weather information and schedule information, it can grasp the user's detailed situation. For example, if the user is driving along the coast, the received location information and weather information can determine that the coast is sunny and clear.

[0524] Based on this analysis data, the music generation AI installed on the server generates appropriate music. A prompt sentence is used to generate the music. The following is an example:

[0525] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0526] The server transmits the generated music data to the user's device in real time, where it is decoded and played back through a playback device (speakers or headphones).

[0527] Terminal side processing

[0528] The device acquires the user's location information using the built-in GPS module, measures the speed of movement using the accelerometer, captures surrounding images using the camera, obtains schedules from a schedule app, and collects current weather information from a weather app.

[0529] The collected data is sent in real time to a server via an internet connection, making it immediately available for analysis.

[0530] The device receives the music data sent from the server, decodes it, and plays it back, allowing users to enjoy a real-time music experience.

[0531] User operations

[0532] Users simply launch the dedicated app on their device to start the service. Once the initial setup is complete, all processes, including data collection, transmission, and music playback, are carried out automatically, with no special operations required from the user.

[0533] Example: Coastal driving

[0534] The device uses GPS to determine that the user is traveling along the coast. The acceleration sensor measures the vehicle's speed and the camera captures the coastal scenery. This data is sent to a server, which analyzes the data and determines that the user is driving along the coast. Weather information is used to confirm that the weather is clear, and music appropriate to the conditions is generated. For example, refreshing, rhythmic music or music including the sound of waves is generated. This generated music data is sent to the user's device and played in real time.

[0535] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0536] Server-side processing

[0537] Step 1: Receiving data

[0538] Input: Location, speed, camera footage, schedule, and weather data sent from the user's device.

[0539] Detailed operation: The server receives this data via HTTP requests or WebSocket communication.

[0540] Output: The received data is stored on the server.

[0541] Step 2: Data analysis

[0542] Input: Location, speed, camera footage, schedule, and weather data received in step 1.

[0543] Detailed operation: The server combines location information and movement speed to identify the user's specific activity status (e.g., driving, jogging, etc.). It also analyzes camera footage using image recognition technology to understand the user's surrounding environment. It checks the user's current activity and upcoming plans from schedule information and identifies the local weather using weather data.

[0544] Output: Analysis results that reflect the user's current state and environment.

[0545] Step 3: Music Generation

[0546] Input: Analysis results obtained in step 2.

[0547] Detailed operation: The music generation AI installed on the server creates a prompt based on the analysis results, and generates music according to the prompt according to the situation.

[0548] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0549] Output: The generated music data.

[0550] Step 4: Send your music

[0551] Input: The music data generated in step 3.

[0552] Detailed operation: The server transmits the generated music data to the user's device in real time via HTTP streaming or WebSocket connection.

[0553] Output: The transmitted music data.

[0554] Terminal side processing

[0555] Step 1: Data collection

[0556] Input: User behavior, surrounding environment information, and weather and schedule information.

[0557] Detailed operation: The device collects location information using the built-in GPS module, measures movement speed using the accelerometer, captures surrounding images using the camera, obtains schedule data from the schedule app, and collects weather information from the weather app.

[0558] Output: Collected location information, movement speed, camera footage, schedule, and weather data.

[0559] Step 2: Send data

[0560] Input: Location, speed, camera footage, schedule, and weather data collected in Step 1.

[0561] How it works in detail: The device sends this data to a server in real time via an internet connection.

[0562] Output: The data sent to the server.

[0563] Step 3: Receive and play music

[0564] Input: Music data sent from the server.

[0565] Detailed operation: The device receives music data in real time, decodes it, and plays it through speakers or headphones.

[0566] Output: Music played for the user to listen to.

[0567] User operations

[0568] Step 1: Start the service

[0569] Detailed operation: The user launches the dedicated app on the device to start the service. After the initial setup is complete, the process of data collection, transmission, and music playback is automatic.

[0570] Step 2: Music experience

[0571] Detailed operation: Users can enjoy music that suits the situation in real time. This process does not require any special operation, as the system automatically generates and plays the music.

[0572] (Application example 1)

[0573] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] In modern life, there are many systems that provide optimal music based on the user's current location, schedule, weather, etc., but there is no established method for automatically generating music suitable for relaxation situations at home and providing it in real time. In particular, when a user wants to relax in a quiet environment at home, providing music that is appropriate for that situation is a challenge.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0576] In this invention, the server includes means for acquiring the user's current location, schedule, weather, and camera footage, means for transmitting the data acquired by the above means to the server, and means for generating relaxing music based on the quiet conditions in the home acquired from the smartphone or home security camera, thereby enabling music appropriate for the conditions to be automatically generated and played when the user wants to relax at home.

[0577] "User's current location" refers to the physical location where the user is currently located.

[0578] "Schedule" refers to data in which a user records future plans and events.

[0579] "Weather" refers to data indicating current weather conditions.

[0580] "Camera Footage" refers to real-time video data captured by a camera.

[0581] A "smartphone" is a type of mobile phone, a small device with advanced computing capabilities.

[0582] "Home security camera" refers to a camera device installed to monitor the safety of a home.

[0583] A "server" refers to a computer system that performs processes such as sending and receiving data and analyzing data over a network.

[0584] "Relaxing music" refers to music designed to help a user relax.

[0585] "User's terminal" refers to an electronic device such as a computer or smartphone used by a user.

[0586] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on this data. This system can provide music that suits the situation in the home in real time. An embodiment of the present invention will be described in detail below.

[0587] System Configuration

[0588] 1. Hardware:

[0589] Smartphone (iOS or Android)

[0590] Home security cameras or doorbell cameras (e.g., regular home security cameras)

[0591] Network Connection

[0592] 2. Software:

[0593] Custom application (app for home music assistant)

[0594] Cloud servers (e.g., Amazon Web Services, Google Cloud)

[0595] Music generation AI (e.g., general generative AI models)

[0596] Program processing

[0597] 1. Data collection methods:

[0598] The system obtains the user's current location, schedule, weather information, and camera footage from smartphones and home security cameras, using the smartphone's GPS, calendar app, weather app, and camera footage.

[0599] 2. Means of data transmission:

[0600] This data is sent to a cloud server, where it is securely transmitted over the network.

[0601] 3. Data analysis methods:

[0602] The cloud server analyzes the transmitted data and determines the user's current situation, such as whether they are in a quiet living room, by using location data, camera footage, weather information, and schedule information.

[0603] 4. Music Generation Means:

[0604] Based on the analysis results, a generative AI model is used to generate relaxing music. For example, if you want to spend some quiet time at home, music using piano or ambient sounds will be generated.

[0605] 5. Music transmission method:

[0606] The generated music data is sent from the cloud server to the user's device.

[0607] 6. Music playback method:

[0608] The user's device (smartphone) decodes the received music data in real time and plays it back, allowing the user to listen to the music through a dedicated app.

[0609] Specific examples

[0610] A situation where the user wants to relax at home would occur if the camera footage detected a quiet living room and the user had no scheduled events on their calendar. In this situation, the user launches a dedicated app on their smartphone, which starts data collection. The data is sent to a cloud server and analyzed. The server determines from the quiet home environment that the user wants to relax, and uses a generative AI model to generate relaxing music. This music data is then sent to the user's smartphone and played in real time.

[0611] Prompt Sentence Examples

[0612] "I'm at home, my camera is showing a quiet living room, and I have nothing planned on my calendar. Please generate some relaxing music that's appropriate for this situation."

[0613] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0614] Step 1:

[0615] A user launches a dedicated application on their smartphone. The application obtains the user's current location using GPS and retrieves their schedule from a schedule application. It also collects weather information from a weather application on the smartphone. It also captures real-time video from a home security camera. The application collects and integrates this data.

[0616] Input: Launching a smartphone app without any operations

[0617] Output: Current location data, schedule data, weather information, camera footage

[0618] Step 2:

[0619] The device sends the collected data to the cloud server, where the network module inside the device ensures efficient and secure transmission. The data is encrypted before transmission.

[0620] Input: Current location data, schedule data, weather information, camera footage

[0621] Output: Consolidated data sent to the cloud server

[0622] Step 3:

[0623] The server analyzes the received data and identifies the user's current situation and environment. It verifies the user's current location based on GPS data and the user's current schedule based on schedule data. It checks local weather conditions based on weather information and analyzes camera footage to understand the situation within the home (for example, the quietness of the living room).

[0624] Input: Integrated data (current location data, schedule data, weather information, camera footage)

[0625] Output: Analysis of the user's current situation and environment

[0626] Step 4:

[0627] Based on the analysis results, the server uses a generative AI model to generate music that is optimal for the current situation. To generate this music, the AI ​​is input with prompts that correspond to the user's situation and environment, and the resulting music data is obtained. For example, a prompt might be, "I'm currently at home, the camera is showing a quiet living room, and there are no particular events on my calendar. Please generate relaxing music that is appropriate for this situation."

[0628] Input: Analysis of the user's current situation and environment

[0629] Output: Generated music data

[0630] Step 5:

[0631] The generated music data is sent from the cloud server to the user's device, where it is compressed and securely transmitted to the device.

[0632] Input: Generated music data

[0633] Output: Music data sent to the user's device

[0634] Step 6:

[0635] The device decodes the received music data and plays it in real time through a dedicated application, allowing users to relax by listening to the provided music.

[0636] Input: Music data sent from the cloud server

[0637] Output: Music played in real time

[0638] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0639] This system combines a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information, with an emotion engine that recognizes the user's emotions. The introduction of the emotion engine makes it possible to generate music according to the user's emotional state, providing a more personalized music experience.

[0640] Server-side processing

[0641] 1. Data Reception

[0642] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, and weather data.

[0643] 2. Data Analysis

[0644] The server analyzes the received data to identify the user's current situation and environment. It also uses an emotion engine to recognize the user's emotions from the camera footage and audio data. This allows the server to identify the user's "emotional state" in addition to their "location" and "activity status."

[0645] 3. Music Generation

[0646] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. For example, if the user is driving along the coast and in a relaxed emotional state, refreshing, rhythmic music will be generated. On the other hand, if the user is feeling stressed, calm music for relaxation will be generated.

[0647] 4. Sending music

[0648] The server transmits the generated music data to the user's terminal.

[0649] Terminal side processing

[0650] 1. Data Collection

[0651] The device collects user location information using a GPS component, measures movement speed using an accelerometer, captures camera footage, and collects schedule and weather information.

[0652] An emotion engine is used to analyze the user's emotions from camera footage and audio data.

[0653] 2. Data Transmission

[0654] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[0655] 3. Music reception and playback

[0656] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0657] User operations

[0658] 1. Start the service

[0659] The service starts when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback will be carried out automatically.

[0660] 2. Music experience

[0661] Users can enjoy music that matches their mood and situation in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play.

[0662] Example: When driving along the coast

[0663] 1. Data Collection

[0664] The device uses GPS to determine that the user is traveling along the coast, an accelerometer to measure the vehicle's speed, a camera to capture the coastal scenery, and an emotion engine to analyze the user's relaxed facial expression.

[0665] 2. Data Transmission

[0666] The device sends the collected location information, movement speed, camera image data, and emotion analysis results to the server.

[0667] 3. Data Analysis

[0668] The server analyzes the transmitted data and determines that the user is driving along the coast, and at the same time, determines from emotion analysis that the user is relaxing.

[0669] 4. Music Generation

[0670] The server generates music suitable for a coastal drive and a relaxed emotional state, for example, refreshing and rhythmic music.

[0671] 5. Sending and playing music

[0672] The server transmits the generated music data to the user's terminal, where it is played in real time.

[0673] In this way, the system of the present invention provides the user with a more comfortable music experience by taking into account not only the user's current situation but also their emotional state and providing optimal music in real time.

[0674] The processing flow will be explained below.

[0675] Step 1:

[0676] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[0677] Step 2:

[0678] The device uses a built-in camera to capture images of the user's face and surroundings. The captured images are sent to an emotion engine, which analyzes emotions from the user's facial expressions and voice data. The camera images are also used to identify the environment (e.g., beach, park, city, etc.).

[0679] Step 3:

[0680] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[0681] Step 4:

[0682] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[0683] Step 5:

[0684] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet includes location information, movement speed, camera image analysis results (environment identification and emotion analysis), schedule information, and weather information.

[0685] Step 6:

[0686] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[0687] Step 7:

[0688] The server's data analysis module analyzes the data to determine the user's current situation, environment, and emotional state. Location and movement speed data are used to determine where the user is and what they are doing. Analysis of the camera footage determines the surrounding environment (e.g., beach, park, etc.), and data from the emotion engine is used to determine the user's emotional state. Schedule information and weather information are also taken into account in the analysis.

[0689] Step 8:

[0690] The server's music generation AI generates music that best suits the situation based on the analysis results and the user's recognized emotional state. The music generation algorithm determines musical parameters such as tempo, key, and instrumentation, and generates the song based on them.

[0691] Step 9:

[0692] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[0693] Step 10:

[0694] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[0695] Step 11:

[0696] The device's music playback module plays the received music data, allowing users to enjoy music that best suits their situation, environment, and emotions in real time.

[0697] In this way, the system of the present invention utilizes a variety of data, such as location information, camera footage, schedules, weather information, and emotional data, to automatically generate music that matches the user's current situation and emotional state and provide it in real time.

[0698] Example 2

[0699] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0700] If the music provided is not suited to the user's real-time situation or emotional state, the music experience will be unsatisfying for the user, which may reduce the user's experience.

[0701] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring the user's current location, schedule, weather, and video data, means for transmitting the data acquired by the above means to a data center, means for analyzing the acquired data at the data center and identifying the user's current situation and environment, and emotion analysis means for recognizing the user's emotional state from the video data and audio data. This makes it possible to provide personalized music in real time based on the user's situation and emotional state.

[0702] "Current location of user" refers to the physical location information of the user in real time.

[0703] "Schedule" refers to information about activities or events that a user has set or planned in advance.

[0704] "Weather" is information about the current weather conditions, including temperature, humidity, wind speed, and precipitation.

[0705] "Video data" is visual data that records the situation and scenery around the user.

[0706] A "data center" is a facility where servers and storage devices are installed and where data is processed and stored.

[0707] "Emotion analysis means" is a technology for recognizing the emotional state of a user using video data and audio data.

[0708] "Location Information" means a user's real-time physical coordinate data obtained using GPS or other means.

[0709] "Moving speed" is data that measures the speed at which the user is moving.

[0710] "Activity status" refers to information about the user's current activities and means of transportation.

[0711] An "information terminal" is an electronic device that is capable of data communication and running applications, such as a smartphone, tablet, or PC.

[0712] "Musical parameters" refer to elements such as tempo, rhythm, and tone that are set when generating music.

[0713] This invention is a system that acquires and analyzes a user's current location, schedule, weather, and video data to understand the user's situation and emotional state, and generates and plays personalized music based on that information.By introducing an emotion analysis means, this invention achieves more advanced personalization according to the user's emotional state.

[0714] Server-side example

[0715] The server receives data sent from the user's device. This data includes the user's current location, schedule, weather, and video data. The data is acquired in real time using cloud services for rapid processing. For example, commonly used cloud services can be used for database storage and streaming services.

[0716] The received data is analyzed on the server. This analysis includes algorithms to understand the user's current situation and environment, as well as emotion analysis to recognize the user's emotional state from video and audio data. For example, image processing and machine learning frameworks such as OpenCV and TensorFlow are used to analyze video data.

[0717] Based on the analysis results, the server-side music generation AI generates music that best suits the user's situation and emotional state. This music generation uses the machine learning library Magenta and other technologies. The generated music is then sent to the user's device via a streaming protocol.

[0718] Example of terminal side

[0719] The device collects data from multiple sensors and applications. Specifically, it uses a GPS module to obtain the user's location and an accelerometer to measure their speed. It also captures camera footage and collects audio data from a microphone. This data is preprocessed by the device's applications and then sent to a data center.

[0720] The acquired data is sent to a server via an Internet connection, usually using the HTTP / HTTPS protocol. The music data sent from the server is decoded on the device and played in real time. The device has a music playback application that runs in the background, so no user interaction is required.

[0721] User operations

[0722] Users first install the dedicated app on their device and then start the service by launching the app. During the initial setup, the app is granted the necessary permissions (location information, camera, microphone, etc.). This allows data collection, transmission, and music playback to occur automatically.

[0723] Users can enjoy music that adapts to their situation and emotions in real time. For example, if you feel relaxed while driving along the coast, your device will automatically play refreshing, rhythmic music. This makes the user's music experience more personalized and improves their satisfaction.

[0724] Specific operation example

[0725] For example, if the user is driving along the coast, the following steps are performed:

[0726] 1. The device's GPS determines that the user is traveling along the coast, and the accelerometer measures that the car is moving at a constant speed.

[0727] 2. The camera captures the coastal scenery, and the emotion engine analyzes the user's relaxed facial expression.

[0728] 3. The device sends location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data to the server.

[0729] 4. The server analyzes this data and determines that the user is driving along the coast, and at the same time, determines from the emotion analysis results that the user is relaxing.

[0730] 5. The server's music generation AI generates music that best suits the user's situation and emotions, such as refreshing and rhythmic music.

[0731] 6. The generated music data is sent to the user's device using a streaming protocol and played in real time.

[0732] Prompt Sentence Examples

[0733] Examples of prompts that users can enter into a generative AI model:

[0734] "The user is driving along the coast and feeling relaxed. Generate refreshing, rhythmic music that suits this situation."

[0735] The system provides a personalized music experience by comprehensively considering the user's current situation and emotional state. The music generated in real time is designed to provide a more comfortable and satisfying experience for the user.

[0736] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0737] Server-side processing

[0738] Step 1: Receiving data

[0739] The server receives location, schedule, weather, and video data sent from the user's device. The server uses a stable Internet connection to receive the data quickly and accurately. The input data includes the user's real-time information, which is stored in the server's storage. The output data is stored in a database.

[0740] Step 2: Data analysis

[0741] The server analyzes the received data. Specifically, it uses location information to identify the user's current location and determines the user's mode of transportation (car, walking, etc.) from their movement speed. It then uses emotion analysis to recognize the user's emotional state (e.g., relaxed, stressed, etc.) from the video and audio data. This analysis uses image processing and machine learning libraries such as OpenCV and TensorFlow. The inputs are the user's location information, movement speed, video data, and audio data, and the output is the analysis results of the user's current location, mode of transportation, and emotional state.

[0742] Step 3: Music Generation

[0743] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. This music generation uses machine learning libraries such as Magenta. For example, if the user is in a relaxed emotional state while driving along the coast, refreshing, rhythmic music will be generated. The input is the analysis results, and the output is the generated music data.

[0744] Step 4: Send your music

[0745] The server sends the generated music data to the user's device. A streaming protocol (e.g., HTTP Live Streaming (HLS)) is used for transmission, allowing the user to enjoy music in real time. The input is the generated music data, and the output is the music data sent to the user's device.

[0746] Terminal side processing

[0747] Step 1: Data collection

[0748] The device collects the user's location information using a GPS module and measures their speed using an acceleration sensor. It also captures camera footage and obtains audio data from a microphone. These data are preprocessed by the device's application. The input is data from the sensors and application, and the output is processed location information, speed, camera footage, and audio data.

[0749] Step 2: Send data

[0750] The device sends the acquired data (location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data) to a server. A stable Internet connection is required for transmission, and the HTTP / HTTPS protocol is usually used. The input is the processed data, and the output is the data sent to the server.

[0751] Step 3: Receive and play music

[0752] The device receives the music data sent from the server, decodes it, and plays it. The device's music playback application runs in the background, playing music continuously even while the user is using other apps. The input is the music data sent from the server, and the output is music played in real time.

[0753] User operations

[0754] Step 1: Start the service

[0755] The service begins when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback are performed automatically. The only specific operations the user must perform are installing the app and configuring the initial setup. The input is installing and configuring the app, and the output is a system that is ready to use.

[0756] Step 2: Music experience

[0757] Users can enjoy music that matches their situation and emotions in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play from your device. The input is the user's real-time situation and emotional data, and the output is a music experience that matches that.

[0758] (Application example 2)

[0759] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0760] In modern food delivery services, delivery workers face a lot of stress and fatigue while performing their duties. This can lead to reduced delivery efficiency and a poor user experience. Furthermore, there is a lack of systems that can grasp the delivery worker's emotional state in real time and provide appropriate support. Therefore, there is a need for a system that provides appropriate music and notifications in real time according to the delivery worker's emotional state and environment.

[0761] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring the user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the above means to the server; means in the server for analyzing the acquired data and identifying the user's current situation and environment; an emotion engine that includes an emotion engine for analyzing the user's emotional state and for recognizing the user's emotional state using the emotion engine; means for generating music based on the situation, environment, and the user's emotional state; means for transmitting the generated music to the user's terminal; and means for playing the music transmitted from the server on the terminal. This reduces stress for delivery personnel by providing appropriate music and notifications in real time according to the delivery personnel's emotional state and environment, thereby enabling them to perform their work more efficiently.

[0762] "User's current location" refers to the location where a person is currently located, obtained using a location information system such as GPS.

[0763] A "schedule" is information that indicates a user's dates and plans, and is data that is managed through a calendar application or online service.

[0764] "Weather" refers to weather information for the user's location, including meteorological data such as temperature, precipitation, and wind speed.

[0765] "Camera footage" refers to video footage and photographic data captured through a digital camera.

[0766] "Means" refers to methods, equipment, or systems used to achieve a certain purpose.

[0767] A "server" refers to a computer system that processes data and provides services over a network.

[0768] "Conditions and environment" refers to the specific situation or surrounding environment in which the user is placed, and is information that includes the state at that time and external factors.

[0769] An "emotion engine" refers to software or algorithms that analyze and recognize a person's emotional state from camera footage and audio data.

[0770] "Generating music" refers to creating new music data using specific algorithms or AI.

[0771] "User's device" refers to electronic devices that are directly used by the user, such as smartphones, tablets, and personal computers.

[0772] "Analysis" refers to the process of carefully analyzing acquired data to derive specific information or conditions.

[0773] "Music playback" refers to the audible reproduction of generated music data through audio equipment such as speakers or headphones.

[0774] The following describes an embodiment of the present invention. First, the overall system configuration will be described, and then the role and processing of each component will be described in detail.

[0775] System configuration

[0776] The system mainly consists of a server, a user's device, and the user. The server analyzes data and generates music, while the user's device collects and transmits data, and receives and plays music. The detailed processing flow is as follows:

[0777] Server Processing

[0778] 1. Data Reception

[0779] The server receives various data sent from the user's device, including the user's current location using GPS, schedule, weather, camera footage, and emotional state analyzed by the emotion engine.

[0780] 2. Data Analysis

[0781] The server uses data analysis software and an emotion engine to analyze the received data, thereby determining the user's current situation, environment, and emotional state.

[0782] 3. Music Generation

[0783] The server's music generation AI generates optimal music based on the analyzed data. For example, if a delivery person is feeling stressed, it will generate relaxing music. The software used is a music generation AI model.

[0784] 4. Sending music

[0785] The server transmits the generated music data to the user's terminal.

[0786] Terminal handling

[0787] 1. Data Collection

[0788] The device uses a GPS component to collect user location information and an acceleration sensor to measure movement speed. It also acquires camera footage and collects schedule and weather information. The emotion engine analyzes the user's emotions from the camera footage and audio data.

[0789] 2. Data Transmission

[0790] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[0791] 3. Music reception and playback

[0792] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0793] User operations

[0794] 1. Start the service

[0795] The user simply launches the dedicated app on the device and performs the initial setup. Data collection, transmission, and music playback then begin automatically, allowing the user to enjoy music tailored to their situation and emotions without any special operations.

[0796] Example: Delivery person

[0797] 1. Data Collection

[0798] The device uses GPS to identify the delivery person's current location, an accelerometer to measure their speed, a camera to capture video footage, and an emotion engine to analyze the delivery person's emotional state.

[0799] 2. Data Transmission

[0800] The collected data is sent to a server, which analyzes it.

[0801] 3. Music Generation

[0802] If the delivery person is feeling stressed, the server generates relaxing music and sends it to the terminal.

[0803] 4. Playing Music

[0804] The music sent is played on the device, allowing delivery personnel to enjoy music that suits their situation and emotions in real time.

[0805] Examples of prompt statements

[0806] Design an application that generates optimal music and notifications in real time based on the user's current location, weather, delivery route, camera footage, and emotional state. The music will help delivery drivers relieve stress and deliver efficiently, while the notifications will guide them to rest stops and generate thank-you messages for the delivery recipients.

[0807] In this way, the system of the present invention can provide a more comfortable working environment for delivery personnel by taking into account the user's current situation and emotional state and providing optimal music and notifications in real time.

[0808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0809] Step 1: Data collection

[0810] The device uses a GPS component to collect the user's location information. It also measures the user's speed using an acceleration sensor. It also captures camera footage and obtains weather and schedule information from a weather API and schedule service. It uses an emotion engine to analyze the user's emotions from the camera footage and audio data. This data is used for subsequent processing, so it undergoes any necessary data processing (e.g., coordinate normalization, speed filtering, video data analysis, etc.).

[0811] Input: location information, movement speed, camera footage, weather information, schedule information, emotional data

[0812] Output: Analyzed location information, speed data, emotion data, weather data, schedule data

[0813] Step 2: Send data

[0814] The device sends the collected and analyzed data to a server. This data transmission is carried out via a network (Internet or mobile data communication). The server receives various data.

[0815] Input: Analyzed location information, speed data, emotion data, weather data, schedule data

[0816] Output: Send data to the server

[0817] Step 3: Receiving data

[0818] The server receives the data sent from the device, temporarily stores the data, and uses it for subsequent analysis.

[0819] Input: Data sent from the terminal

[0820] Output: Data stored on the server

[0821] Step 4: Data analysis

[0822] The server analyzes the received data and identifies the user's current situation and environment. This analysis process uses data analysis software and an emotion engine. For example, it determines the user's location from location information and recognizes their current emotional state from emotion data. Based on this, it sets the parameters necessary for music generation.

[0823] Input: Data stored on the server

[0824] Output: Analysis results (user situation, environment, emotional state)

[0825] Step 5: Music Generation

[0826] The server uses a music generation AI model to generate optimal music based on the analyzed data. For example, if the user is stressed, it generates relaxing music, and if the user is relaxed, it generates lively music. The generated music data has a style and tempo that suits the user's emotions and environment.

[0827] Input: Analysis results

[0828] Output: Generated music data

[0829] Step 6: Send music

[0830] The server transmits the generated music data to the user's terminal via a network.

[0831] Input: Generated music data

[0832] Output: Sending music data to the device

[0833] Step 7: Receive and play music

[0834] The device receives the music data sent from the server, decodes it, and plays it. The music is provided to the user in real time, providing an appropriate experience based on the user's current emotions and environment.

[0835] Input: Music data sent from the server

[0836] Output: Music being played

[0837] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0838] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0840] [Third embodiment]

[0841] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0842] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0843] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0844] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0845] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0846] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0847] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0848] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0849] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0850] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0851] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0852] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0853] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information. This system is composed of multiple components and can provide music that matches the user's situation in real time. An embodiment of the present invention will be described in detail below.

[0854] Server-side processing

[0855] 1. Data Reception

[0856] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, weather data, etc.

[0857] 2. Data Analysis

[0858] The server analyzes the received data and identifies the user's current situation and environment. Based on location information and movement speed data, it determines where the user is currently located and what activities they are engaged in. Weather information and schedule information are also used to obtain a more detailed understanding of the situation.

[0859] 3. Music Generation

[0860] The server's music generation AI uses the analyzed data to generate music suited to the situation. For example, if you're driving along the coast, refreshing, rhythmic music will be generated. Parameters such as tempo, key, and instrumentation are adjusted according to the user's situation.

[0861] 4. Sending music

[0862] The server transmits the generated music data to the user's terminal.

[0863] Terminal side processing

[0864] 1. Data Collection

[0865] The device collects the user's location using a GPS component, measures the speed of movement using an accelerometer, captures camera footage, retrieves schedules from a schedule app, and collects weather information from a weather app.

[0866] 2. Data Transmission

[0867] The terminal transmits the collected data to the server.

[0868] 3. Music reception and playback

[0869] The terminal receives the music data sent from the server, decodes it, and plays it back.

[0870] User operations

[0871] 1. Start the service

[0872] The service starts when the user launches a dedicated app on the device.

[0873] After the initial setup, data collection, transmission, and music playback are performed automatically, so no special operations are required by the user.

[0874] 2. Music experience

[0875] Users can enjoy music that is suited to the situation in real time. For example, refreshing rhythmic music will be automatically played while driving on the beach.

[0876] Example: Coastal driving

[0877] 1. Data Collection

[0878] The device uses GPS to determine when the user is traveling along the coast, an accelerometer to measure the speed of the vehicle, and a camera to capture the coastal scenery.

[0879] 2. Data Transmission

[0880] The device transmits the collected location information, movement speed, and camera image data to the server.

[0881] 3. Data Analysis

[0882] The server analyzes the data and determines that the user is driving along the coast. It also checks the weather and sees that it is a clear day.

[0883] 4. Music Generation

[0884] The server generates music suitable for a drive to the coast, such as refreshing rhythmic music or music that includes the sound of waves.

[0885] 5. Sending and playing music

[0886] The server transmits the generated music data to the user's terminal, where it is played in real time.

[0887] In this way, the system of the present invention can generate and play music that is optimal for the user's current situation, providing a comfortable musical experience.

[0888] The processing flow will be explained below.

[0889] Step 1:

[0890] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[0891] Step 2:

[0892] The device captures the surroundings using the built-in camera. The captured image is sent to the image analysis module, which analyzes the scene characteristics (e.g., beach, park, city, etc.). The analysis results are also temporarily stored in storage.

[0893] Step 3:

[0894] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[0895] Step 4:

[0896] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[0897] Step 5:

[0898] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet sent includes location information, movement speed, analysis results of camera footage, schedule information, and weather information.

[0899] Step 6:

[0900] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[0901] Step 7:

[0902] The server's data analysis module analyzes the data to determine the user's current situation and environment. Based on location information and movement speed data, it determines where the user is and what they are doing. The analysis of camera footage determines the surrounding environment (e.g., beach, park, etc.), and schedule information determines the user's current schedule. Weather information is also used as a reference.

[0903] Step 8:

[0904] The server's music generation AI uses the analysis results to generate music that is optimal for the user's current situation and environment. The music generation algorithm determines musical parameters such as tempo, key, and instrument composition, and generates the song based on them.

[0905] Step 9:

[0906] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[0907] Step 10:

[0908] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[0909] Step 11:

[0910] The device's music playback module plays the received music data, allowing users to enjoy music that is best suited to their situation and environment in real time.

[0911] In this way, the system of the present invention uses a variety of data, such as location information, camera footage, schedules, and weather information, to automatically generate music that matches the user's current situation and provide it in real time.

[0912] Example 1

[0913] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0914] Conventional music playback systems have been unable to provide music that is suited to a user's individual state or environment in real time. As a result, it has been difficult to provide a music experience that matches the user's activity and emotions. It has been particularly challenging to provide appropriate music to users who are on the move or attending a specific event.

[0915] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0916] In this invention, the server includes means for acquiring user location information, schedule, weather, and video data, means for transmitting the data acquired by the above means to the server, means for analyzing the acquired data in the server and identifying the user's current state and environment, means for generating music based on the state and environment, means for transmitting the generated music to the user's device, and means for playing the music transmitted from the server in the device, thereby making it possible to provide music that is optimal for the user's real-time situation and environment.

[0917] "Location information" is geographical data for identifying the user's current location.

[0918] "Schedule" is data that includes information about a user's plans and events.

[0919] "Weather" is data about weather conditions in a particular area.

[0920] "Video data" is visual information captured by a device such as a camera.

[0921] A "server" is a computer system for processing and managing data.

[0922] "Device" refers to a terminal or device used by a user, including a smartphone, tablet, etc.

[0923] "Analysis" is the process of interpreting acquired data and identifying the user's condition and environment.

[0924] "Music generation" is the process of creating music based on specific conditions or circumstances.

[0925] "Transmission" is the process of transferring data or information from one point to another.

[0926] "Playback" is the process of decoding received music data and playing the music through an audio output device.

[0927] The present invention is a system that acquires a user's current location, schedule, weather, and video data, and generates and plays optimal music based on that information. This system is composed of a server and a user's terminal, and specific embodiments are shown below.

[0928] Server-side processing

[0929] The server receives location information, weather information, schedule data, and video data sent from the user's device via an Internet connection.

[0930] The server analyzes the various data it receives. It analyzes location information and movement speed data to determine the user's current location and movement status. By combining this with weather information and schedule information, it can grasp the user's detailed situation. For example, if the user is driving along the coast, the received location information and weather information can determine that the coast is sunny and clear.

[0931] Based on this analysis data, the music generation AI installed on the server generates appropriate music. A prompt sentence is used to generate the music. The following is an example:

[0932] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0933] The server transmits the generated music data to the user's device in real time, where it is decoded and played back through a playback device (speakers or headphones).

[0934] Terminal side processing

[0935] The device acquires the user's location information using the built-in GPS module, measures the speed of movement using the accelerometer, captures surrounding images using the camera, obtains schedules from a schedule app, and collects current weather information from a weather app.

[0936] The collected data is sent in real time to a server via an internet connection, making it immediately available for analysis.

[0937] The device receives the music data sent from the server, decodes it, and plays it back, allowing users to enjoy a real-time music experience.

[0938] User operations

[0939] Users simply launch the dedicated app on their device to start the service. Once the initial setup is complete, all processes, including data collection, transmission, and music playback, are carried out automatically, with no special operations required from the user.

[0940] Example: Coastal driving

[0941] The device uses GPS to determine that the user is traveling along the coast. The acceleration sensor measures the vehicle's speed and the camera captures the coastal scenery. This data is sent to a server, which analyzes the data and determines that the user is driving along the coast. Weather information is used to confirm that the weather is clear, and music appropriate to the conditions is generated. For example, refreshing, rhythmic music or music including the sound of waves is generated. This generated music data is sent to the user's device and played in real time.

[0942] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0943] Server-side processing

[0944] Step 1: Receiving data

[0945] Input: Location, speed, camera footage, schedule, and weather data sent from the user's device.

[0946] Detailed operation: The server receives this data via HTTP requests or WebSocket communication.

[0947] Output: The received data is stored on the server.

[0948] Step 2: Data analysis

[0949] Input: Location, speed, camera footage, schedule, and weather data received in step 1.

[0950] Detailed operation: The server combines location information and movement speed to identify the user's specific activity status (e.g., driving, jogging, etc.). It also analyzes camera footage using image recognition technology to understand the user's surrounding environment. It checks the user's current activity and upcoming plans from schedule information and identifies the local weather using weather data.

[0951] Output: Analysis results that reflect the user's current state and environment.

[0952] Step 3: Music Generation

[0953] Input: Analysis results obtained in step 2.

[0954] Detailed operation: The music generation AI installed on the server creates a prompt based on the analysis results, and generates music according to the prompt according to the situation.

[0955] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[0956] Output: The generated music data.

[0957] Step 4: Send your music

[0958] Input: The music data generated in step 3.

[0959] Detailed operation: The server transmits the generated music data to the user's device in real time via HTTP streaming or WebSocket connection.

[0960] Output: The transmitted music data.

[0961] Terminal side processing

[0962] Step 1: Data collection

[0963] Input: User behavior, surrounding environment information, and weather and schedule information.

[0964] Detailed operation: The device collects location information using the built-in GPS module, measures movement speed using the accelerometer, captures surrounding images using the camera, obtains schedule data from the schedule app, and collects weather information from the weather app.

[0965] Output: Collected location information, movement speed, camera footage, schedule, and weather data.

[0966] Step 2: Send data

[0967] Input: Location, speed, camera footage, schedule, and weather data collected in Step 1.

[0968] How it works in detail: The device sends this data to a server in real time via an internet connection.

[0969] Output: The data sent to the server.

[0970] Step 3: Receive and play music

[0971] Input: Music data sent from the server.

[0972] Detailed operation: The device receives music data in real time, decodes it, and plays it through speakers or headphones.

[0973] Output: Music played for the user to listen to.

[0974] User operations

[0975] Step 1: Start the service

[0976] Detailed operation: The user launches the dedicated app on the device to start the service. After the initial setup is complete, the process of data collection, transmission, and music playback is automatic.

[0977] Step 2: Music experience

[0978] Detailed operation: Users can enjoy music that suits the situation in real time. This process does not require any special operation, as the system automatically generates and plays the music.

[0979] (Application example 1)

[0980] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] In modern life, there are many systems that provide optimal music based on the user's current location, schedule, weather, etc., but there is no established method for automatically generating music suitable for relaxation situations at home and providing it in real time. In particular, when a user wants to relax in a quiet environment at home, providing music that is appropriate for that situation is a challenge.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0983] In this invention, the server includes means for acquiring the user's current location, schedule, weather, and camera footage, means for transmitting the data acquired by the above means to the server, and means for generating relaxing music based on the quiet conditions in the home acquired from the smartphone or home security camera, thereby enabling music appropriate for the conditions to be automatically generated and played when the user wants to relax at home.

[0984] "User's current location" refers to the physical location where the user is currently located.

[0985] "Schedule" refers to data in which a user records future plans and events.

[0986] "Weather" refers to data indicating current weather conditions.

[0987] "Camera Footage" refers to real-time video data captured by a camera.

[0988] A "smartphone" is a type of mobile phone, a small device with advanced computing capabilities.

[0989] "Home security camera" refers to a camera device installed to monitor the safety of a home.

[0990] A "server" refers to a computer system that performs processes such as sending and receiving data and analyzing data over a network.

[0991] "Relaxing music" refers to music designed to help a user relax.

[0992] "User's terminal" refers to an electronic device such as a computer or smartphone used by a user.

[0993] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on this data. This system can provide music that suits the situation in the home in real time. An embodiment of the present invention will be described in detail below.

[0994] System Configuration

[0995] 1. Hardware:

[0996] Smartphone (iOS or Android)

[0997] Home security cameras or doorbell cameras (e.g., regular home security cameras)

[0998] Network Connection

[0999] 2. Software:

[1000] Custom application (app for home music assistant)

[1001] Cloud servers (e.g., Amazon Web Services, Google Cloud)

[1002] Music generation AI (e.g., general generative AI models)

[1003] Program processing

[1004] 1. Data collection methods:

[1005] The system obtains the user's current location, schedule, weather information, and camera footage from smartphones and home security cameras, using the smartphone's GPS, calendar app, weather app, and camera footage.

[1006] 2. Means of data transmission:

[1007] This data is sent to a cloud server, where it is securely transmitted over the network.

[1008] 3. Data analysis methods:

[1009] The cloud server analyzes the transmitted data and determines the user's current situation, such as whether they are in a quiet living room, by using location data, camera footage, weather information, and schedule information.

[1010] 4. Music Generation Means:

[1011] Based on the analysis results, a generative AI model is used to generate relaxing music. For example, if you want to spend some quiet time at home, music using piano or ambient sounds will be generated.

[1012] 5. Music transmission method:

[1013] The generated music data is sent from the cloud server to the user's device.

[1014] 6. Music playback method:

[1015] The user's device (smartphone) decodes the received music data in real time and plays it back, allowing the user to listen to the music through a dedicated app.

[1016] Specific examples

[1017] A situation where the user wants to relax at home would occur if the camera footage detected a quiet living room and the user had no scheduled events on their calendar. In this situation, the user launches a dedicated app on their smartphone, which starts data collection. The data is sent to a cloud server and analyzed. The server determines from the quiet home environment that the user wants to relax, and uses a generative AI model to generate relaxing music. This music data is then sent to the user's smartphone and played in real time.

[1018] Prompt Sentence Examples

[1019] "I'm at home, my camera is showing a quiet living room, and I have nothing planned on my calendar. Please generate some relaxing music that's appropriate for this situation."

[1020] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1021] Step 1:

[1022] A user launches a dedicated application on their smartphone. The application obtains the user's current location using GPS and retrieves their schedule from a schedule application. It also collects weather information from a weather application on the smartphone. It also captures real-time video from a home security camera. The application collects and integrates this data.

[1023] Input: Launching a smartphone app without any operations

[1024] Output: Current location data, schedule data, weather information, camera footage

[1025] Step 2:

[1026] The device sends the collected data to the cloud server, where the network module inside the device ensures efficient and secure transmission. The data is encrypted before transmission.

[1027] Input: Current location data, schedule data, weather information, camera footage

[1028] Output: Consolidated data sent to the cloud server

[1029] Step 3:

[1030] The server analyzes the received data and identifies the user's current situation and environment. It verifies the user's current location based on GPS data and the user's current schedule based on schedule data. It checks local weather conditions based on weather information and analyzes camera footage to understand the situation within the home (for example, the quietness of the living room).

[1031] Input: Integrated data (current location data, schedule data, weather information, camera footage)

[1032] Output: Analysis of the user's current situation and environment

[1033] Step 4:

[1034] Based on the analysis results, the server uses a generative AI model to generate music that is optimal for the current situation. To generate this music, the AI ​​is input with prompts that correspond to the user's situation and environment, and the resulting music data is obtained. For example, a prompt might be, "I'm currently at home, the camera is showing a quiet living room, and there are no particular events on my calendar. Please generate relaxing music that is appropriate for this situation."

[1035] Input: Analysis of the user's current situation and environment

[1036] Output: Generated music data

[1037] Step 5:

[1038] The generated music data is sent from the cloud server to the user's device, where it is compressed and securely transmitted to the device.

[1039] Input: Generated music data

[1040] Output: Music data sent to the user's device

[1041] Step 6:

[1042] The device decodes the received music data and plays it in real time through a dedicated application, allowing users to relax by listening to the provided music.

[1043] Input: Music data sent from the cloud server

[1044] Output: Music played in real time

[1045] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1046] This system combines a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information, with an emotion engine that recognizes the user's emotions. The introduction of the emotion engine makes it possible to generate music according to the user's emotional state, providing a more personalized music experience.

[1047] Server-side processing

[1048] 1. Data Reception

[1049] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, and weather data.

[1050] 2. Data Analysis

[1051] The server analyzes the received data to identify the user's current situation and environment. It also uses an emotion engine to recognize the user's emotions from the camera footage and audio data. This allows the server to identify the user's "emotional state" in addition to their "location" and "activity status."

[1052] 3. Music Generation

[1053] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. For example, if the user is driving along the coast and in a relaxed emotional state, refreshing, rhythmic music will be generated. On the other hand, if the user is feeling stressed, calm music for relaxation will be generated.

[1054] 4. Sending music

[1055] The server transmits the generated music data to the user's terminal.

[1056] Terminal side processing

[1057] 1. Data Collection

[1058] The device collects user location information using a GPS component, measures movement speed using an accelerometer, captures camera footage, and collects schedule and weather information.

[1059] An emotion engine is used to analyze the user's emotions from camera footage and audio data.

[1060] 2. Data Transmission

[1061] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[1062] 3. Music reception and playback

[1063] The terminal receives the music data sent from the server, decodes it, and plays it back.

[1064] User operations

[1065] 1. Start the service

[1066] The service starts when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback will be carried out automatically.

[1067] 2. Music experience

[1068] Users can enjoy music that matches their mood and situation in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play.

[1069] Example: When driving along the coast

[1070] 1. Data Collection

[1071] The device uses GPS to determine that the user is traveling along the coast, an accelerometer to measure the vehicle's speed, a camera to capture the coastal scenery, and an emotion engine to analyze the user's relaxed facial expression.

[1072] 2. Data Transmission

[1073] The device sends the collected location information, movement speed, camera image data, and emotion analysis results to the server.

[1074] 3. Data Analysis

[1075] The server analyzes the transmitted data and determines that the user is driving along the coast, and at the same time, determines from emotion analysis that the user is relaxing.

[1076] 4. Music Generation

[1077] The server generates music suitable for a coastal drive and a relaxed emotional state, for example, refreshing and rhythmic music.

[1078] 5. Sending and playing music

[1079] The server transmits the generated music data to the user's terminal, where it is played in real time.

[1080] In this way, the system of the present invention provides the user with a more comfortable music experience by taking into account not only the user's current situation but also their emotional state and providing optimal music in real time.

[1081] The processing flow will be explained below.

[1082] Step 1:

[1083] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[1084] Step 2:

[1085] The device uses a built-in camera to capture images of the user's face and surroundings. The captured images are sent to an emotion engine, which analyzes emotions from the user's facial expressions and voice data. The camera images are also used to identify the environment (e.g., beach, park, city, etc.).

[1086] Step 3:

[1087] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[1088] Step 4:

[1089] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[1090] Step 5:

[1091] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet includes location information, movement speed, camera image analysis results (environment identification and emotion analysis), schedule information, and weather information.

[1092] Step 6:

[1093] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[1094] Step 7:

[1095] The server's data analysis module analyzes the data to determine the user's current situation, environment, and emotional state. Location and movement speed data are used to determine where the user is and what they are doing. Analysis of the camera footage determines the surrounding environment (e.g., beach, park, etc.), and data from the emotion engine is used to determine the user's emotional state. Schedule information and weather information are also taken into account in the analysis.

[1096] Step 8:

[1097] The server's music generation AI generates music that best suits the situation based on the analysis results and the user's recognized emotional state. The music generation algorithm determines musical parameters such as tempo, key, and instrumentation, and generates the song based on them.

[1098] Step 9:

[1099] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[1100] Step 10:

[1101] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[1102] Step 11:

[1103] The device's music playback module plays the received music data, allowing users to enjoy music that best suits their situation, environment, and emotions in real time.

[1104] In this way, the system of the present invention utilizes a variety of data, such as location information, camera footage, schedules, weather information, and emotional data, to automatically generate music that matches the user's current situation and emotional state and provide it in real time.

[1105] Example 2

[1106] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] If the music provided is not suited to the user's real-time situation or emotional state, the music experience will be unsatisfying for the user, which may reduce the user's experience.

[1108] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring the user's current location, schedule, weather, and video data, means for transmitting the data acquired by the above means to a data center, means for analyzing the acquired data at the data center and identifying the user's current situation and environment, and emotion analysis means for recognizing the user's emotional state from the video data and audio data. This makes it possible to provide personalized music in real time based on the user's situation and emotional state.

[1109] "Current location of user" refers to the physical location information of the user in real time.

[1110] "Schedule" refers to information about activities or events that a user has set or planned in advance.

[1111] "Weather" is information about the current weather conditions, including temperature, humidity, wind speed, and precipitation.

[1112] "Video data" is visual data that records the situation and scenery around the user.

[1113] A "data center" is a facility where servers and storage devices are installed and where data is processed and stored.

[1114] "Emotion analysis means" is a technology for recognizing the emotional state of a user using video data and audio data.

[1115] "Location Information" means a user's real-time physical coordinate data obtained using GPS or other means.

[1116] "Moving speed" is data that measures the speed at which the user is moving.

[1117] "Activity status" refers to information about the user's current activities and means of transportation.

[1118] An "information terminal" is an electronic device that is capable of data communication and running applications, such as a smartphone, tablet, or PC.

[1119] "Musical parameters" refer to elements such as tempo, rhythm, and tone that are set when generating music.

[1120] This invention is a system that acquires and analyzes a user's current location, schedule, weather, and video data to understand the user's situation and emotional state, and generates and plays personalized music based on that information.By introducing an emotion analysis means, this invention achieves more advanced personalization according to the user's emotional state.

[1121] Server-side example

[1122] The server receives data sent from the user's device. This data includes the user's current location, schedule, weather, and video data. The data is acquired in real time using cloud services for rapid processing. For example, commonly used cloud services can be used for database storage and streaming services.

[1123] The received data is analyzed on the server. This analysis includes algorithms to understand the user's current situation and environment, as well as emotion analysis to recognize the user's emotional state from video and audio data. For example, image processing and machine learning frameworks such as OpenCV and TensorFlow are used to analyze video data.

[1124] Based on the analysis results, the server-side music generation AI generates music that best suits the user's situation and emotional state. This music generation uses the machine learning library Magenta and other technologies. The generated music is then sent to the user's device via a streaming protocol.

[1125] Example of terminal side

[1126] The device collects data from multiple sensors and applications. Specifically, it uses a GPS module to obtain the user's location and an accelerometer to measure their speed. It also captures camera footage and collects audio data from a microphone. This data is preprocessed by the device's applications and then sent to a data center.

[1127] The acquired data is sent to a server via an Internet connection, usually using the HTTP / HTTPS protocol. The music data sent from the server is decoded on the device and played in real time. The device has a music playback application that runs in the background, so no user interaction is required.

[1128] User operations

[1129] Users first install the dedicated app on their device and then start the service by launching the app. During the initial setup, the app is granted the necessary permissions (location information, camera, microphone, etc.). This allows data collection, transmission, and music playback to occur automatically.

[1130] Users can enjoy music that adapts to their situation and emotions in real time. For example, if you feel relaxed while driving along the coast, your device will automatically play refreshing, rhythmic music. This makes the user's music experience more personalized and improves their satisfaction.

[1131] Specific operation example

[1132] For example, if the user is driving along the coast, the following steps are performed:

[1133] 1. The device's GPS determines that the user is traveling along the coast, and the accelerometer measures that the car is moving at a constant speed.

[1134] 2. The camera captures the coastal scenery, and the emotion engine analyzes the user's relaxed facial expression.

[1135] 3. The device sends location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data to the server.

[1136] 4. The server analyzes this data and determines that the user is driving along the coast, and at the same time, determines from the emotion analysis results that the user is relaxing.

[1137] 5. The server's music generation AI generates music that best suits the user's situation and emotions, such as refreshing and rhythmic music.

[1138] 6. The generated music data is sent to the user's device using a streaming protocol and played in real time.

[1139] Prompt Sentence Examples

[1140] Examples of prompts that users can enter into a generative AI model:

[1141] "The user is driving along the coast and feeling relaxed. Generate refreshing, rhythmic music that suits this situation."

[1142] The system provides a personalized music experience by comprehensively considering the user's current situation and emotional state. The music generated in real time is designed to provide a more comfortable and satisfying experience for the user.

[1143] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1144] Server-side processing

[1145] Step 1: Receiving data

[1146] The server receives location, schedule, weather, and video data sent from the user's device. The server uses a stable Internet connection to receive the data quickly and accurately. The input data includes the user's real-time information, which is stored in the server's storage. The output data is stored in a database.

[1147] Step 2: Data analysis

[1148] The server analyzes the received data. Specifically, it uses location information to identify the user's current location and determines the user's mode of transportation (car, walking, etc.) from their movement speed. It then uses emotion analysis to recognize the user's emotional state (e.g., relaxed, stressed, etc.) from the video and audio data. This analysis uses image processing and machine learning libraries such as OpenCV and TensorFlow. The inputs are the user's location information, movement speed, video data, and audio data, and the output is the analysis results of the user's current location, mode of transportation, and emotional state.

[1149] Step 3: Music Generation

[1150] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. This music generation uses machine learning libraries such as Magenta. For example, if the user is in a relaxed emotional state while driving along the coast, refreshing, rhythmic music will be generated. The input is the analysis results, and the output is the generated music data.

[1151] Step 4: Send your music

[1152] The server sends the generated music data to the user's device. A streaming protocol (e.g., HTTP Live Streaming (HLS)) is used for transmission, allowing the user to enjoy music in real time. The input is the generated music data, and the output is the music data sent to the user's device.

[1153] Terminal side processing

[1154] Step 1: Data collection

[1155] The device collects the user's location information using a GPS module and measures their speed using an acceleration sensor. It also captures camera footage and obtains audio data from a microphone. These data are preprocessed by the device's application. The input is data from the sensors and application, and the output is processed location information, speed, camera footage, and audio data.

[1156] Step 2: Send data

[1157] The device sends the acquired data (location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data) to a server. A stable Internet connection is required for transmission, and the HTTP / HTTPS protocol is usually used. The input is the processed data, and the output is the data sent to the server.

[1158] Step 3: Receive and play music

[1159] The device receives the music data sent from the server, decodes it, and plays it. The device's music playback application runs in the background, playing music continuously even while the user is using other apps. The input is the music data sent from the server, and the output is music played in real time.

[1160] User operations

[1161] Step 1: Start the service

[1162] The service begins when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback are performed automatically. The only specific operations the user must perform are installing the app and configuring the initial setup. The input is installing and configuring the app, and the output is a system that is ready to use.

[1163] Step 2: Music experience

[1164] Users can enjoy music that matches their situation and emotions in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play from your device. The input is the user's real-time situation and emotional data, and the output is a music experience that matches that.

[1165] (Application example 2)

[1166] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1167] In modern food delivery services, delivery workers face a lot of stress and fatigue while performing their duties. This can lead to reduced delivery efficiency and a poor user experience. Furthermore, there is a lack of systems that can grasp the delivery worker's emotional state in real time and provide appropriate support. Therefore, there is a need for a system that provides appropriate music and notifications in real time according to the delivery worker's emotional state and environment.

[1168] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring the user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the above means to the server; means in the server for analyzing the acquired data and identifying the user's current situation and environment; an emotion engine that includes an emotion engine for analyzing the user's emotional state and for recognizing the user's emotional state using the emotion engine; means for generating music based on the situation, environment, and the user's emotional state; means for transmitting the generated music to the user's terminal; and means for playing the music transmitted from the server on the terminal. This reduces stress for delivery personnel by providing appropriate music and notifications in real time according to the delivery personnel's emotional state and environment, thereby enabling them to perform their work more efficiently.

[1169] "User's current location" refers to the location where a person is currently located, obtained using a location information system such as GPS.

[1170] A "schedule" is information that indicates a user's dates and plans, and is data that is managed through a calendar application or online service.

[1171] "Weather" refers to weather information for the user's location, including meteorological data such as temperature, precipitation, and wind speed.

[1172] "Camera footage" refers to video footage and photographic data captured through a digital camera.

[1173] "Means" refers to methods, equipment, or systems used to achieve a certain purpose.

[1174] A "server" refers to a computer system that processes data and provides services over a network.

[1175] "Conditions and environment" refers to the specific situation or surrounding environment in which the user is placed, and is information that includes the state at that time and external factors.

[1176] An "emotion engine" refers to software or algorithms that analyze and recognize a person's emotional state from camera footage and audio data.

[1177] "Generating music" refers to creating new music data using specific algorithms or AI.

[1178] "User's device" refers to electronic devices that are directly used by the user, such as smartphones, tablets, and personal computers.

[1179] "Analysis" refers to the process of carefully analyzing acquired data to derive specific information or conditions.

[1180] "Music playback" refers to the audible reproduction of generated music data through audio equipment such as speakers or headphones.

[1181] The following describes an embodiment of the present invention. First, the overall system configuration will be described, and then the role and processing of each component will be described in detail.

[1182] System configuration

[1183] The system mainly consists of a server, a user's device, and the user. The server analyzes data and generates music, while the user's device collects and transmits data, and receives and plays music. The detailed processing flow is as follows:

[1184] Server Processing

[1185] 1. Data Reception

[1186] The server receives various data sent from the user's device, including the user's current location using GPS, schedule, weather, camera footage, and emotional state analyzed by the emotion engine.

[1187] 2. Data Analysis

[1188] The server uses data analysis software and an emotion engine to analyze the received data, thereby determining the user's current situation, environment, and emotional state.

[1189] 3. Music Generation

[1190] The server's music generation AI generates optimal music based on the analyzed data. For example, if a delivery person is feeling stressed, it will generate relaxing music. The software used is a music generation AI model.

[1191] 4. Sending music

[1192] The server transmits the generated music data to the user's terminal.

[1193] Terminal handling

[1194] 1. Data Collection

[1195] The device uses a GPS component to collect user location information and an acceleration sensor to measure movement speed. It also acquires camera footage and collects schedule and weather information. The emotion engine analyzes the user's emotions from the camera footage and audio data.

[1196] 2. Data Transmission

[1197] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[1198] 3. Music reception and playback

[1199] The terminal receives the music data sent from the server, decodes it, and plays it back.

[1200] User operations

[1201] 1. Start the service

[1202] The user simply launches the dedicated app on the device and performs the initial setup. Data collection, transmission, and music playback then begin automatically, allowing the user to enjoy music tailored to their situation and emotions without any special operations.

[1203] Example: Delivery person

[1204] 1. Data Collection

[1205] The device uses GPS to identify the delivery person's current location, an accelerometer to measure their speed, a camera to capture video footage, and an emotion engine to analyze the delivery person's emotional state.

[1206] 2. Data Transmission

[1207] The collected data is sent to a server, which analyzes it.

[1208] 3. Music Generation

[1209] If the delivery person is feeling stressed, the server generates relaxing music and sends it to the terminal.

[1210] 4. Playing Music

[1211] The music sent is played on the device, allowing delivery personnel to enjoy music that suits their situation and emotions in real time.

[1212] Examples of prompt statements

[1213] Design an application that generates optimal music and notifications in real time based on the user's current location, weather, delivery route, camera footage, and emotional state. The music will help delivery drivers relieve stress and deliver efficiently, while the notifications will guide them to rest stops and generate thank-you messages for the delivery recipients.

[1214] In this way, the system of the present invention can provide a more comfortable working environment for delivery personnel by taking into account the user's current situation and emotional state and providing optimal music and notifications in real time.

[1215] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1216] Step 1: Data collection

[1217] The device uses a GPS component to collect the user's location information. It also measures the user's speed using an acceleration sensor. It also captures camera footage and obtains weather and schedule information from a weather API and schedule service. It uses an emotion engine to analyze the user's emotions from the camera footage and audio data. This data is used for subsequent processing, so it undergoes any necessary data processing (e.g., coordinate normalization, speed filtering, video data analysis, etc.).

[1218] Input: location information, movement speed, camera footage, weather information, schedule information, emotional data

[1219] Output: Analyzed location information, speed data, emotion data, weather data, schedule data

[1220] Step 2: Send data

[1221] The device sends the collected and analyzed data to a server. This data transmission is carried out via a network (Internet or mobile data communication). The server receives various data.

[1222] Input: Analyzed location information, speed data, emotion data, weather data, schedule data

[1223] Output: Send data to the server

[1224] Step 3: Receiving data

[1225] The server receives the data sent from the device, temporarily stores the data, and uses it for subsequent analysis.

[1226] Input: Data sent from the terminal

[1227] Output: Data stored on the server

[1228] Step 4: Data analysis

[1229] The server analyzes the received data and identifies the user's current situation and environment. This analysis process uses data analysis software and an emotion engine. For example, it determines the user's location from location information and recognizes their current emotional state from emotion data. Based on this, it sets the parameters necessary for music generation.

[1230] Input: Data stored on the server

[1231] Output: Analysis results (user situation, environment, emotional state)

[1232] Step 5: Music Generation

[1233] The server uses a music generation AI model to generate optimal music based on the analyzed data. For example, if the user is stressed, it generates relaxing music, and if the user is relaxed, it generates lively music. The generated music data has a style and tempo that suits the user's emotions and environment.

[1234] Input: Analysis results

[1235] Output: Generated music data

[1236] Step 6: Send music

[1237] The server transmits the generated music data to the user's terminal via a network.

[1238] Input: Generated music data

[1239] Output: Sending music data to the device

[1240] Step 7: Receive and play music

[1241] The device receives the music data sent from the server, decodes it, and plays it. The music is provided to the user in real time, providing an appropriate experience based on the user's current emotions and environment.

[1242] Input: Music data sent from the server

[1243] Output: Music being played

[1244] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1245] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1246] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1247] [Fourth embodiment]

[1248] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1249] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1250] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1251] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1252] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1253] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1254] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1255] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1256] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1257] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1258] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1259] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1260] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1261] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information. This system is composed of multiple components and can provide music that matches the user's situation in real time. An embodiment of the present invention will be described in detail below.

[1262] Server-side processing

[1263] 1. Data Reception

[1264] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, weather data, etc.

[1265] 2. Data Analysis

[1266] The server analyzes the received data and identifies the user's current situation and environment. Based on location information and movement speed data, it determines where the user is currently located and what activities they are engaged in. Weather information and schedule information are also used to obtain a more detailed understanding of the situation.

[1267] 3. Music Generation

[1268] The server's music generation AI uses the analyzed data to generate music suited to the situation. For example, if you're driving along the coast, refreshing, rhythmic music will be generated. Parameters such as tempo, key, and instrumentation are adjusted according to the user's situation.

[1269] 4. Sending music

[1270] The server transmits the generated music data to the user's terminal.

[1271] Terminal side processing

[1272] 1. Data Collection

[1273] The device collects the user's location using a GPS component, measures the speed of movement using an accelerometer, captures camera footage, retrieves schedules from a schedule app, and collects weather information from a weather app.

[1274] 2. Data Transmission

[1275] The terminal transmits the collected data to the server.

[1276] 3. Music reception and playback

[1277] The terminal receives the music data sent from the server, decodes it, and plays it back.

[1278] User operations

[1279] 1. Start the service

[1280] The service starts when the user launches a dedicated app on the device.

[1281] After the initial setup, data collection, transmission, and music playback are performed automatically, so no special operations are required by the user.

[1282] 2. Music experience

[1283] Users can enjoy music that is suited to the situation in real time. For example, refreshing rhythmic music will be automatically played while driving on the beach.

[1284] Example: Coastal driving

[1285] 1. Data Collection

[1286] The device uses GPS to determine when the user is traveling along the coast, an accelerometer to measure the speed of the vehicle, and a camera to capture the coastal scenery.

[1287] 2. Data Transmission

[1288] The device transmits the collected location information, movement speed, and camera image data to the server.

[1289] 3. Data Analysis

[1290] The server analyzes the data and determines that the user is driving along the coast. It also checks the weather and sees that it is a clear day.

[1291] 4. Music Generation

[1292] The server generates music suitable for a drive to the coast, such as refreshing rhythmic music or music that includes the sound of waves.

[1293] 5. Sending and playing music

[1294] The server transmits the generated music data to the user's terminal, where it is played in real time.

[1295] In this way, the system of the present invention can generate and play music that is optimal for the user's current situation, providing a comfortable musical experience.

[1296] The processing flow will be explained below.

[1297] Step 1:

[1298] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[1299] Step 2:

[1300] The device captures the surroundings using the built-in camera. The captured image is sent to the image analysis module, which analyzes the scene characteristics (e.g., beach, park, city, etc.). The analysis results are also temporarily stored in storage.

[1301] Step 3:

[1302] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[1303] Step 4:

[1304] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[1305] Step 5:

[1306] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet sent includes location information, movement speed, analysis results of camera footage, schedule information, and weather information.

[1307] Step 6:

[1308] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[1309] Step 7:

[1310] The server's data analysis module analyzes the data to determine the user's current situation and environment. Based on location information and movement speed data, it determines where the user is and what they are doing. The analysis of camera footage determines the surrounding environment (e.g., beach, park, etc.), and schedule information determines the user's current schedule. Weather information is also used as a reference.

[1311] Step 8:

[1312] The server's music generation AI uses the analysis results to generate music that is optimal for the user's current situation and environment. The music generation algorithm determines musical parameters such as tempo, key, and instrument composition, and generates the song based on them.

[1313] Step 9:

[1314] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[1315] Step 10:

[1316] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[1317] Step 11:

[1318] The device's music playback module plays the received music data, allowing users to enjoy music that is best suited to their situation and environment in real time.

[1319] In this way, the system of the present invention uses a variety of data, such as location information, camera footage, schedules, and weather information, to automatically generate music that matches the user's current situation and provide it in real time.

[1320] Example 1

[1321] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] Conventional music playback systems have been unable to provide music that is suited to a user's individual state or environment in real time. As a result, it has been difficult to provide a music experience that matches the user's activity and emotions. It has been particularly challenging to provide appropriate music to users who are on the move or attending a specific event.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1324] In this invention, the server includes means for acquiring user location information, schedule, weather, and video data, means for transmitting the data acquired by the above means to the server, means for analyzing the acquired data in the server and identifying the user's current state and environment, means for generating music based on the state and environment, means for transmitting the generated music to the user's device, and means for playing the music transmitted from the server in the device, thereby making it possible to provide music that is optimal for the user's real-time situation and environment.

[1325] "Location information" is geographical data for identifying the user's current location.

[1326] "Schedule" is data that includes information about a user's plans and events.

[1327] "Weather" is data about weather conditions in a particular area.

[1328] "Video data" is visual information captured by a device such as a camera.

[1329] A "server" is a computer system for processing and managing data.

[1330] "Device" refers to a terminal or device used by a user, including a smartphone, tablet, etc.

[1331] "Analysis" is the process of interpreting acquired data and identifying the user's condition and environment.

[1332] "Music generation" is the process of creating music based on specific conditions or circumstances.

[1333] "Transmission" is the process of transferring data or information from one point to another.

[1334] "Playback" is the process of decoding received music data and playing the music through an audio output device.

[1335] The present invention is a system that acquires a user's current location, schedule, weather, and video data, and generates and plays optimal music based on that information. This system is composed of a server and a user's terminal, and specific embodiments are shown below.

[1336] Server-side processing

[1337] The server receives location information, weather information, schedule data, and video data sent from the user's device via an Internet connection.

[1338] The server analyzes the various data it receives. It analyzes location information and movement speed data to determine the user's current location and movement status. By combining this with weather information and schedule information, it can grasp the user's detailed situation. For example, if the user is driving along the coast, the received location information and weather information can determine that the coast is sunny and clear.

[1339] Based on this analysis data, the music generation AI installed on the server generates appropriate music. A prompt sentence is used to generate the music. The following is an example:

[1340] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[1341] The server transmits the generated music data to the user's device in real time, where it is decoded and played back through a playback device (speakers or headphones).

[1342] Terminal side processing

[1343] The device acquires the user's location information using the built-in GPS module, measures the speed of movement using the accelerometer, captures surrounding images using the camera, obtains schedules from a schedule app, and collects current weather information from a weather app.

[1344] The collected data is sent in real time to a server via an internet connection, making it immediately available for analysis.

[1345] The device receives the music data sent from the server, decodes it, and plays it back, allowing users to enjoy a real-time music experience.

[1346] User operations

[1347] Users simply launch the dedicated app on their device to start the service. Once the initial setup is complete, all processes, including data collection, transmission, and music playback, are carried out automatically, with no special operations required from the user.

[1348] Example: Coastal driving

[1349] The device uses GPS to determine that the user is traveling along the coast. The acceleration sensor measures the vehicle's speed and the camera captures the coastal scenery. This data is sent to a server, which analyzes the data and determines that the user is driving along the coast. Weather information is used to confirm that the weather is clear, and music appropriate to the conditions is generated. For example, refreshing, rhythmic music or music including the sound of waves is generated. This generated music data is sent to the user's device and played in real time.

[1350] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1351] Server-side processing

[1352] Step 1: Receiving data

[1353] Input: Location, speed, camera footage, schedule, and weather data sent from the user's device.

[1354] Detailed operation: The server receives this data via HTTP requests or WebSocket communication.

[1355] Output: The received data is stored on the server.

[1356] Step 2: Data analysis

[1357] Input: Location, speed, camera footage, schedule, and weather data received in step 1.

[1358] Detailed operation: The server combines location information and movement speed to identify the user's specific activity status (e.g., driving, jogging, etc.). It also analyzes camera footage using image recognition technology to understand the user's surrounding environment. It checks the user's current activity and upcoming plans from schedule information and identifies the local weather using weather data.

[1359] Output: Analysis results that reflect the user's current state and environment.

[1360] Step 3: Music Generation

[1361] Input: Analysis results obtained in step 2.

[1362] Detailed operation: The music generation AI installed on the server creates a prompt based on the analysis results, and generates music according to the prompt according to the situation.

[1363] The user is currently driving along a coastal road on a sunny day. Generate some refreshing, rhythmic music. The tempo is 120 BPM and the key is C major. Use guitar and piano as the primary instruments.

[1364] Output: The generated music data.

[1365] Step 4: Send your music

[1366] Input: The music data generated in step 3.

[1367] Detailed operation: The server transmits the generated music data to the user's device in real time via HTTP streaming or WebSocket connection.

[1368] Output: The transmitted music data.

[1369] Terminal side processing

[1370] Step 1: Data collection

[1371] Input: User behavior, surrounding environment information, and weather and schedule information.

[1372] Detailed operation: The device collects location information using the built-in GPS module, measures movement speed using the accelerometer, captures surrounding images using the camera, obtains schedule data from the schedule app, and collects weather information from the weather app.

[1373] Output: Collected location information, movement speed, camera footage, schedule, and weather data.

[1374] Step 2: Send data

[1375] Input: Location, speed, camera footage, schedule, and weather data collected in Step 1.

[1376] How it works in detail: The device sends this data to a server in real time via an internet connection.

[1377] Output: The data sent to the server.

[1378] Step 3: Receive and play music

[1379] Input: Music data sent from the server.

[1380] Detailed operation: The device receives music data in real time, decodes it, and plays it through speakers or headphones.

[1381] Output: Music played for the user to listen to.

[1382] User operations

[1383] Step 1: Start the service

[1384] Detailed operation: The user launches the dedicated app on the device to start the service. After the initial setup is complete, the process of data collection, transmission, and music playback is automatic.

[1385] Step 2: Music experience

[1386] Detailed operation: Users can enjoy music that suits the situation in real time. This process does not require any special operation, as the system automatically generates and plays the music.

[1387] (Application example 1)

[1388] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1389] In modern life, there are many systems that provide optimal music based on the user's current location, schedule, weather, etc., but there is no established method for automatically generating music suitable for relaxation situations at home and providing it in real time. In particular, when a user wants to relax in a quiet environment at home, providing music that is appropriate for that situation is a challenge.

[1390] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1391] In this invention, the server includes means for acquiring the user's current location, schedule, weather, and camera footage, means for transmitting the data acquired by the above means to the server, and means for generating relaxing music based on the quiet conditions in the home acquired from the smartphone or home security camera, thereby enabling music appropriate for the conditions to be automatically generated and played when the user wants to relax at home.

[1392] "User's current location" refers to the physical location where the user is currently located.

[1393] "Schedule" refers to data in which a user records future plans and events.

[1394] "Weather" refers to data indicating current weather conditions.

[1395] "Camera Footage" refers to real-time video data captured by a camera.

[1396] A "smartphone" is a type of mobile phone, a small device with advanced computing capabilities.

[1397] "Home security camera" refers to a camera device installed to monitor the safety of a home.

[1398] A "server" refers to a computer system that performs processes such as sending and receiving data and analyzing data over a network.

[1399] "Relaxing music" refers to music designed to help a user relax.

[1400] "User's terminal" refers to an electronic device such as a computer or smartphone used by a user.

[1401] The present invention is a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on this data. This system can provide music that suits the situation in the home in real time. An embodiment of the present invention will be described in detail below.

[1402] System Configuration

[1403] 1. Hardware:

[1404] Smartphone (iOS or Android)

[1405] Home security cameras or doorbell cameras (e.g., regular home security cameras)

[1406] Network Connection

[1407] 2. Software:

[1408] Custom application (app for home music assistant)

[1409] Cloud servers (e.g., Amazon Web Services, Google Cloud)

[1410] Music generation AI (e.g., general generative AI models)

[1411] Program processing

[1412] 1. Data collection methods:

[1413] The system obtains the user's current location, schedule, weather information, and camera footage from smartphones and home security cameras, using the smartphone's GPS, calendar app, weather app, and camera footage.

[1414] 2. Means of data transmission:

[1415] This data is sent to a cloud server, where it is securely transmitted over the network.

[1416] 3. Data analysis methods:

[1417] The cloud server analyzes the transmitted data and determines the user's current situation, such as whether they are in a quiet living room, by using location data, camera footage, weather information, and schedule information.

[1418] 4. Music Generation Means:

[1419] Based on the analysis results, a generative AI model is used to generate relaxing music. For example, if you want to spend some quiet time at home, music using piano or ambient sounds will be generated.

[1420] 5. Music transmission method:

[1421] The generated music data is sent from the cloud server to the user's device.

[1422] 6. Music playback method:

[1423] The user's device (smartphone) decodes the received music data in real time and plays it back, allowing the user to listen to the music through a dedicated app.

[1424] Specific examples

[1425] A situation where the user wants to relax at home would occur if the camera footage detected a quiet living room and the user had no scheduled events on their calendar. In this situation, the user launches a dedicated app on their smartphone, which starts data collection. The data is sent to a cloud server and analyzed. The server determines from the quiet home environment that the user wants to relax, and uses a generative AI model to generate relaxing music. This music data is then sent to the user's smartphone and played in real time.

[1426] Prompt Sentence Examples

[1427] "I'm at home, my camera is showing a quiet living room, and I have nothing planned on my calendar. Please generate some relaxing music that's appropriate for this situation."

[1428] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1429] Step 1:

[1430] A user launches a dedicated application on their smartphone. The application obtains the user's current location using GPS and retrieves their schedule from a schedule application. It also collects weather information from a weather application on the smartphone. It also captures real-time video from a home security camera. The application collects and integrates this data.

[1431] Input: Launching a smartphone app without any operations

[1432] Output: Current location data, schedule data, weather information, camera footage

[1433] Step 2:

[1434] The device sends the collected data to the cloud server, where the network module inside the device ensures efficient and secure transmission. The data is encrypted before transmission.

[1435] Input: Current location data, schedule data, weather information, camera footage

[1436] Output: Consolidated data sent to the cloud server

[1437] Step 3:

[1438] The server analyzes the received data and identifies the user's current situation and environment. It verifies the user's current location based on GPS data and the user's current schedule based on schedule data. It checks local weather conditions based on weather information and analyzes camera footage to understand the situation within the home (for example, the quietness of the living room).

[1439] Input: Integrated data (current location data, schedule data, weather information, camera footage)

[1440] Output: Analysis of the user's current situation and environment

[1441] Step 4:

[1442] Based on the analysis results, the server uses a generative AI model to generate music that is optimal for the current situation. To generate this music, the AI ​​is input with prompts that correspond to the user's situation and environment, and the resulting music data is obtained. For example, a prompt might be, "I'm currently at home, the camera is showing a quiet living room, and there are no particular events on my calendar. Please generate relaxing music that is appropriate for this situation."

[1443] Input: Analysis of the user's current situation and environment

[1444] Output: Generated music data

[1445] Step 5:

[1446] The generated music data is sent from the cloud server to the user's device, where it is compressed and securely transmitted to the device.

[1447] Input: Generated music data

[1448] Output: Music data sent to the user's device

[1449] Step 6:

[1450] The device decodes the received music data and plays it in real time through a dedicated application, allowing users to relax by listening to the provided music.

[1451] Input: Music data sent from the cloud server

[1452] Output: Music played in real time

[1453] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1454] This system combines a system that acquires a user's current location, schedule, weather, and camera footage, and generates and plays optimal music based on that information, with an emotion engine that recognizes the user's emotions. The introduction of the emotion engine makes it possible to generate music according to the user's emotional state, providing a more personalized music experience.

[1455] Server-side processing

[1456] 1. Data Reception

[1457] The server receives various data sent from the user's device, including the user's current location, movement speed, camera footage, schedule, and weather data.

[1458] 2. Data Analysis

[1459] The server analyzes the received data to identify the user's current situation and environment. It also uses an emotion engine to recognize the user's emotions from the camera footage and audio data. This allows the server to identify the user's "emotional state" in addition to their "location" and "activity status."

[1460] 3. Music Generation

[1461] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. For example, if the user is driving along the coast and in a relaxed emotional state, refreshing, rhythmic music will be generated. On the other hand, if the user is feeling stressed, calm music for relaxation will be generated.

[1462] 4. Sending music

[1463] The server transmits the generated music data to the user's terminal.

[1464] Terminal side processing

[1465] 1. Data Collection

[1466] The device collects user location information using a GPS component, measures movement speed using an accelerometer, captures camera footage, and collects schedule and weather information.

[1467] An emotion engine is used to analyze the user's emotions from camera footage and audio data.

[1468] 2. Data Transmission

[1469] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[1470] 3. Music reception and playback

[1471] The terminal receives the music data sent from the server, decodes it, and plays it back.

[1472] User operations

[1473] 1. Start the service

[1474] The service starts when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback will be carried out automatically.

[1475] 2. Music experience

[1476] Users can enjoy music that matches their mood and situation in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play.

[1477] Example: When driving along the coast

[1478] 1. Data Collection

[1479] The device uses GPS to determine that the user is traveling along the coast, an accelerometer to measure the vehicle's speed, a camera to capture the coastal scenery, and an emotion engine to analyze the user's relaxed facial expression.

[1480] 2. Data Transmission

[1481] The device sends the collected location information, movement speed, camera image data, and emotion analysis results to the server.

[1482] 3. Data Analysis

[1483] The server analyzes the transmitted data and determines that the user is driving along the coast, and at the same time, determines from emotion analysis that the user is relaxing.

[1484] 4. Music Generation

[1485] The server generates music suitable for a coastal drive and a relaxed emotional state, for example, refreshing and rhythmic music.

[1486] 5. Sending and playing music

[1487] The server transmits the generated music data to the user's terminal, where it is played in real time.

[1488] In this way, the system of the present invention provides the user with a more comfortable music experience by taking into account not only the user's current situation but also their emotional state and providing optimal music in real time.

[1489] The processing flow will be explained below.

[1490] Step 1:

[1491] The device uses GPS to obtain the user's current location. The user's location information is obtained in the form of latitude and longitude. In addition, the device uses an acceleration sensor to measure the user's speed, and this data is temporarily stored in storage.

[1492] Step 2:

[1493] The device uses a built-in camera to capture images of the user's face and surroundings. The captured images are sent to an emotion engine, which analyzes emotions from the user's facial expressions and voice data. The camera images are also used to identify the environment (e.g., beach, park, city, etc.).

[1494] Step 3:

[1495] The device retrieves the current schedule from the user's schedule app. The retrieved schedule information, including the event name, start time, and end time, is also temporarily stored in storage.

[1496] Step 4:

[1497] The device obtains weather information for the current location through the weather app, including temperature, humidity, chance of precipitation, wind speed, etc., and stores it temporarily in the storage.

[1498] Step 5:

[1499] The device compiles all the data collected in steps 1 to 4 and sends it to the server. The data packet includes location information, movement speed, camera image analysis results (environment identification and emotion analysis), schedule information, and weather information.

[1500] Step 6:

[1501] The server receives the data sent from the terminal, and the received data is first passed to the data analysis module.

[1502] Step 7:

[1503] The server's data analysis module analyzes the data to determine the user's current situation, environment, and emotional state. Location and movement speed data are used to determine where the user is and what they are doing. Analysis of the camera footage determines the surrounding environment (e.g., beach, park, etc.), and data from the emotion engine is used to determine the user's emotional state. Schedule information and weather information are also taken into account in the analysis.

[1504] Step 8:

[1505] The server's music generation AI generates music that best suits the situation based on the analysis results and the user's recognized emotional state. The music generation algorithm determines musical parameters such as tempo, key, and instrumentation, and generates the song based on them.

[1506] Step 9:

[1507] The server encodes the generated music data and transmits it to the user's device in a data packet containing the music file and its associated metadata.

[1508] Step 10:

[1509] The terminal receives the music data sent from the server, and the received data is first decoded and then passed to the music playback module.

[1510] Step 11:

[1511] The device's music playback module plays the received music data, allowing users to enjoy music that best suits their situation, environment, and emotions in real time.

[1512] In this way, the system of the present invention utilizes a variety of data, such as location information, camera footage, schedules, weather information, and emotional data, to automatically generate music that matches the user's current situation and emotional state and provide it in real time.

[1513] Example 2

[1514] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] If the music provided is not suited to the user's real-time situation or emotional state, the music experience will be unsatisfying for the user, which may reduce the user's experience.

[1516] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring the user's current location, schedule, weather, and video data, means for transmitting the data acquired by the above means to a data center, means for analyzing the acquired data at the data center and identifying the user's current situation and environment, and emotion analysis means for recognizing the user's emotional state from the video data and audio data. This makes it possible to provide personalized music in real time based on the user's situation and emotional state.

[1517] "Current location of user" refers to the physical location information of the user in real time.

[1518] "Schedule" refers to information about activities or events that a user has set or planned in advance.

[1519] "Weather" is information about the current weather conditions, including temperature, humidity, wind speed, and precipitation.

[1520] "Video data" is visual data that records the situation and scenery around the user.

[1521] A "data center" is a facility where servers and storage devices are installed and where data is processed and stored.

[1522] "Emotion analysis means" is a technology for recognizing the emotional state of a user using video data and audio data.

[1523] "Location Information" means a user's real-time physical coordinate data obtained using GPS or other means.

[1524] "Moving speed" is data that measures the speed at which the user is moving.

[1525] "Activity status" refers to information about the user's current activities and means of transportation.

[1526] An "information terminal" is an electronic device that is capable of data communication and running applications, such as a smartphone, tablet, or PC.

[1527] "Musical parameters" refer to elements such as tempo, rhythm, and tone that are set when generating music.

[1528] This invention is a system that acquires and analyzes a user's current location, schedule, weather, and video data to understand the user's situation and emotional state, and generates and plays personalized music based on that information.By introducing an emotion analysis means, this invention achieves more advanced personalization according to the user's emotional state.

[1529] Server-side example

[1530] The server receives data sent from the user's device. This data includes the user's current location, schedule, weather, and video data. The data is acquired in real time using cloud services for rapid processing. For example, commonly used cloud services can be used for database storage and streaming services.

[1531] The received data is analyzed on the server. This analysis includes algorithms to understand the user's current situation and environment, as well as emotion analysis to recognize the user's emotional state from video and audio data. For example, image processing and machine learning frameworks such as OpenCV and TensorFlow are used to analyze video data.

[1532] Based on the analysis results, the server-side music generation AI generates music that best suits the user's situation and emotional state. This music generation uses the machine learning library Magenta and other technologies. The generated music is then sent to the user's device via a streaming protocol.

[1533] Example of terminal side

[1534] The device collects data from multiple sensors and applications. Specifically, it uses a GPS module to obtain the user's location and an accelerometer to measure their speed. It also captures camera footage and collects audio data from a microphone. This data is preprocessed by the device's applications and then sent to a data center.

[1535] The acquired data is sent to a server via an Internet connection, usually using the HTTP / HTTPS protocol. The music data sent from the server is decoded on the device and played in real time. The device has a music playback application that runs in the background, so no user interaction is required.

[1536] User operations

[1537] Users first install the dedicated app on their device and then start the service by launching the app. During the initial setup, the app is granted the necessary permissions (location information, camera, microphone, etc.). This allows data collection, transmission, and music playback to occur automatically.

[1538] Users can enjoy music that adapts to their situation and emotions in real time. For example, if you feel relaxed while driving along the coast, your device will automatically play refreshing, rhythmic music. This makes the user's music experience more personalized and improves their satisfaction.

[1539] Specific operation example

[1540] For example, if the user is driving along the coast, the following steps are performed:

[1541] 1. The device's GPS determines that the user is traveling along the coast, and the accelerometer measures that the car is moving at a constant speed.

[1542] 2. The camera captures the coastal scenery, and the emotion engine analyzes the user's relaxed facial expression.

[1543] 3. The device sends location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data to the server.

[1544] 4. The server analyzes this data and determines that the user is driving along the coast, and at the same time, determines from the emotion analysis results that the user is relaxing.

[1545] 5. The server's music generation AI generates music that best suits the user's situation and emotions, such as refreshing and rhythmic music.

[1546] 6. The generated music data is sent to the user's device using a streaming protocol and played in real time.

[1547] Prompt Sentence Examples

[1548] Examples of prompts that users can enter into a generative AI model:

[1549] "The user is driving along the coast and feeling relaxed. Generate refreshing, rhythmic music that suits this situation."

[1550] The system provides a personalized music experience by comprehensively considering the user's current situation and emotional state. The music generated in real time is designed to provide a more comfortable and satisfying experience for the user.

[1551] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1552] Server-side processing

[1553] Step 1: Receiving data

[1554] The server receives location, schedule, weather, and video data sent from the user's device. The server uses a stable Internet connection to receive the data quickly and accurately. The input data includes the user's real-time information, which is stored in the server's storage. The output data is stored in a database.

[1555] Step 2: Data analysis

[1556] The server analyzes the received data. Specifically, it uses location information to identify the user's current location and determines the user's mode of transportation (car, walking, etc.) from their movement speed. It then uses emotion analysis to recognize the user's emotional state (e.g., relaxed, stressed, etc.) from the video and audio data. This analysis uses image processing and machine learning libraries such as OpenCV and TensorFlow. The inputs are the user's location information, movement speed, video data, and audio data, and the output is the analysis results of the user's current location, mode of transportation, and emotional state.

[1557] Step 3: Music Generation

[1558] The server's music generation AI generates music that best suits the situation based on the analyzed data and the user's recognized emotional state. This music generation uses machine learning libraries such as Magenta. For example, if the user is in a relaxed emotional state while driving along the coast, refreshing, rhythmic music will be generated. The input is the analysis results, and the output is the generated music data.

[1559] Step 4: Send your music

[1560] The server sends the generated music data to the user's device. A streaming protocol (e.g., HTTP Live Streaming (HLS)) is used for transmission, allowing the user to enjoy music in real time. The input is the generated music data, and the output is the music data sent to the user's device.

[1561] Terminal side processing

[1562] Step 1: Data collection

[1563] The device collects the user's location information using a GPS module and measures their speed using an acceleration sensor. It also captures camera footage and obtains audio data from a microphone. These data are preprocessed by the device's application. The input is data from the sensors and application, and the output is processed location information, speed, camera footage, and audio data.

[1564] Step 2: Send data

[1565] The device sends the acquired data (location information, movement speed, analysis results of camera footage, schedule information, weather information, and recognized emotion data) to a server. A stable Internet connection is required for transmission, and the HTTP / HTTPS protocol is usually used. The input is the processed data, and the output is the data sent to the server.

[1566] Step 3: Receive and play music

[1567] The device receives the music data sent from the server, decodes it, and plays it. The device's music playback application runs in the background, playing music continuously even while the user is using other apps. The input is the music data sent from the server, and the output is music played in real time.

[1568] User operations

[1569] Step 1: Start the service

[1570] The service begins when the user launches the dedicated app on the device. After the initial setup, data collection, transmission, and music playback are performed automatically. The only specific operations the user must perform are installing the app and configuring the initial setup. The input is installing and configuring the app, and the output is a system that is ready to use.

[1571] Step 2: Music experience

[1572] Users can enjoy music that matches their situation and emotions in real time. For example, if you're feeling relaxed while driving along the coast, music with a refreshing rhythm will automatically play from your device. The input is the user's real-time situation and emotional data, and the output is a music experience that matches that.

[1573] (Application example 2)

[1574] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1575] In modern food delivery services, delivery workers face a lot of stress and fatigue while performing their duties. This can lead to reduced delivery efficiency and a poor user experience. Furthermore, there is a lack of systems that can grasp the delivery worker's emotional state in real time and provide appropriate support. Therefore, there is a need for a system that provides appropriate music and notifications in real time according to the delivery worker's emotional state and environment.

[1576] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for acquiring the user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the above means to the server; means in the server for analyzing the acquired data and identifying the user's current situation and environment; an emotion engine that includes an emotion engine for analyzing the user's emotional state and for recognizing the user's emotional state using the emotion engine; means for generating music based on the situation, environment, and the user's emotional state; means for transmitting the generated music to the user's terminal; and means for playing the music transmitted from the server on the terminal. This reduces stress for delivery personnel by providing appropriate music and notifications in real time according to the delivery personnel's emotional state and environment, thereby enabling them to perform their work more efficiently.

[1577] "User's current location" refers to the location where a person is currently located, obtained using a location information system such as GPS.

[1578] A "schedule" is information that indicates a user's dates and plans, and is data that is managed through a calendar application or online service.

[1579] "Weather" refers to weather information for the user's location, including meteorological data such as temperature, precipitation, and wind speed.

[1580] "Camera footage" refers to video footage and photographic data captured through a digital camera.

[1581] "Means" refers to methods, equipment, or systems used to achieve a certain purpose.

[1582] A "server" refers to a computer system that processes data and provides services over a network.

[1583] "Conditions and environment" refers to the specific situation or surrounding environment in which the user is placed, and is information that includes the state at that time and external factors.

[1584] An "emotion engine" refers to software or algorithms that analyze and recognize a person's emotional state from camera footage and audio data.

[1585] "Generating music" refers to creating new music data using specific algorithms or AI.

[1586] "User's device" refers to electronic devices that are directly used by the user, such as smartphones, tablets, and personal computers.

[1587] "Analysis" refers to the process of carefully analyzing acquired data to derive specific information or conditions.

[1588] "Music playback" refers to the audible reproduction of generated music data through audio equipment such as speakers or headphones.

[1589] The following describes an embodiment of the present invention. First, the overall system configuration will be described, and then the role and processing of each component will be described in detail.

[1590] System configuration

[1591] The system mainly consists of a server, a user's device, and the user. The server analyzes data and generates music, while the user's device collects and transmits data, and receives and plays music. The detailed processing flow is as follows:

[1592] Server Processing

[1593] 1. Data Reception

[1594] The server receives various data sent from the user's device, including the user's current location using GPS, schedule, weather, camera footage, and emotional state analyzed by the emotion engine.

[1595] 2. Data Analysis

[1596] The server uses data analysis software and an emotion engine to analyze the received data, thereby determining the user's current situation, environment, and emotional state.

[1597] 3. Music Generation

[1598] The server's music generation AI generates optimal music based on the analyzed data. For example, if a delivery person is feeling stressed, it will generate relaxing music. The software used is a music generation AI model.

[1599] 4. Sending music

[1600] The server transmits the generated music data to the user's terminal.

[1601] Terminal handling

[1602] 1. Data Collection

[1603] The device uses a GPS component to collect user location information and an acceleration sensor to measure movement speed. It also acquires camera footage and collects schedule and weather information. The emotion engine analyzes the user's emotions from the camera footage and audio data.

[1604] 2. Data Transmission

[1605] The terminal transmits the acquired location information, movement speed, analysis results of the camera image, schedule information, weather information, and recognized emotion data to the server.

[1606] 3. Music reception and playback

[1607] The terminal receives the music data sent from the server, decodes it, and plays it back.

[1608] User operations

[1609] 1. Start the service

[1610] The user simply launches the dedicated app on the device and performs the initial setup. Data collection, transmission, and music playback then begin automatically, allowing the user to enjoy music tailored to their situation and emotions without any special operations.

[1611] Example: Delivery person

[1612] 1. Data Collection

[1613] The device uses GPS to identify the delivery person's current location, an accelerometer to measure their speed, a camera to capture video footage, and an emotion engine to analyze the delivery person's emotional state.

[1614] 2. Data Transmission

[1615] The collected data is sent to a server, which analyzes it.

[1616] 3. Music Generation

[1617] If the delivery person is feeling stressed, the server generates relaxing music and sends it to the terminal.

[1618] 4. Playing Music

[1619] The music sent is played on the device, allowing delivery personnel to enjoy music that suits their situation and emotions in real time.

[1620] Examples of prompt statements

[1621] Design an application that generates optimal music and notifications in real time based on the user's current location, weather, delivery route, camera footage, and emotional state. The music will help delivery drivers relieve stress and deliver efficiently, while the notifications will guide them to rest stops and generate thank-you messages for the delivery recipients.

[1622] In this way, the system of the present invention can provide a more comfortable working environment for delivery personnel by taking into account the user's current situation and emotional state and providing optimal music and notifications in real time.

[1623] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1624] Step 1: Data collection

[1625] The device uses a GPS component to collect the user's location information. It also measures the user's speed using an acceleration sensor. It also captures camera footage and obtains weather and schedule information from a weather API and schedule service. It uses an emotion engine to analyze the user's emotions from the camera footage and audio data. This data is used for subsequent processing, so it undergoes any necessary data processing (e.g., coordinate normalization, speed filtering, video data analysis, etc.).

[1626] Input: location information, movement speed, camera footage, weather information, schedule information, emotional data

[1627] Output: Analyzed location information, speed data, emotion data, weather data, schedule data

[1628] Step 2: Send data

[1629] The device sends the collected and analyzed data to a server. This data transmission is carried out via a network (Internet or mobile data communication). The server receives various data.

[1630] Input: Analyzed location information, speed data, emotion data, weather data, schedule data

[1631] Output: Send data to the server

[1632] Step 3: Receiving data

[1633] The server receives the data sent from the device, temporarily stores the data, and uses it for subsequent analysis.

[1634] Input: Data sent from the terminal

[1635] Output: Data stored on the server

[1636] Step 4: Data analysis

[1637] The server analyzes the received data and identifies the user's current situation and environment. This analysis process uses data analysis software and an emotion engine. For example, it determines the user's location from location information and recognizes their current emotional state from emotion data. Based on this, it sets the parameters necessary for music generation.

[1638] Input: Data stored on the server

[1639] Output: Analysis results (user situation, environment, emotional state)

[1640] Step 5: Music Generation

[1641] The server uses a music generation AI model to generate optimal music based on the analyzed data. For example, if the user is stressed, it generates relaxing music, and if the user is relaxed, it generates lively music. The generated music data has a style and tempo that suits the user's emotions and environment.

[1642] Input: Analysis results

[1643] Output: Generated music data

[1644] Step 6: Send music

[1645] The server transmits the generated music data to the user's terminal via a network.

[1646] Input: Generated music data

[1647] Output: Sending music data to the device

[1648] Step 7: Receive and play music

[1649] The device receives the music data sent from the server, decodes it, and plays it. The music is provided to the user in real time, providing an appropriate experience based on the user's current emotions and environment.

[1650] Input: Music data sent from the server

[1651] Output: Music being played

[1652] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1653] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1654] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1655] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1656] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1657] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1658] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1659] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1660] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1661] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1662] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1663] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1664] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1665] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1666] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1667] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1668] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1669] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1670] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1671] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1672] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1673] The following is further disclosed regarding the above embodiment.

[1674] (Claim 1)

[1675] [Means for obtaining a user's current location, schedule, weather, and camera footage;

[1676] [Means for transmitting the data acquired by the above means to a server;

[1677] [Means for analyzing the acquired data in the server and identifying the user's current situation and environment;

[1678] [Means for generating music based on the situation or environment;

[1679] [means for transmitting the generated music to a user's terminal;

[1680] [Means for playing music transmitted from the server in the terminal;

[1681] A system including:

[1682] (Claim 2)

[1683] The system according to claim 1, wherein location information and movement speed are acquired to identify the user's activity status.

[1684] (Claim 3)

[1685] [The system according to claim 1, wherein the parameters of the music are determined based on the analysis results and taking into account the weather and time of day.

[1686] "Example 1"

[1687] (Claim 1)

[1688] [Means for obtaining user location information, schedule, weather and video data;

[1689] [Means for transmitting the data acquired by the above means to a server;

[1690] [Means for analyzing the acquired data in the server and identifying the user's current state and environment;

[1691] [Means for generating music based on the state and environment;

[1692] [means for transmitting the generated music to a user device;

[1693] [In the device, means for playing music transmitted from a server;

[1694] A system including:

[1695] (Claim 2)

[1696] The system according to claim 1, wherein location information and movement speed are acquired to identify the user's behavioral state.

[1697] (Claim 3)

[1698] [The system according to claim 1, which determines music parameters based on the analysis results, taking into account the weather and time of day.

[1699] "Application Example 1"

[1700] (Claim 1)

[1701] [Means for obtaining a user's current location, schedule, weather, and camera footage;

[1702] [Means for transmitting the data acquired by the above means to a server;

[1703] [Means for analyzing the acquired data in the server and identifying the user's current situation and environment;

[1704] [Means for generating music based on the situation or environment;

[1705] [means for transmitting the generated music to a user's terminal;

[1706] [Means for playing music transmitted from the server in the terminal;

[1707] [Means for generating relaxing music based on a quiet situation in the home obtained from a smartphone or a home security camera;

[1708] A system including:

[1709] (Claim 2)

[1710] The system according to claim 1, wherein location information and movement speed are acquired to identify the user's activity status.

[1711] (Claim 3)

[1712] [The system according to claim 1, wherein the parameters of the music are determined based on the analysis results and taking into account the weather and time of day.

[1713] "Example 2: Combining Emotion Engines"

[1714] (Claim 1)

[1715] [means for obtaining a user's current location, schedule, weather, and video data;

[1716] [Means for transmitting the data acquired by the above means to a data center;

[1717] [Means for analyzing the acquired data at the data center and identifying the user's current situation and environment;

[1718] [Means including emotion analysis means for recognizing an emotional state from video data and audio data;

[1719] [means for generating music based on the situation, environment and emotional state;

[1720] [Means for transmitting the generated music to a user's information terminal;

[1721] [Means for playing music transmitted from the data center in the information terminal;

[1722] A system including:

[1723] (Claim 2)

[1724] The system according to claim 1, wherein location information and movement speed are acquired to identify the user's activity status.

[1725] (Claim 3)

[1726] [The system according to claim 1, wherein the parameters of the music are determined based on the analysis results, taking into consideration the weather, time of day, and the emotional state of the user.

[1727] "Application example 2 when combining emotion engines"

[1728] (Claim 1)

[1729] [Means for obtaining a user's current location, schedule, weather, and camera footage;

[1730] [Means for transmitting the data acquired by the above means to a server;

[1731] [Means for analyzing the acquired data in the server and identifying the user's current situation and environment;

[1732] [including an emotion engine for analyzing the user's emotion state, and a means for recognizing the user's emotion state by the emotion engine;

[1733] [means for generating music based on the situation, environment, and the user's emotional state;

[1734] [means for transmitting the generated music to a user's terminal;

[1735] [Means for playing music transmitted from the server in the terminal;

[1736] A system including:

[1737] (Claim 2)

[1738] The system according to claim 1, wherein location information and movement speed are acquired to identify the user's activity status.

[1739] (Claim 3)

[1740] [The system according to claim 1, wherein the parameters of the music are determined based on the analysis results and taking into account the weather and time of day. [Explanation of symbols]

[1741] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for obtaining a user's current location, schedule, weather, and camera footage; means for transmitting the data acquired by the means to a server; a means for analyzing the acquired data in the server and identifying the user's current situation and environment; means for generating music based on the situation or environment; means for transmitting the generated music to a user terminal; a means for playing the music transmitted from the server in the terminal; A system including:

2. The system according to claim 1 , wherein location information and movement speed are acquired to identify the user's activity status.

3. 2. The system according to claim 1, wherein the parameters of the music are determined based on the analysis results and taking into account the weather and time of day.

Images