System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving a specific request from a user; means for setting a parameter based on the received request; means for invoking a generative model that generates music using the set parameter; and means for providing the generated music to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current music streaming services require users to spend time searching for music they like, and listening to the same music repeatedly can lead to boredom. It's also difficult to instantly find the perfect music for a given situation or mood, making it difficult to select the right music to improve work efficiency or concentration. Therefore, there's a need for a new music experience that provides an environment where users can instantly find the perfect music for a given situation or mood, helping them to work more efficiently and change their mood. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: A system including means for receiving a specific request from a user, means for setting parameters based on the received request, means for invoking a generative model that generates music using the set parameters, and means for providing the generated music to the user. The system also includes means for saving the generated music and playing it later. Finally, the system includes means for generating music based on information about favorite music artists and songs registered in advance by the user. This allows the user to instantly experience music that best suits the scene and mood of the moment, contributing to improved work efficiency and a change of mood.

[0006] "Means for accepting requests" refers to the interface and functionality for receiving and analyzing requests from users.

[0007] "Means for setting parameters" refers to a process that determines, based on a user's request, the necessary setting values ​​to generate music that best suits that request.

[0008] A "generative model" refers to a system that includes an algorithm or machine learning model for generating music based on input parameters.

[0009] "Means for generating music" refers to a process and associated software that creates music in real time using set parameters.

[0010] "Means for providing music" refers to the process and associated interface that transmits the generated music to a user's device and makes it playable.

[0011] "Means for storing music" refers to a database or storage system for permanently storing the generated music.

[0012] "Means for playing music" refers to the process and playback functionality for retrieving and playing stored music.

[0013] "Means for generating music based on information" refers to the process of setting parameters in a music generation model based on information about artists and songs that the user has registered in advance, and creating music that matches that style.

[0014] "User" means any person or entity that uses the System to generate, store, and play music.

[0015] "Terminal" refers to any device used by a User to access the System, provide input, and play generated Music.

[0016] "Server" refers to a central computer installed to manage the entire system, process requests, generate music, etc. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] System Overview

[0039] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[0040] System configuration

[0041] 1. User Interface (UI)

[0042] The user enters the music request through a dedicated interface on the terminal, which includes input fields such as text boxes and drop-down menus.

[0043] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[0044] 2. Request Analysis Module

[0045] The server analyzes the request sent by the user, and this module extracts specific keywords and phrases and sets appropriate parameters based on the content.

[0046] How it works: For example, the tempo and genre (classical) parameters are set based on the keyword "improve concentration."

[0047] 3. Music Generation Module

[0048] The server invokes a music generation model based on the configured parameters, which generates music in real time using machine learning algorithms or pre-trained models.

[0049] What it does: For example, generate classical music with a tempo of 90 BPM and classical instrument parameters.

[0050] 4. Music provision module

[0051] The server provides the generated music to the user's device. This module converts the generated music data into an appropriate format and streams it in real time.

[0052] What it does: The generated music file is sent to the user as an HTTP response and is played immediately.

[0053] 5. Storage and playback module

[0054] Users can save their favorite music, which is then stored in a database.

[0055] The server receives requests to play stored music and provides it to the user.

[0056] How it works: By selecting the music you have created and pressing the save button, it will be saved in the database. The saved music can be played back later.

[0057] 6. Artist Registration Module

[0058] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0059] The server stores the registered information in a database and uses it when generating music.

[0060] How it works: After registering Artist A, when a user requests "music in Artist A's style that helps with concentration," the generative model generates music based on this information.

[0061] Specific examples

[0062] 1. Generate classical music to improve concentration

[0063] A user requests "classical music to improve concentration."

[0064] The terminal sends this request to the server.

[0065] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[0066] The music generation model generates music based on this.

[0067] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0068] 2. Generate music in the style of registered artists

[0069] The user has registered artist A in advance.

[0070] A user requests "relaxing music in the style of artist A."

[0071] The server refers to the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[0072] The music generation model generates music based on this.

[0073] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0074] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] The user inputs "concentration-boosting classical music" through a dedicated interface on the device.

[0078] How it works: The user enters their request into a text box on the interface and presses the submit button.

[0079] Step 2:

[0080] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[0081] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[0082] Step 3:

[0083] The server analyzes the received request and extracts the requested content.

[0084] What it does: Analyzes the data in the request body using JSON deserialization and extracts keywords such as "increases concentration" and "classical music."

[0085] Step 4:

[0086] The server sets the necessary parameters for the generative AI model based on the request.

[0087] What it does: Sets parameters such as tempo, genre, and instrumentation based on keywords (e.g., tempo 90 BPM, piano and strings).

[0088] Step 5:

[0089] The server calls the AI ​​music generation model using the set parameters.

[0090] What it does: Converts parameters to the model's input format and sends them to the model.

[0091] Step 6:

[0092] The generative AI model generates music based on the parameters it receives.

[0093] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[0094] Step 7:

[0095] The server acquires the generated music data.

[0096] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[0097] Step 8:

[0098] The server returns the generated music data to the terminal.

[0099] Operation: Sends a music file to the device as a response.

[0100] Step 9:

[0101] The device plays the received music.

[0102] Operation: Launches the device's audio player and plays the received music data.

[0103] Step 10:

[0104] The user inputs instructions into the terminal to save the music that he or she likes.

[0105] What it does: Press the Save button to send a save request.

[0106] Step 11:

[0107] The terminal sends a save request to the server.

[0108] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[0109] Step 12:

[0110] The server stores the music data in a database.

[0111] What it does: Inserts music data and related information into a database.

[0112] Step 13:

[0113] The user selects the music they want to play from the saved list.

[0114] How it works: Select a specific song from the saved list and press the play button.

[0115] Step 14:

[0116] The terminal sends a playback request to the server.

[0117] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[0118] Step 15:

[0119] The server retrieves the music data from the stored database and returns it to the terminal.

[0120] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[0121] Step 16:

[0122] The device plays the received music.

[0123] Operation: Launches the device's audio player and plays the received music data.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] Conventional music provision systems have difficulty generating music according to user requests and are unable to provide music that is optimal for a specific mood or situation in real time. This means that users have to spend time and effort finding music that suits their mood or situation. They also lack the ability to save their favorite music for easy access later. Furthermore, even if a user wanted music generated based on a specific artist or song, there was no way to achieve this.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes means for accepting specific requests from a user, means for setting parameters based on the accepted request, means for calling a generative AI model that generates music using the set parameters, means for providing the generated music to the user, means for saving the generated music, means for playing the saved music, means for pre-registering favorite music artists and songs, and means for generating music based on the registered information. This allows users to generate music in real time based on specific requests, allowing them to obtain music that best suits their mood and situation. It also allows users to easily save their favorite music and play it later. Furthermore, music can be generated based on specific artists or songs.

[0129] "User" refers to a person who utilizes the system to input specific requests and generate music.

[0130] "Request" refers to data input by a user to the system indicating specific musical generation requirements and wishes.

[0131] "Parameters" refer to the specific conditions and characteristics required for music generation that are set based on requirements, such as tempo and genre.

[0132] "Generative AI model" refers to a system component that uses machine learning algorithms or pre-trained models to generate music based on input parameters.

[0133] "Providing" refers to the act of the server sending the generated music to the user's device and the user receiving it in real time.

[0134] "Storage" refers to storing the generated music data in a storage device such as a database so that it can be accessed later.

[0135] "Playback" refers to retrieving stored music data and making it available for the user to listen to at any time.

[0136] "Artist" refers to the composer or performer of music that a user registers as a favorite.

[0137] "Song" refers to a specific piece of music that a user registers as a favorite.

[0138] MODE FOR CARRYING OUT THE INVENTION

[0139] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[0140] System configuration

[0141] 1. User Interface (UI)

[0142] A user accesses a dedicated interface via a terminal to input music requests, which includes input fields such as text boxes and drop-down menus.

[0143] The terminal receives a request input by a user and transmits it to the server.

[0144] 2. Request Analysis Module

[0145] The server analyzes requests sent by users. This module uses natural language processing (NLP) to extract keywords and phrases from the request and sets appropriate parameters based on the content. For example, a request for "classical music to improve concentration" would be converted to parameters of 90 BPM tempo and classical genre.

[0146] 3. Music Generation Module

[0147] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time.

[0148] The generative AI model generates music data in the form of MIDI notes or audio files based on parameters such as tempo and genre.

[0149] 4. Music provision module

[0150] The server sends the generated music data to the user's device via an HTTP response.

[0151] The device immediately plays the received music data. For example, it is possible to play streaming music on a browser using the Web Audio API.

[0152] 5. Storage and playback module

[0153] Users can use the interface to save their favorite music.

[0154] The server receives the save request and saves the generated music data in a database (e.g., MySQL). The user can then play the saved music later.

[0155] 6. Artist Registration Module

[0156] Users can register their favorite artists and songs in advance. This information is referenced when creating music, and the parameters of the music are set based on the registered information.

[0157] The server stores the registered artist information in a database and uses it when generating music. For example, if a user requests "relaxing music in the style of artist A," the generative AI model will generate music based on this information.

[0158] Examples of prompt statements

[0159] "Generate classical music to help you focus."

[0160] "Generate relaxing music in the style of artist A."

[0161] Specific examples

[0162] 1. Generate classical music to improve concentration

[0163] A user requests "classical music to improve concentration."

[0164] The terminal sends this request to the server.

[0165] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[0166] The music generation model generates music based on this.

[0167] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0168] 2. Generate music in the style of registered artists

[0169] The user has registered artist A in advance.

[0170] A user requests "relaxing music in the style of artist A."

[0171] The server references the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[0172] The music generation model generates music based on this.

[0173] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0174] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[0175] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0176] Step 1:

[0177] User request input

[0178] The user accesses a dedicated interface on the device and inputs a music request, specifically, by typing "classical music to improve concentration" in the text box.

[0179] Input: Request text from the user.

[0180] Output: The data format in which the request text is sent to the server.

[0181] Step 2:

[0182] Submitting a request

[0183] The terminal sends the user's request to the server using an HTTP POST request, which includes the requested data.

[0184] Input: The request text entered by the user.

[0185] Output: An HTTP POST request containing the request text.

[0186] Step 3:

[0187] Parsing the request

[0188] The server analyzes the request received from the user. Specifically, it uses natural language processing (NLP) to extract keywords and phrases from the request. For example, from a request for "classical music that improves concentration," it recognizes the keyword "improves concentration" and sets the tempo and genre (classical) as parameters.

[0189] Input: The request data received from the user.

[0190] Data processing: Use natural language processing to extract keywords and phrases and set appropriate parameters.

[0191] Output: Parameter data (tempo 90BPM, classical genre).

[0192] Step 4:

[0193] Music Generation

[0194] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time. The generated music data is in the form of MIDI notes or audio files.

[0195] Input: Set parameters (e.g. tempo 90 BPM, classical genre).

[0196] Data Computation: Generate parameter-based music using a music generation model.

[0197] Output: The generated musical data (e.g. MIDI notes or an audio file).

[0198] Step 5:

[0199] Music provided by

[0200] The server sends the generated music data to the user's device as an HTTP response.

[0201] The device immediately plays the received music data, specifically by streaming the music on the browser using the Web Audio API.

[0202] Input: Generated music data.

[0203] Output: The music stream that plays on the user's device.

[0204] Step 6:

[0205] Music Storage

[0206] The user has an interface (e.g., a save button) to save their favorite music.

[0207] The terminal sends a save request to the server.

[0208] The server receives the save request and saves the generated music data in a database (e.g., MySQL).

[0209] Input: The save request and the music data to save.

[0210] Output: Music data stored in a database.

[0211] Step 7:

[0212] Use of registered artist information

[0213] Users can register their favorite artists and songs in advance.

[0214] The terminal inputs the artist information and transmits a registration request to the server.

[0215] The server stores the registration information in a database. When generating music, parameters are set based on the registered artist information. For example, if you request "relaxing music in the style of artist A," the generative model will generate music based on this information.

[0216] Input: Registration information (artist name, etc.).

[0217] Data calculation: Set generation parameters based on artist information.

[0218] Output: The generated music data.

[0219] (Application example 1)

[0220] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0221] In brick-and-mortar stores, providing appropriate music in real time according to the situation and atmosphere of the store is important for attracting customers and improving customer satisfaction. However, conventional music playback systems only play pre-prepared playlists and lack the ability to dynamically adapt to the environment. This makes it difficult to instantly adjust the atmosphere of a brick-and-mortar store.

[0222] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0223] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, means for inputting a music request based on the situation of the physical store using a user interface, and means for generating music in real time based on the input request and providing it to the store's sound system, thereby making it possible to provide optimal music in real time based on the environment and situation of the physical store.

[0224] A "means for accepting specific user requests" is an interface through which a user can input their specific musical needs or requests.

[0225] The "means for setting parameters based on the received request" is a mechanism for analyzing the request received from the user and automatically setting the parameters required for music generation.

[0226] "Means for invoking a generative model that generates music using set parameters" is a function that invokes an algorithm or generative model for automatically generating music based on set parameters.

[0227] The "means for providing the generated music to the user" is a system for converting the generated music into an appropriate format and providing it to the user in real time.

[0228] "Means for inputting music requests based on the current situation of the physical store using a user interface" refers to an interface that allows store staff to input music requests based on the current situation of the store using a smartphone or other device.

[0229] "Means for generating music in real time based on input requests and providing it to the store's sound system" refers to a system for quickly generating music based on newly input music requests and instantly playing that sound source on the store's sound equipment.

[0230] The present invention includes a system for providing optimal music in real time in a brick-and-mortar store according to the environment and situation at the time. Specific embodiments of the system will be described below.

[0231] System Overview

[0232] The system includes a series of processes for receiving a specific request from a user, setting parameters based on the request, generating music based on the set parameters, and providing the generated music in real time. The system includes a means for inputting a music request according to the situation of the physical store through a user interface, and a means for generating music based on the input request and providing the music to the store's sound system.

[0233] Hardware and software configuration

[0234] Smartphone: A device used by store staff that provides a user interface for inputting music requests.

[0235] Store sound system: A speaker system for playing the generated music.

[0236] Server: Hosts the music generation API, receives and parses music requests, and invokes the music generation model with the appropriate parameters.

[0237] Music Generation API: A web service for generating music based on user requests. It uses machine learning algorithms and pre-trained models to generate music.

[0238] Processing flow

[0239] A user (store staff member) inputs a music request through a smartphone user interface, such as "relaxing jazz music."

[0240] The server analyzes the request received from the user and sets appropriate parameters (e.g., genre, tempo, atmosphere, etc.), then executes this based on the music generation API to generate music in real time.

[0241] The generated music is sent to the store's sound system via a server and played instantly, making it possible to provide optimal music in real time according to the environment and situation of the physical store.

[0242] Specific examples

[0243] For example, if a store staff member selects "relaxing jazz music" and inputs it into their smartphone, the request is sent to the server. The server then calls the music generation API based on the request, generates music with the appropriate parameters, and returns a URL for the generated music, which is then played over the store's speaker system.

[0244] Example prompt sentence:

[0245] Relaxing Jazz Music

[0246] This makes it possible to instantly adjust the atmosphere in the store according to the situation at hand.

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] A user inputs a music request using the smartphone's user interface. The input request is text data such as "relaxing jazz music." This input data is sent to the server.

[0250] Step 2:

[0251] The server analyzes the received request. Keywords such as "relax" or "jazz" are extracted from the input text data, and the parameters required for music generation (e.g., genre, tempo, atmosphere) are set based on this information. The parameters set by this data analysis are passed to the next step.

[0252] Step 3:

[0253] The server calls the music generation API based on the set parameters. The music generation API uses a machine learning algorithm to generate music in real time according to the specified parameters. The generated music data, such as a music file URL, is returned to the server.

[0254] Step 4:

[0255] The server receives the URL of the generated music and sends it to the smartphone, at which point the URL for playing the generated music is displayed on the user's smartphone.

[0256] Step 5:

[0257] The device then passes the URL to the store's sound system, which then uses the URL to stream the generated music and play it through the speakers. This process allows the music to be played in real time within the physical store.

[0258] The specific operations of each step are as follows:

[0259] Step 1: The user enters "relaxing jazz music" into the smartphone UI and presses the send button.

[0260] Step 2: The server analyzes the text data and sets the genre to "jazz" and the tempo to "relaxed."

[0261] Step 3: The server calls the music generation API, requests it to generate music using the specified parameters, and the API returns the URL of the corresponding music file.

[0262] Step 4: The server sends the URL to the smartphone and notifies the user.

[0263] Step 5: The smartphone provides the URL to the store's sound system, which plays the music through the speakers.

[0264] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0265] System Overview

[0266] The present invention is a system that generates and provides music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's input and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about artists and songs previously registered by the user.

[0267] System configuration

[0268] 1. User Interface (UI)

[0269] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[0270] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[0271] 2. Emotion Recognition Engine

[0272] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0273] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., happy, sad, angry).

[0274] How it works: For example, if the user is smiling, the emotion engine recognizes the emotion "joy."

[0275] 3. Request Analysis Module

[0276] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[0277] How it works: For example, set tempo and genre parameters based on keywords like "improve concentration" and "classical music" and the emotion "joy."

[0278] 4. Music Generation Module

[0279] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0280] Operation: Generate music at a tempo of 90 BPM with classical instrument parameters.

[0281] 5. Music Provision Module

[0282] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[0283] Operation: The generated music file is sent to the device as an HTTP response and is played in real time.

[0284] 6. Storage and playback module

[0285] Users can save their favorite music, which is then stored in a database.

[0286] The server receives requests to play stored music and provides it to the user.

[0287] How it works: By selecting the generated music and pressing the save button, it will be saved in the database and can be played back later.

[0288] 7. Artist Registration Module

[0289] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0290] The server stores the registered information in a database and uses it when generating the data.

[0291] How it works: When a user registers Artist A and requests "music in Artist A's style that helps with concentration," the generative model generates music based on that information.

[0292] Specific examples

[0293] 1. Generate classical music to improve concentration

[0294] The user requests "classical music to enhance concentration" and the emotion is recognized through the camera.

[0295] The terminal transmits the request and the emotion data to the server.

[0296] The server analyzes the music and sets additional parameters based on the tempo (90 BPM), classical genre, and recognized emotions.

[0297] A generative AI model generates music based on this.

[0298] The server sends the generated music to the device so that the user can listen to it.

[0299] 2. Generate music in the style of registered artists

[0300] A user registers artist A in advance and requests "relaxing music in artist A's style."

[0301] The terminal transmits the request and the emotion data to the server.

[0302] The server refers to the registration information and analysis data and sets appropriate parameters (e.g., artist A's characteristics, emotion "relaxed").

[0303] A generative AI model generates music based on this.

[0304] The server sends the generated music to the device so that the user can listen to it.

[0305] This system allows users to easily enjoy music that best suits their situation and emotions, improving work efficiency and changing their mood.

[0306] The processing flow will be explained below.

[0307] Step 1:

[0308] The user inputs "classical music to improve concentration" through the device's dedicated interface and presses the send button.

[0309] How it works: The user types their request into a text box on the interface.

[0310] Step 2:

[0311] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[0312] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[0313] Step 3:

[0314] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0315] How it works: Video and audio data acquired through sensors is processed in real time to extract facial expressions and vocal characteristics as emotional data.

[0316] Step 4:

[0317] The device sends the acquired emotion data to the server as an HTTP request.

[0318] What it does: Sends JSON-formatted data containing emotion data to the specified API endpoint.

[0319] Step 5:

[0320] The server analyzes the received request and extracts the request content and emotion data.

[0321] What it does: Analyzes the data in the request body using JSON deserialization and extracts "Improve concentration", "Classical music", and emotional data.

[0322] Step 6:

[0323] The server sets the necessary parameters for the generative AI model based on the request content and emotion data.

[0324] What it does: Sets parameters such as tempo, genre, and instrumentation based on the request and emotion data (e.g., tempo 90 BPM, classical genre, emotion "joy").

[0325] Step 7:

[0326] The server calls the AI ​​music generation model using the set parameters.

[0327] What it does: Converts parameters to the model's input format and sends them to the model.

[0328] Step 8:

[0329] The generative AI model generates music based on the parameters it receives.

[0330] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[0331] Step 9:

[0332] The server acquires the generated music data.

[0333] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[0334] Step 10:

[0335] The server returns the generated music data to the terminal.

[0336] Operation: Sends a music file to the device as a response.

[0337] Step 11:

[0338] The device plays the received music.

[0339] Operation: Launches the device's audio player and plays the received music data.

[0340] Step 12:

[0341] The user inputs instructions into the terminal to save the music that he or she likes.

[0342] What it does: Press the Save button to send a save request.

[0343] Step 13:

[0344] The terminal sends a save request to the server.

[0345] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[0346] Step 14:

[0347] The server stores the music data in a database.

[0348] What it does: Inserts music data and related information into a database.

[0349] Step 15:

[0350] The user selects the music they want to play from the saved list.

[0351] How it works: Select a specific song from the saved list and press the play button.

[0352] Step 16:

[0353] The terminal sends a playback request to the server.

[0354] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[0355] Step 17:

[0356] The server retrieves the music data from the stored database and returns it to the terminal.

[0357] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[0358] Step 18:

[0359] The device plays the received music.

[0360] Operation: Launches the device's audio player and plays the received music data.

[0361] Example 2

[0362] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0363] Conventional music generation systems have difficulty providing music that fully reflects a user's specific needs and emotions. Furthermore, the generated music may not be saved or played smoothly. Furthermore, they lack the functionality to generate music that reflects a user's favorite artists or songs. These issues limit users' ability to enjoy music that best suits their situation and emotions.

[0364] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0365] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for identifying an emotion based on the set parameters and facial expression analysis data, means for invoking a generative model that generates music based on the set parameters and the identified emotion, and means for providing the generated music to the user, thereby enabling the generation and provision of music that reflects the user's specific request and emotion.

[0366] The "means for accepting a specific request from a user" is a means for providing an interface for a user to input a specific request regarding music generation to the system.

[0367] The "means for setting parameters" is a means for determining various setting values ​​required for music generation based on the user's request and emotional data.

[0368] The "means for identifying emotions based on facial expression analysis data" is a means for analyzing sensor data such as the user's facial expressions and voice to identify the user's current emotional state.

[0369] The "means for invoking a generative model for generating music" refers to a means for invoking a machine learning model for generating music based on set parameters and identified emotions.

[0370] The "means for providing the generated music to the user" refers to a means for transmitting the generated music data to the user's terminal in an appropriate format so that the music data can be played back.

[0371] "Means for saving the generated music" refers to means for recording the generated music files in a database or storage so that they can be played back later.

[0372] "Means for playing stored music" refers to means for retrieving music files stored in a database or storage and making them playable on the user's device.

[0373] The "means of using sensors to acquire user emotional data" refers to a means of operating sensors such as cameras and microphones to collect data that grasps the user's current emotional state.

[0374] The "means for pre-registering favorite music artists and songs" is a means for providing an interface for a user to pre-record specific artists and songs into the system.

[0375] "Means for generating music based on registered information and emotional data" refers to means for setting music generation parameters and creating specific music, based on information about artists and songs registered in advance by the user and current emotional data.

[0376] MODE FOR CARRYING OUT THE INVENTION

[0377] The present invention is a system for generating and providing music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's requests and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about music artists and songs previously registered by the user.

[0378] System configuration

[0379] 1. User Interface (UI)

[0380] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[0381] Example: A user enters "Classical music to help me concentrate" as a request into the text box and clicks the submit button.

[0382] 2. Emotion Recognition Engine

[0383] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0384] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., "happy," "sad," "anger").

[0385] Example: If the user is smiling, the emotion engine will recognize the emotion "joy".

[0386] 3. Request Analysis Module

[0387] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[0388] Example: Set tempo and genre parameters based on the keywords "improves concentration" and "classical music" and the emotion "joy."

[0389] 4. Music Generation Module

[0390] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0391] Example: Generate music with a tempo of 90 BPM and classical instrument parameters.

[0392] 5. Music Provision Module

[0393] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[0394] Example: The generated music file is sent to the terminal as an HTTP response and played in real time.

[0395] 6. Storage and playback module

[0396] Users can save their favorite music, which is then stored in a database.

[0397] The server receives requests to play stored music and provides it to the user.

[0398] Example: By selecting the generated music and pressing the save button, it can be saved in the database and played back later.

[0399] 7. Artist Registration Module

[0400] Users can register their favorite music artists and songs in advance, and this information is referenced when creating music.

[0401] The server stores the registered information in a database and uses it when generating the data.

[0402] Example: If a user registers artist A and requests "music in artist A's style that helps with concentration," the generative model will generate music based on that information.

[0403] This allows users to easily enjoy music that best suits their situation and emotions. For example, if a user requests "classical music that enhances concentration through feelings of joy," the system generates music based on the user's request and emotions and provides it in real time. As another example, if a user requests "relaxing music in the style of artist A," the system can generate and provide music after setting parameters based on pre-registered artist information and the user's emotional data.

[0404] Prompt Sentence Examples

[0405] "Generate classical music that enhances concentration with feelings of joy"

[0406] "Generate relaxing music in the style of artist A"

[0407] As a result, users can easily create and enjoy music suited to a variety of situations.

[0408] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0409] Step 1: Enter and submit a user request

[0410] The user opens a dedicated interface on the device and enters a specific request for the music to be generated (e.g., "classical music to improve concentration") via text boxes and drop-down menus.

[0411] The terminal converts this input data into JSON format and sends it to the server as an HTTP request.

[0412] Input: User's music request

[0413] Output: HTTP request sent to the server

[0414] Step 2: Obtaining emotion data

[0415] The device uses sensors such as a camera and microphone to capture the user's facial expressions and tone of voice and analyze emotional data.

[0416] The server receives the emotion data sent from the terminal and sends it to an emotion recognition engine, which analyzes the data and identifies the user's current emotion.

[0417] Input: User facial and voice data

[0418] Output: Emotion data analyzed by the emotion recognition engine

[0419] Step 3: Analyzing Request and Sentiment Data

[0420] The server analyzes the request and emotion data sent by the user. The request analysis module sets parameters for music generation based on specific keywords, phrases, and emotions.

[0421] Input: User's music request and emotional data

[0422] Output: Parameters for music generation (e.g. tempo 90 BPM, classical genre)

[0423] Step 4: Invoke the model to generate music

[0424] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0425] Input: parameters for music generation

[0426] Output: Generated music file

[0427] Step 5: Submit your music

[0428] The server converts the generated music into an appropriate format (e.g., MP3) and provides it to the user's device. This module sends the generated music data as an HTTP response.

[0429] The device plays this music in real time.

[0430] Input: Generated music file

[0431] Output: Music data sent to the user's device

[0432] Step 6: Store and play your music

[0433] The user can save the generated music via a dedicated button.

[0434] The server records the saved music files in a database and provides them to the user upon receiving a playback request later.

[0435] Input: Generated music file

[0436] Output: Music files stored in the database and music files for playback

[0437] Step 7: Pre-register your artists and songs

[0438] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0439] The server stores information about registered artists and songs in a database and references it when necessary.

[0440] Input: Artist and song information registered by the user

[0441] Output: Artist and song information stored in a database

[0442] (Application example 2)

[0443] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0444] In conventional virtual stores, music generation based on user emotions was not possible, making it impossible to provide background music suited to individual users' emotions and requests in real time, which resulted in a lack of user engagement and personalized shopping experiences.

[0445] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0446] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request and emotion data acquired in real time, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, and means for playing the music generated based on the user's emotion in real time in a virtual store. This makes it possible to generate and provide music in real time that corresponds to the user's emotion.

[0447] A "specific request" is information that indicates a user's desires or requirements regarding specific music or emotions.

[0448] "Emotional data acquired in real time" refers to emotional information acquired by analyzing a user's facial expressions and tone of voice in real time using devices such as cameras and microphones.

[0449] "Parameters" are the settings and conditions used by the music generation model when generating music.

[0450] A "generative model" is a program for generating music using machine learning algorithms.

[0451] A "virtual store" is an online platform for providing products and services in a virtual space on the Internet.

[0452] "Means for providing music" refers to a method or device for allowing a user to listen to the generated music.

[0453] "Saved music" refers to music data that has been generated and stored in a database.

[0454] "Means for pre-registering favorite music artists and songs" refers to a method or interface that allows a user to pre-register specific artists or songs.

[0455] To implement this invention, it is necessary to build a system that generates music based on specific user requests and emotional data acquired in real time, and provides the music in a virtual store. This system is composed of a user interface, an emotion recognition engine, a request analysis module, a music generation module, a music provision module, a storage and playback module, and an artist registration module.

[0456] First, the user interface (UI) uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The user can then input their music request into the device's interface, for example, requesting "relaxing music."

[0457] An emotion recognition engine then analyzes the data captured by the camera and microphone to determine the user's current emotion, using OpenCV and Dlib for facial expression recognition and the Google Cloud Speech-to-Text API for voice emotion recognition.

[0458] The request analysis module then sets appropriate parameters based on the user's request text and the analyzed sentiment data, using a natural language processing library (e.g., spaCy) and TensorFlow for sentiment data analysis.

[0459] The music generation module uses a generative AI model (e.g., Magenta) based on the set parameters to generate music in real time. Here, a machine learning model is built using Python and Keras to generate music.

[0460] The music providing module plays the generated music in real time within the virtual store. The music data is sent to the terminal as an HTTP response and is played within the virtual store to which the user is connected.

[0461] Users can then use the save and play module to save the generated music. The saved music data is stored in a database such as MySQL or PostgreSQL, and can be played back from a playlist at any time.

[0462] In addition, users can use the artist registration module to pre-register their favorite artists and songs, which will allow for more personalized music to be generated based on the registered information.

[0463] Specific examples

[0464] For example, if a user requests "I want to relax" and uses the camera to recognize their emotions, the device sends the request and emotional data to the server. The server then analyzes the data to determine specific parameters for relaxation, such as tempo and music genre, and generates music in real time using a generative AI model (such as Magenta). The generated music is played in the virtual store, allowing users to enjoy shopping while listening to it. Users can also save the music if they want to play it again later.

[0465] Prompt Sentence Examples

[0466] As an example of input from the user, the following prompt sentence can be considered:

[0467] "The user is currently relaxing. The latest request is for relaxing music. Generative parameters for playing soft classical music as background noise in a virtual store setting."

[0468] As described above, the embodiments of the present invention make it possible to provide personalized music suited to the user's emotions, thereby dramatically improving the shopping experience in a virtual store.

[0469] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0470] Step 1:

[0471] Accepting user requests

[0472] The user inputs their music request through the smartphone interface, for example, requesting "relaxing music," and the input data is sent to the device in text format.

[0473] Input: User request (e.g. "relaxing music")

[0474] Output: Request text data

[0475] Step 2:

[0476] Collecting Emotional Data

[0477] The device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Image data is acquired from the camera, and audio data is acquired from the microphone.

[0478] Input: Real-time facial and voice data of the user

[0479] Output: Image data, audio data

[0480] Step 3:

[0481] Analysis by emotion recognition engine

[0482] The server analyzes the data obtained from the camera and microphone to identify the user's emotions. It uses OpenCV and Dlib for facial recognition and Google Cloud Speech-to-Text API for voice recognition. The analysis results in the user's current emotion.

[0483] Input: Image data, audio data

[0484] Data processing: facial expression analysis, voice tone analysis

[0485] Output: Emotion data (e.g. "Relaxed")

[0486] Step 4:

[0487] Request analysis and parameter setting

[0488] The server sets appropriate parameters based on the received user request text and the analyzed emotion data. This analysis uses a natural language processing library (e.g., spaCy) and TensorFlow for emotion data analysis.

[0489] Input: Request text data, emotion data

[0490] Data processing: Natural language processing analysis, emotion data integration

[0491] Output: Parameter setting data (e.g. tempo, music genre)

[0492] Step 5:

[0493] Music Generation

[0494] The server calls a generative AI model (e.g., Magenta) based on the set parameters and generates music in real time. The generative AI model uses machine learning algorithms to generate music.

[0495] Input: Parameter setting data

[0496] Data Computation: Real-time Music Generation with Generative AI Models

[0497] Output: Generated music data

[0498] Step 6:

[0499] Music provided by

[0500] The server provides the generated music to the virtual store in real time. The music data is sent to the terminal as an HTTP response, and the music is played in the virtual store to which the user is connected.

[0501] Input: Generated music data

[0502] Data processing: Music data format conversion and transmission

[0503] Output: Music playback on the user's device

[0504] Step 7:

[0505] Music Storage

[0506] Users can save their favorite music. The server accepts the save request and stores the music data in a database.

[0507] Input: User save request, generated music data

[0508] Data processing: Saving to database

[0509] Output: Data stored in the database

[0510] Step 8:

[0511] Playing music

[0512] The server receives a request to play the stored music and provides the music data to the user, which is retrieved from the playlist and played on the user's device.

[0513] Input: User playback request, data stored in database

[0514] Data processing: Retrieving and sending data from the database

[0515] Output: Music playback on the user's device

[0516] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0517] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0518] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0519] [Second embodiment]

[0520] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0521] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0522] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0523] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0524] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0525] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0526] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0527] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0528] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0529] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0530] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0531] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0532] System Overview

[0533] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[0534] System configuration

[0535] 1. User Interface (UI)

[0536] The user enters the music request through a dedicated interface on the terminal, which includes input fields such as text boxes and drop-down menus.

[0537] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[0538] 2. Request Analysis Module

[0539] The server analyzes the request sent by the user, and this module extracts specific keywords and phrases and sets appropriate parameters based on the content.

[0540] How it works: For example, the tempo and genre (classical) parameters are set based on the keyword "improve concentration."

[0541] 3. Music Generation Module

[0542] The server invokes a music generation model based on the configured parameters, which generates music in real time using machine learning algorithms or pre-trained models.

[0543] What it does: For example, generate classical music with a tempo of 90 BPM and classical instrument parameters.

[0544] 4. Music provision module

[0545] The server provides the generated music to the user's device. This module converts the generated music data into an appropriate format and streams it in real time.

[0546] What it does: The generated music file is sent to the user as an HTTP response and is played immediately.

[0547] 5. Storage and playback module

[0548] Users can save their favorite music, which is then stored in a database.

[0549] The server receives requests to play stored music and provides it to the user.

[0550] How it works: By selecting the music you have created and pressing the save button, it will be saved in the database. The saved music can be played back later.

[0551] 6. Artist Registration Module

[0552] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0553] The server stores the registered information in a database and uses it when generating music.

[0554] How it works: After registering Artist A, when a user requests "music in Artist A's style that helps with concentration," the generative model generates music based on this information.

[0555] Specific examples

[0556] 1. Generate classical music to improve concentration

[0557] A user requests "classical music to improve concentration."

[0558] The terminal sends this request to the server.

[0559] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[0560] The music generation model generates music based on this.

[0561] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0562] 2. Generate music in the style of registered artists

[0563] The user has registered artist A in advance.

[0564] A user requests "relaxing music in the style of artist A."

[0565] The server refers to the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[0566] The music generation model generates music based on this.

[0567] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0568] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[0569] The processing flow will be explained below.

[0570] Step 1:

[0571] The user inputs "concentration-boosting classical music" through a dedicated interface on the device.

[0572] How it works: The user enters their request into a text box on the interface and presses the submit button.

[0573] Step 2:

[0574] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[0575] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[0576] Step 3:

[0577] The server analyzes the received request and extracts the requested content.

[0578] What it does: Analyzes the data in the request body using JSON deserialization and extracts keywords such as "increases concentration" and "classical music."

[0579] Step 4:

[0580] The server sets the necessary parameters for the generative AI model based on the request.

[0581] What it does: Sets parameters such as tempo, genre, and instrumentation based on keywords (e.g., tempo 90 BPM, piano and strings).

[0582] Step 5:

[0583] The server calls the AI ​​music generation model using the set parameters.

[0584] What it does: Converts parameters to the model's input format and sends them to the model.

[0585] Step 6:

[0586] The generative AI model generates music based on the parameters it receives.

[0587] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[0588] Step 7:

[0589] The server acquires the generated music data.

[0590] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[0591] Step 8:

[0592] The server returns the generated music data to the terminal.

[0593] Operation: Sends a music file to the device as a response.

[0594] Step 9:

[0595] The device plays the received music.

[0596] Operation: Launches the device's audio player and plays the received music data.

[0597] Step 10:

[0598] The user inputs instructions into the terminal to save the music that he or she likes.

[0599] What it does: Press the Save button to send a save request.

[0600] Step 11:

[0601] The terminal sends a save request to the server.

[0602] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[0603] Step 12:

[0604] The server stores the music data in a database.

[0605] What it does: Inserts music data and related information into a database.

[0606] Step 13:

[0607] The user selects the music they want to play from the saved list.

[0608] How it works: Select a specific song from the saved list and press the play button.

[0609] Step 14:

[0610] The terminal sends a playback request to the server.

[0611] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[0612] Step 15:

[0613] The server retrieves the music data from the stored database and returns it to the terminal.

[0614] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[0615] Step 16:

[0616] The device plays the received music.

[0617] Operation: Launches the device's audio player and plays the received music data.

[0618] Example 1

[0619] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] Conventional music provision systems have difficulty generating music according to user requests and are unable to provide music that is optimal for a specific mood or situation in real time. This means that users have to spend time and effort finding music that suits their mood or situation. They also lack the ability to save their favorite music for easy access later. Furthermore, even if a user wanted music generated based on a specific artist or song, there was no way to achieve this.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0622] In this invention, the server includes means for accepting specific requests from a user, means for setting parameters based on the accepted request, means for calling a generative AI model that generates music using the set parameters, means for providing the generated music to the user, means for saving the generated music, means for playing the saved music, means for pre-registering favorite music artists and songs, and means for generating music based on the registered information. This allows users to generate music in real time based on specific requests, allowing them to obtain music that best suits their mood and situation. It also allows users to easily save their favorite music and play it later. Furthermore, music can be generated based on specific artists or songs.

[0623] "User" refers to a person who utilizes the system to input specific requests and generate music.

[0624] "Request" refers to data input by a user to the system indicating specific musical generation requirements and wishes.

[0625] "Parameters" refer to the specific conditions and characteristics required for music generation that are set based on requirements, such as tempo and genre.

[0626] "Generative AI model" refers to a system component that uses machine learning algorithms or pre-trained models to generate music based on input parameters.

[0627] "Providing" refers to the act of the server sending the generated music to the user's device and the user receiving it in real time.

[0628] "Storage" refers to storing the generated music data in a storage device such as a database so that it can be accessed later.

[0629] "Playback" refers to retrieving stored music data and making it available for the user to listen to at any time.

[0630] "Artist" refers to the composer or performer of music that a user registers as a favorite.

[0631] "Song" refers to a specific piece of music that a user registers as a favorite.

[0632] MODE FOR CARRYING OUT THE INVENTION

[0633] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[0634] System configuration

[0635] 1. User Interface (UI)

[0636] A user accesses a dedicated interface via a terminal to input music requests, which includes input fields such as text boxes and drop-down menus.

[0637] The terminal receives a request input by a user and transmits it to the server.

[0638] 2. Request Analysis Module

[0639] The server analyzes requests sent by users. This module uses natural language processing (NLP) to extract keywords and phrases from the request and sets appropriate parameters based on the content. For example, a request for "classical music to improve concentration" would be converted to parameters of 90 BPM tempo and classical genre.

[0640] 3. Music Generation Module

[0641] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time.

[0642] The generative AI model generates music data in the form of MIDI notes or audio files based on parameters such as tempo and genre.

[0643] 4. Music provision module

[0644] The server sends the generated music data to the user's device via an HTTP response.

[0645] The device immediately plays the received music data. For example, it is possible to play streaming music on a browser using the Web Audio API.

[0646] 5. Storage and playback module

[0647] Users can use the interface to save their favorite music.

[0648] The server receives the save request and saves the generated music data in a database (e.g., MySQL). The user can then play the saved music later.

[0649] 6. Artist Registration Module

[0650] Users can register their favorite artists and songs in advance. This information is referenced when creating music, and the parameters of the music are set based on the registered information.

[0651] The server stores the registered artist information in a database and uses it when generating music. For example, if a user requests "relaxing music in the style of artist A," the generative AI model will generate music based on this information.

[0652] Examples of prompt statements

[0653] "Generate classical music to help you focus."

[0654] "Generate relaxing music in the style of artist A."

[0655] Specific examples

[0656] 1. Generate classical music to improve concentration

[0657] A user requests "classical music to improve concentration."

[0658] The terminal sends this request to the server.

[0659] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[0660] The music generation model generates music based on this.

[0661] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0662] 2. Generate music in the style of registered artists

[0663] The user has registered artist A in advance.

[0664] A user requests "relaxing music in the style of artist A."

[0665] The server references the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[0666] The music generation model generates music based on this.

[0667] The server sends the generated music to the terminal so that the user can listen to it immediately.

[0668] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[0669] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] User request input

[0672] The user accesses a dedicated interface on the device and inputs a music request, specifically, by typing "classical music to improve concentration" in the text box.

[0673] Input: Request text from the user.

[0674] Output: The data format in which the request text is sent to the server.

[0675] Step 2:

[0676] Submitting a request

[0677] The terminal sends the user's request to the server using an HTTP POST request, which includes the requested data.

[0678] Input: The request text entered by the user.

[0679] Output: An HTTP POST request containing the request text.

[0680] Step 3:

[0681] Parsing the request

[0682] The server analyzes the request received from the user. Specifically, it uses natural language processing (NLP) to extract keywords and phrases from the request. For example, from a request for "classical music that improves concentration," it recognizes the keyword "improves concentration" and sets the tempo and genre (classical) as parameters.

[0683] Input: The request data received from the user.

[0684] Data processing: Use natural language processing to extract keywords and phrases and set appropriate parameters.

[0685] Output: Parameter data (tempo 90BPM, classical genre).

[0686] Step 4:

[0687] Music Generation

[0688] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time. The generated music data is in the form of MIDI notes or audio files.

[0689] Input: Set parameters (e.g. tempo 90 BPM, classical genre).

[0690] Data Computation: Generate parameter-based music using a music generation model.

[0691] Output: The generated musical data (e.g. MIDI notes or an audio file).

[0692] Step 5:

[0693] Music provided by

[0694] The server sends the generated music data to the user's device as an HTTP response.

[0695] The device immediately plays the received music data, specifically by streaming the music on the browser using the Web Audio API.

[0696] Input: Generated music data.

[0697] Output: The music stream that plays on the user's device.

[0698] Step 6:

[0699] Music Storage

[0700] The user has an interface (e.g., a save button) to save their favorite music.

[0701] The terminal sends a save request to the server.

[0702] The server receives the save request and saves the generated music data in a database (e.g., MySQL).

[0703] Input: The save request and the music data to save.

[0704] Output: Music data stored in a database.

[0705] Step 7:

[0706] Use of registered artist information

[0707] Users can register their favorite artists and songs in advance.

[0708] The terminal inputs the artist information and transmits a registration request to the server.

[0709] The server stores the registration information in a database. When generating music, parameters are set based on the registered artist information. For example, if you request "relaxing music in the style of artist A," the generative model will generate music based on this information.

[0710] Input: Registration information (artist name, etc.).

[0711] Data calculation: Set generation parameters based on artist information.

[0712] Output: The generated music data.

[0713] (Application example 1)

[0714] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0715] In brick-and-mortar stores, providing appropriate music in real time according to the situation and atmosphere of the store is important for attracting customers and improving customer satisfaction. However, conventional music playback systems only play pre-prepared playlists and lack the ability to dynamically adapt to the environment. This makes it difficult to instantly adjust the atmosphere of a brick-and-mortar store.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0717] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, means for inputting a music request based on the situation of the physical store using a user interface, and means for generating music in real time based on the input request and providing it to the store's sound system, thereby making it possible to provide optimal music in real time based on the environment and situation of the physical store.

[0718] A "means for accepting specific user requests" is an interface through which a user can input their specific musical needs or requests.

[0719] The "means for setting parameters based on the received request" is a mechanism for analyzing the request received from the user and automatically setting the parameters required for music generation.

[0720] "Means for invoking a generative model that generates music using set parameters" is a function that invokes an algorithm or generative model for automatically generating music based on set parameters.

[0721] The "means for providing the generated music to the user" is a system for converting the generated music into an appropriate format and providing it to the user in real time.

[0722] "Means for inputting music requests based on the current situation of the physical store using a user interface" refers to an interface that allows store staff to input music requests based on the current situation of the store using a smartphone or other device.

[0723] "Means for generating music in real time based on input requests and providing it to the store's sound system" refers to a system for quickly generating music based on newly input music requests and instantly playing that sound source on the store's sound equipment.

[0724] The present invention includes a system for providing optimal music in real time in a brick-and-mortar store according to the environment and situation at the time. Specific embodiments of the system will be described below.

[0725] System Overview

[0726] The system includes a series of processes for receiving a specific request from a user, setting parameters based on the request, generating music based on the set parameters, and providing the generated music in real time. The system includes a means for inputting a music request according to the situation of the physical store through a user interface, and a means for generating music based on the input request and providing the music to the store's sound system.

[0727] Hardware and software configuration

[0728] Smartphone: A device used by store staff that provides a user interface for inputting music requests.

[0729] Store sound system: A speaker system for playing the generated music.

[0730] Server: Hosts the music generation API, receives and parses music requests, and invokes the music generation model with the appropriate parameters.

[0731] Music Generation API: A web service for generating music based on user requests. It uses machine learning algorithms and pre-trained models to generate music.

[0732] Processing flow

[0733] A user (store staff member) inputs a music request through a smartphone user interface, such as "relaxing jazz music."

[0734] The server analyzes the request received from the user and sets appropriate parameters (e.g., genre, tempo, atmosphere, etc.), then executes this based on the music generation API to generate music in real time.

[0735] The generated music is sent to the store's sound system via a server and played instantly, making it possible to provide optimal music in real time according to the environment and situation of the physical store.

[0736] Specific examples

[0737] For example, if a store staff member selects "relaxing jazz music" and inputs it into their smartphone, the request is sent to the server. The server then calls the music generation API based on the request, generates music with the appropriate parameters, and returns a URL for the generated music, which is then played over the store's speaker system.

[0738] Example prompt sentence:

[0739] Relaxing Jazz Music

[0740] This makes it possible to instantly adjust the atmosphere in the store according to the situation at hand.

[0741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] A user inputs a music request using the smartphone's user interface. The input request is text data such as "relaxing jazz music." This input data is sent to the server.

[0744] Step 2:

[0745] The server analyzes the received request. Keywords such as "relax" or "jazz" are extracted from the input text data, and the parameters required for music generation (e.g., genre, tempo, atmosphere) are set based on this information. The parameters set by this data analysis are passed to the next step.

[0746] Step 3:

[0747] The server calls the music generation API based on the set parameters. The music generation API uses a machine learning algorithm to generate music in real time according to the specified parameters. The generated music data, such as a music file URL, is returned to the server.

[0748] Step 4:

[0749] The server receives the URL of the generated music and sends it to the smartphone, at which point the URL for playing the generated music is displayed on the user's smartphone.

[0750] Step 5:

[0751] The device then passes the URL to the store's sound system, which then uses the URL to stream the generated music and play it through the speakers. This process allows the music to be played in real time within the physical store.

[0752] The specific operations of each step are as follows:

[0753] Step 1: The user enters "relaxing jazz music" into the smartphone UI and presses the send button.

[0754] Step 2: The server analyzes the text data and sets the genre to "jazz" and the tempo to "relaxed."

[0755] Step 3: The server calls the music generation API, requests it to generate music using the specified parameters, and the API returns the URL of the corresponding music file.

[0756] Step 4: The server sends the URL to the smartphone and notifies the user.

[0757] Step 5: The smartphone provides the URL to the store's sound system, which plays the music through the speakers.

[0758] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0759] System Overview

[0760] The present invention is a system that generates and provides music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's input and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about artists and songs previously registered by the user.

[0761] System configuration

[0762] 1. User Interface (UI)

[0763] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[0764] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[0765] 2. Emotion Recognition Engine

[0766] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0767] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., happy, sad, angry).

[0768] How it works: For example, if the user is smiling, the emotion engine recognizes the emotion "joy."

[0769] 3. Request Analysis Module

[0770] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[0771] How it works: For example, set tempo and genre parameters based on keywords like "improve concentration" and "classical music" and the emotion "joy."

[0772] 4. Music Generation Module

[0773] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0774] Operation: Generate music at a tempo of 90 BPM with classical instrument parameters.

[0775] 5. Music Provision Module

[0776] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[0777] Operation: The generated music file is sent to the device as an HTTP response and is played in real time.

[0778] 6. Storage and playback module

[0779] Users can save their favorite music, which is then stored in a database.

[0780] The server receives requests to play stored music and provides it to the user.

[0781] How it works: By selecting the generated music and pressing the save button, it will be saved in the database and can be played back later.

[0782] 7. Artist Registration Module

[0783] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0784] The server stores the registered information in a database and uses it when generating the data.

[0785] How it works: When a user registers Artist A and requests "music in Artist A's style that helps with concentration," the generative model generates music based on that information.

[0786] Specific examples

[0787] 1. Generate classical music to improve concentration

[0788] The user requests "classical music to enhance concentration" and the emotion is recognized through the camera.

[0789] The terminal transmits the request and the emotion data to the server.

[0790] The server analyzes the music and sets additional parameters based on the tempo (90 BPM), classical genre, and recognized emotions.

[0791] A generative AI model generates music based on this.

[0792] The server sends the generated music to the device so that the user can listen to it.

[0793] 2. Generate music in the style of registered artists

[0794] A user registers artist A in advance and requests "relaxing music in artist A's style."

[0795] The terminal transmits the request and the emotion data to the server.

[0796] The server refers to the registration information and analysis data and sets appropriate parameters (e.g., artist A's characteristics, emotion "relaxed").

[0797] A generative AI model generates music based on this.

[0798] The server sends the generated music to the device so that the user can listen to it.

[0799] This system allows users to easily enjoy music that best suits their situation and emotions, improving work efficiency and changing their mood.

[0800] The processing flow will be explained below.

[0801] Step 1:

[0802] The user inputs "classical music to improve concentration" through the device's dedicated interface and presses the send button.

[0803] How it works: The user types their request into a text box on the interface.

[0804] Step 2:

[0805] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[0806] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[0807] Step 3:

[0808] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0809] How it works: Video and audio data acquired through sensors is processed in real time to extract facial expressions and vocal characteristics as emotional data.

[0810] Step 4:

[0811] The device sends the acquired emotion data to the server as an HTTP request.

[0812] What it does: Sends JSON-formatted data containing emotion data to the specified API endpoint.

[0813] Step 5:

[0814] The server analyzes the received request and extracts the request content and emotion data.

[0815] What it does: Analyzes the data in the request body using JSON deserialization and extracts "Improve concentration", "Classical music", and emotional data.

[0816] Step 6:

[0817] The server sets the necessary parameters for the generative AI model based on the request content and emotion data.

[0818] What it does: Sets parameters such as tempo, genre, and instrumentation based on the request and emotion data (e.g., tempo 90 BPM, classical genre, emotion "joy").

[0819] Step 7:

[0820] The server calls the AI ​​music generation model using the set parameters.

[0821] What it does: Converts parameters to the model's input format and sends them to the model.

[0822] Step 8:

[0823] The generative AI model generates music based on the parameters it receives.

[0824] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[0825] Step 9:

[0826] The server acquires the generated music data.

[0827] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[0828] Step 10:

[0829] The server returns the generated music data to the terminal.

[0830] Operation: Sends a music file to the device as a response.

[0831] Step 11:

[0832] The device plays the received music.

[0833] Operation: Launches the device's audio player and plays the received music data.

[0834] Step 12:

[0835] The user inputs instructions into the terminal to save the music that he or she likes.

[0836] What it does: Press the Save button to send a save request.

[0837] Step 13:

[0838] The terminal sends a save request to the server.

[0839] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[0840] Step 14:

[0841] The server stores the music data in a database.

[0842] What it does: Inserts music data and related information into a database.

[0843] Step 15:

[0844] The user selects the music they want to play from the saved list.

[0845] How it works: Select a specific song from the saved list and press the play button.

[0846] Step 16:

[0847] The terminal sends a playback request to the server.

[0848] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[0849] Step 17:

[0850] The server retrieves the music data from the stored database and returns it to the terminal.

[0851] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[0852] Step 18:

[0853] The device plays the received music.

[0854] Operation: Launches the device's audio player and plays the received music data.

[0855] Example 2

[0856] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0857] Conventional music generation systems have difficulty providing music that fully reflects a user's specific needs and emotions. Furthermore, the generated music may not be saved or played smoothly. Furthermore, they lack the functionality to generate music that reflects a user's favorite artists or songs. These issues limit users' ability to enjoy music that best suits their situation and emotions.

[0858] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0859] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for identifying an emotion based on the set parameters and facial expression analysis data, means for invoking a generative model that generates music based on the set parameters and the identified emotion, and means for providing the generated music to the user, thereby enabling the generation and provision of music that reflects the user's specific request and emotion.

[0860] The "means for accepting a specific request from a user" is a means for providing an interface for a user to input a specific request regarding music generation to the system.

[0861] The "means for setting parameters" is a means for determining various setting values ​​required for music generation based on the user's request and emotional data.

[0862] The "means for identifying emotions based on facial expression analysis data" is a means for analyzing sensor data such as the user's facial expressions and voice to identify the user's current emotional state.

[0863] The "means for invoking a generative model for generating music" refers to a means for invoking a machine learning model for generating music based on set parameters and identified emotions.

[0864] The "means for providing the generated music to the user" refers to a means for transmitting the generated music data to the user's terminal in an appropriate format so that the music data can be played back.

[0865] "Means for saving the generated music" refers to means for recording the generated music files in a database or storage so that they can be played back later.

[0866] "Means for playing stored music" refers to means for retrieving music files stored in a database or storage and making them playable on the user's device.

[0867] The "means of using sensors to acquire user emotional data" refers to a means of operating sensors such as cameras and microphones to collect data that grasps the user's current emotional state.

[0868] The "means for pre-registering favorite music artists and songs" is a means for providing an interface for a user to pre-record specific artists and songs into the system.

[0869] "Means for generating music based on registered information and emotional data" refers to means for setting music generation parameters and creating specific music, based on information about artists and songs registered in advance by the user and current emotional data.

[0870] MODE FOR CARRYING OUT THE INVENTION

[0871] The present invention is a system for generating and providing music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's requests and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about music artists and songs previously registered by the user.

[0872] System configuration

[0873] 1. User Interface (UI)

[0874] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[0875] Example: A user enters "Classical music to help me concentrate" as a request into the text box and clicks the submit button.

[0876] 2. Emotion Recognition Engine

[0877] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[0878] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., "happy," "sad," "anger").

[0879] Example: If the user is smiling, the emotion engine will recognize the emotion "joy".

[0880] 3. Request Analysis Module

[0881] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[0882] Example: Set tempo and genre parameters based on the keywords "improves concentration" and "classical music" and the emotion "joy."

[0883] 4. Music Generation Module

[0884] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0885] Example: Generate music with a tempo of 90 BPM and classical instrument parameters.

[0886] 5. Music Provision Module

[0887] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[0888] Example: The generated music file is sent to the terminal as an HTTP response and played in real time.

[0889] 6. Storage and playback module

[0890] Users can save their favorite music, which is then stored in a database.

[0891] The server receives requests to play stored music and provides it to the user.

[0892] Example: By selecting the generated music and pressing the save button, it can be saved in the database and played back later.

[0893] 7. Artist Registration Module

[0894] Users can register their favorite music artists and songs in advance, and this information is referenced when creating music.

[0895] The server stores the registered information in a database and uses it when generating the data.

[0896] Example: If a user registers artist A and requests "music in artist A's style that helps with concentration," the generative model will generate music based on that information.

[0897] This allows users to easily enjoy music that best suits their situation and emotions. For example, if a user requests "classical music that enhances concentration through feelings of joy," the system generates music based on the user's request and emotions and provides it in real time. As another example, if a user requests "relaxing music in the style of artist A," the system can generate and provide music after setting parameters based on pre-registered artist information and the user's emotional data.

[0898] Prompt Sentence Examples

[0899] "Generate classical music that enhances concentration with feelings of joy"

[0900] "Generate relaxing music in the style of artist A"

[0901] As a result, users can easily create and enjoy music suited to a variety of situations.

[0902] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0903] Step 1: Enter and submit a user request

[0904] The user opens a dedicated interface on the device and enters a specific request for the music to be generated (e.g., "classical music to improve concentration") via text boxes and drop-down menus.

[0905] The terminal converts this input data into JSON format and sends it to the server as an HTTP request.

[0906] Input: User's music request

[0907] Output: HTTP request sent to the server

[0908] Step 2: Obtaining emotion data

[0909] The device uses sensors such as a camera and microphone to capture the user's facial expressions and tone of voice and analyze emotional data.

[0910] The server receives the emotion data sent from the terminal and sends it to an emotion recognition engine, which analyzes the data and identifies the user's current emotion.

[0911] Input: User facial and voice data

[0912] Output: Emotion data analyzed by the emotion recognition engine

[0913] Step 3: Analyzing Request and Sentiment Data

[0914] The server analyzes the request and emotion data sent by the user. The request analysis module sets parameters for music generation based on specific keywords, phrases, and emotions.

[0915] Input: User's music request and emotional data

[0916] Output: Parameters for music generation (e.g. tempo 90 BPM, classical genre)

[0917] Step 4: Invoke the model to generate music

[0918] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[0919] Input: parameters for music generation

[0920] Output: Generated music file

[0921] Step 5: Submit your music

[0922] The server converts the generated music into an appropriate format (e.g., MP3) and provides it to the user's device. This module sends the generated music data as an HTTP response.

[0923] The device plays this music in real time.

[0924] Input: Generated music file

[0925] Output: Music data sent to the user's device

[0926] Step 6: Store and play your music

[0927] The user can save the generated music via a dedicated button.

[0928] The server records the saved music files in a database and provides them to the user upon receiving a playback request later.

[0929] Input: Generated music file

[0930] Output: Music files stored in the database and music files for playback

[0931] Step 7: Pre-register your artists and songs

[0932] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[0933] The server stores information about registered artists and songs in a database and references it when necessary.

[0934] Input: Artist and song information registered by the user

[0935] Output: Artist and song information stored in a database

[0936] (Application example 2)

[0937] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0938] In conventional virtual stores, music generation based on user emotions was not possible, making it impossible to provide background music suited to individual users' emotions and requests in real time, which resulted in a lack of user engagement and personalized shopping experiences.

[0939] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0940] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request and emotion data acquired in real time, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, and means for playing the music generated based on the user's emotion in real time in a virtual store. This makes it possible to generate and provide music in real time that corresponds to the user's emotion.

[0941] A "specific request" is information that indicates a user's desires or requirements regarding specific music or emotions.

[0942] "Emotional data acquired in real time" refers to emotional information acquired by analyzing a user's facial expressions and tone of voice in real time using devices such as cameras and microphones.

[0943] "Parameters" are the settings and conditions used by the music generation model when generating music.

[0944] A "generative model" is a program for generating music using machine learning algorithms.

[0945] A "virtual store" is an online platform for providing products and services in a virtual space on the Internet.

[0946] "Means for providing music" refers to a method or device for allowing a user to listen to the generated music.

[0947] "Saved music" refers to music data that has been generated and stored in a database.

[0948] "Means for pre-registering favorite music artists and songs" refers to a method or interface that allows a user to pre-register specific artists or songs.

[0949] To implement this invention, it is necessary to build a system that generates music based on specific user requests and emotional data acquired in real time, and provides the music in a virtual store. This system is composed of a user interface, an emotion recognition engine, a request analysis module, a music generation module, a music provision module, a storage and playback module, and an artist registration module.

[0950] First, the user interface (UI) uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The user can then input their music request into the device's interface, for example, requesting "relaxing music."

[0951] An emotion recognition engine then analyzes the data captured by the camera and microphone to determine the user's current emotion, using OpenCV and Dlib for facial expression recognition and the Google Cloud Speech-to-Text API for voice emotion recognition.

[0952] The request analysis module then sets appropriate parameters based on the user's request text and the analyzed sentiment data, using a natural language processing library (e.g., spaCy) and TensorFlow for sentiment data analysis.

[0953] The music generation module uses a generative AI model (e.g., Magenta) based on the set parameters to generate music in real time. Here, a machine learning model is built using Python and Keras to generate music.

[0954] The music providing module plays the generated music in real time within the virtual store. The music data is sent to the terminal as an HTTP response and is played within the virtual store to which the user is connected.

[0955] Users can then use the save and play module to save the generated music. The saved music data is stored in a database such as MySQL or PostgreSQL, and can be played back from a playlist at any time.

[0956] In addition, users can use the artist registration module to pre-register their favorite artists and songs, which will allow for more personalized music to be generated based on the registered information.

[0957] Specific examples

[0958] For example, if a user requests "I want to relax" and uses the camera to recognize their emotions, the device sends the request and emotional data to the server. The server then analyzes the data to determine specific parameters for relaxation, such as tempo and music genre, and generates music in real time using a generative AI model (such as Magenta). The generated music is played in the virtual store, allowing users to enjoy shopping while listening to it. Users can also save the music if they want to play it again later.

[0959] Prompt Sentence Examples

[0960] As an example of input from the user, the following prompt sentence can be considered:

[0961] "The user is currently relaxing. The latest request is for relaxing music. Generative parameters for playing soft classical music as background noise in a virtual store setting."

[0962] As described above, the embodiments of the present invention make it possible to provide personalized music suited to the user's emotions, thereby dramatically improving the shopping experience in a virtual store.

[0963] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0964] Step 1:

[0965] Accepting user requests

[0966] The user inputs their music request through the smartphone interface, for example, requesting "relaxing music," and the input data is sent to the device in text format.

[0967] Input: User request (e.g. "relaxing music")

[0968] Output: Request text data

[0969] Step 2:

[0970] Collecting Emotional Data

[0971] The device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Image data is acquired from the camera, and audio data is acquired from the microphone.

[0972] Input: Real-time facial and voice data of the user

[0973] Output: Image data, audio data

[0974] Step 3:

[0975] Analysis by emotion recognition engine

[0976] The server analyzes the data obtained from the camera and microphone to identify the user's emotions. It uses OpenCV and Dlib for facial recognition and Google Cloud Speech-to-Text API for voice recognition. The analysis results in the user's current emotion.

[0977] Input: Image data, audio data

[0978] Data processing: facial expression analysis, voice tone analysis

[0979] Output: Emotion data (e.g. "Relaxed")

[0980] Step 4:

[0981] Request analysis and parameter setting

[0982] The server sets appropriate parameters based on the received user request text and the analyzed emotion data. This analysis uses a natural language processing library (e.g., spaCy) and TensorFlow for emotion data analysis.

[0983] Input: Request text data, emotion data

[0984] Data processing: Natural language processing analysis, emotion data integration

[0985] Output: Parameter setting data (e.g. tempo, music genre)

[0986] Step 5:

[0987] Music Generation

[0988] The server calls a generative AI model (e.g., Magenta) based on the set parameters and generates music in real time. The generative AI model uses machine learning algorithms to generate music.

[0989] Input: Parameter setting data

[0990] Data Computation: Real-time Music Generation with Generative AI Models

[0991] Output: Generated music data

[0992] Step 6:

[0993] Music provided by

[0994] The server provides the generated music to the virtual store in real time. The music data is sent to the terminal as an HTTP response, and the music is played in the virtual store to which the user is connected.

[0995] Input: Generated music data

[0996] Data processing: Music data format conversion and transmission

[0997] Output: Music playback on the user's device

[0998] Step 7:

[0999] Music Storage

[1000] Users can save their favorite music. The server accepts the save request and stores the music data in a database.

[1001] Input: User save request, generated music data

[1002] Data processing: Saving to database

[1003] Output: Data stored in the database

[1004] Step 8:

[1005] Playing music

[1006] The server receives a request to play the stored music and provides the music data to the user, which is retrieved from the playlist and played on the user's device.

[1007] Input: User playback request, data stored in database

[1008] Data processing: Retrieving and sending data from the database

[1009] Output: Music playback on the user's device

[1010] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1011] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1012] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1013] [Third embodiment]

[1014] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1015] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1016] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1017] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1018] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1019] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1020] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1021] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1022] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1023] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1024] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1025] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1026] System Overview

[1027] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[1028] System configuration

[1029] 1. User Interface (UI)

[1030] The user enters the music request through a dedicated interface on the terminal, which includes input fields such as text boxes and drop-down menus.

[1031] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[1032] 2. Request Analysis Module

[1033] The server analyzes the request sent by the user, and this module extracts specific keywords and phrases and sets appropriate parameters based on the content.

[1034] How it works: For example, the tempo and genre (classical) parameters are set based on the keyword "improve concentration."

[1035] 3. Music Generation Module

[1036] The server invokes a music generation model based on the configured parameters, which generates music in real time using machine learning algorithms or pre-trained models.

[1037] What it does: For example, generate classical music with a tempo of 90 BPM and classical instrument parameters.

[1038] 4. Music provision module

[1039] The server provides the generated music to the user's device. This module converts the generated music data into an appropriate format and streams it in real time.

[1040] What it does: The generated music file is sent to the user as an HTTP response and is played immediately.

[1041] 5. Storage and playback module

[1042] Users can save their favorite music, which is then stored in a database.

[1043] The server receives requests to play stored music and provides it to the user.

[1044] How it works: By selecting the music you have created and pressing the save button, it will be saved in the database. The saved music can be played back later.

[1045] 6. Artist Registration Module

[1046] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1047] The server stores the registered information in a database and uses it when generating music.

[1048] How it works: After registering Artist A, when a user requests "music in Artist A's style that helps with concentration," the generative model generates music based on this information.

[1049] Specific examples

[1050] 1. Generate classical music to improve concentration

[1051] A user requests "classical music to improve concentration."

[1052] The terminal sends this request to the server.

[1053] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[1054] The music generation model generates music based on this.

[1055] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1056] 2. Generate music in the style of registered artists

[1057] The user has registered artist A in advance.

[1058] A user requests "relaxing music in the style of artist A."

[1059] The server refers to the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[1060] The music generation model generates music based on this.

[1061] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1062] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[1063] The processing flow will be explained below.

[1064] Step 1:

[1065] The user inputs "concentration-boosting classical music" through a dedicated interface on the device.

[1066] How it works: The user enters their request into a text box on the interface and presses the submit button.

[1067] Step 2:

[1068] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[1069] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[1070] Step 3:

[1071] The server analyzes the received request and extracts the requested content.

[1072] What it does: Analyzes the data in the request body using JSON deserialization and extracts keywords such as "increases concentration" and "classical music."

[1073] Step 4:

[1074] The server sets the necessary parameters for the generative AI model based on the request.

[1075] What it does: Sets parameters such as tempo, genre, and instrumentation based on keywords (e.g., tempo 90 BPM, piano and strings).

[1076] Step 5:

[1077] The server calls the AI ​​music generation model using the set parameters.

[1078] What it does: Converts parameters to the model's input format and sends them to the model.

[1079] Step 6:

[1080] The generative AI model generates music based on the parameters it receives.

[1081] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[1082] Step 7:

[1083] The server acquires the generated music data.

[1084] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[1085] Step 8:

[1086] The server returns the generated music data to the terminal.

[1087] Operation: Sends a music file to the device as a response.

[1088] Step 9:

[1089] The device plays the received music.

[1090] Operation: Launches the device's audio player and plays the received music data.

[1091] Step 10:

[1092] The user inputs instructions into the terminal to save the music that he or she likes.

[1093] What it does: Press the Save button to send a save request.

[1094] Step 11:

[1095] The terminal sends a save request to the server.

[1096] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[1097] Step 12:

[1098] The server stores the music data in a database.

[1099] What it does: Inserts music data and related information into a database.

[1100] Step 13:

[1101] The user selects the music they want to play from the saved list.

[1102] How it works: Select a specific song from the saved list and press the play button.

[1103] Step 14:

[1104] The terminal sends a playback request to the server.

[1105] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[1106] Step 15:

[1107] The server retrieves the music data from the stored database and returns it to the terminal.

[1108] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[1109] Step 16:

[1110] The device plays the received music.

[1111] Operation: Launches the device's audio player and plays the received music data.

[1112] Example 1

[1113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1114] Conventional music provision systems have difficulty generating music according to user requests and are unable to provide music that is optimal for a specific mood or situation in real time. This means that users have to spend time and effort finding music that suits their mood or situation. They also lack the ability to save their favorite music for easy access later. Furthermore, even if a user wanted music generated based on a specific artist or song, there was no way to achieve this.

[1115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1116] In this invention, the server includes means for accepting specific requests from a user, means for setting parameters based on the accepted request, means for calling a generative AI model that generates music using the set parameters, means for providing the generated music to the user, means for saving the generated music, means for playing the saved music, means for pre-registering favorite music artists and songs, and means for generating music based on the registered information. This allows users to generate music in real time based on specific requests, allowing them to obtain music that best suits their mood and situation. It also allows users to easily save their favorite music and play it later. Furthermore, music can be generated based on specific artists or songs.

[1117] "User" refers to a person who utilizes the system to input specific requests and generate music.

[1118] "Request" refers to data input by a user to the system indicating specific musical generation requirements and wishes.

[1119] "Parameters" refer to the specific conditions and characteristics required for music generation that are set based on requirements, such as tempo and genre.

[1120] "Generative AI model" refers to a system component that uses machine learning algorithms or pre-trained models to generate music based on input parameters.

[1121] "Providing" refers to the act of the server sending the generated music to the user's device and the user receiving it in real time.

[1122] "Storage" refers to storing the generated music data in a storage device such as a database so that it can be accessed later.

[1123] "Playback" refers to retrieving stored music data and making it available for the user to listen to at any time.

[1124] "Artist" refers to the composer or performer of music that a user registers as a favorite.

[1125] "Song" refers to a specific piece of music that a user registers as a favorite.

[1126] MODE FOR CARRYING OUT THE INVENTION

[1127] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[1128] System configuration

[1129] 1. User Interface (UI)

[1130] A user accesses a dedicated interface via a terminal to input music requests, which includes input fields such as text boxes and drop-down menus.

[1131] The terminal receives a request input by a user and transmits it to the server.

[1132] 2. Request Analysis Module

[1133] The server analyzes requests sent by users. This module uses natural language processing (NLP) to extract keywords and phrases from the request and sets appropriate parameters based on the content. For example, a request for "classical music to improve concentration" would be converted to parameters of 90 BPM tempo and classical genre.

[1134] 3. Music Generation Module

[1135] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time.

[1136] The generative AI model generates music data in the form of MIDI notes or audio files based on parameters such as tempo and genre.

[1137] 4. Music provision module

[1138] The server sends the generated music data to the user's device via an HTTP response.

[1139] The device immediately plays the received music data. For example, it is possible to play streaming music on a browser using the Web Audio API.

[1140] 5. Storage and playback module

[1141] Users can use the interface to save their favorite music.

[1142] The server receives the save request and saves the generated music data in a database (e.g., MySQL). The user can then play the saved music later.

[1143] 6. Artist Registration Module

[1144] Users can register their favorite artists and songs in advance. This information is referenced when creating music, and the parameters of the music are set based on the registered information.

[1145] The server stores the registered artist information in a database and uses it when generating music. For example, if a user requests "relaxing music in the style of artist A," the generative AI model will generate music based on this information.

[1146] Examples of prompt statements

[1147] "Generate classical music to help you focus."

[1148] "Generate relaxing music in the style of artist A."

[1149] Specific examples

[1150] 1. Generate classical music to improve concentration

[1151] A user requests "classical music to improve concentration."

[1152] The terminal sends this request to the server.

[1153] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[1154] The music generation model generates music based on this.

[1155] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1156] 2. Generate music in the style of registered artists

[1157] The user has registered artist A in advance.

[1158] A user requests "relaxing music in the style of artist A."

[1159] The server references the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[1160] The music generation model generates music based on this.

[1161] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1162] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[1163] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1164] Step 1:

[1165] User request input

[1166] The user accesses a dedicated interface on the device and inputs a music request, specifically, by typing "classical music to improve concentration" in the text box.

[1167] Input: Request text from the user.

[1168] Output: The data format in which the request text is sent to the server.

[1169] Step 2:

[1170] Submitting a request

[1171] The terminal sends the user's request to the server using an HTTP POST request, which includes the requested data.

[1172] Input: The request text entered by the user.

[1173] Output: An HTTP POST request containing the request text.

[1174] Step 3:

[1175] Parsing the request

[1176] The server analyzes the request received from the user. Specifically, it uses natural language processing (NLP) to extract keywords and phrases from the request. For example, from a request for "classical music that improves concentration," it recognizes the keyword "improves concentration" and sets the tempo and genre (classical) as parameters.

[1177] Input: The request data received from the user.

[1178] Data processing: Use natural language processing to extract keywords and phrases and set appropriate parameters.

[1179] Output: Parameter data (tempo 90BPM, classical genre).

[1180] Step 4:

[1181] Music Generation

[1182] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time. The generated music data is in the form of MIDI notes or audio files.

[1183] Input: Set parameters (e.g. tempo 90 BPM, classical genre).

[1184] Data Computation: Generate parameter-based music using a music generation model.

[1185] Output: The generated musical data (e.g. MIDI notes or an audio file).

[1186] Step 5:

[1187] Music provided by

[1188] The server sends the generated music data to the user's device as an HTTP response.

[1189] The device immediately plays the received music data, specifically by streaming the music on the browser using the Web Audio API.

[1190] Input: Generated music data.

[1191] Output: The music stream that plays on the user's device.

[1192] Step 6:

[1193] Music Storage

[1194] The user has an interface (e.g., a save button) to save their favorite music.

[1195] The terminal sends a save request to the server.

[1196] The server receives the save request and saves the generated music data in a database (e.g., MySQL).

[1197] Input: The save request and the music data to save.

[1198] Output: Music data stored in a database.

[1199] Step 7:

[1200] Use of registered artist information

[1201] Users can register their favorite artists and songs in advance.

[1202] The terminal inputs the artist information and transmits a registration request to the server.

[1203] The server stores the registration information in a database. When generating music, parameters are set based on the registered artist information. For example, if you request "relaxing music in the style of artist A," the generative model will generate music based on this information.

[1204] Input: Registration information (artist name, etc.).

[1205] Data calculation: Set generation parameters based on artist information.

[1206] Output: The generated music data.

[1207] (Application example 1)

[1208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1209] In brick-and-mortar stores, providing appropriate music in real time according to the situation and atmosphere of the store is important for attracting customers and improving customer satisfaction. However, conventional music playback systems only play pre-prepared playlists and lack the ability to dynamically adapt to the environment. This makes it difficult to instantly adjust the atmosphere of a brick-and-mortar store.

[1210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1211] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, means for inputting a music request based on the situation of the physical store using a user interface, and means for generating music in real time based on the input request and providing it to the store's sound system, thereby making it possible to provide optimal music in real time based on the environment and situation of the physical store.

[1212] A "means for accepting specific user requests" is an interface through which a user can input their specific musical needs or requests.

[1213] The "means for setting parameters based on the received request" is a mechanism for analyzing the request received from the user and automatically setting the parameters required for music generation.

[1214] "Means for invoking a generative model that generates music using set parameters" is a function that invokes an algorithm or generative model for automatically generating music based on set parameters.

[1215] The "means for providing the generated music to the user" is a system for converting the generated music into an appropriate format and providing it to the user in real time.

[1216] "Means for inputting music requests based on the current situation of the physical store using a user interface" refers to an interface that allows store staff to input music requests based on the current situation of the store using a smartphone or other device.

[1217] "Means for generating music in real time based on input requests and providing it to the store's sound system" refers to a system for quickly generating music based on newly input music requests and instantly playing that sound source on the store's sound equipment.

[1218] The present invention includes a system for providing optimal music in real time in a brick-and-mortar store according to the environment and situation at the time. Specific embodiments of the system will be described below.

[1219] System Overview

[1220] The system includes a series of processes for receiving a specific request from a user, setting parameters based on the request, generating music based on the set parameters, and providing the generated music in real time. The system includes a means for inputting a music request according to the situation of the physical store through a user interface, and a means for generating music based on the input request and providing the music to the store's sound system.

[1221] Hardware and software configuration

[1222] Smartphone: A device used by store staff that provides a user interface for inputting music requests.

[1223] Store sound system: A speaker system for playing the generated music.

[1224] Server: Hosts the music generation API, receives and parses music requests, and invokes the music generation model with the appropriate parameters.

[1225] Music Generation API: A web service for generating music based on user requests. It uses machine learning algorithms and pre-trained models to generate music.

[1226] Processing flow

[1227] A user (store staff member) inputs a music request through a smartphone user interface, such as "relaxing jazz music."

[1228] The server analyzes the request received from the user and sets appropriate parameters (e.g., genre, tempo, atmosphere, etc.), then executes this based on the music generation API to generate music in real time.

[1229] The generated music is sent to the store's sound system via a server and played instantly, making it possible to provide optimal music in real time according to the environment and situation of the physical store.

[1230] Specific examples

[1231] For example, if a store staff member selects "relaxing jazz music" and inputs it into their smartphone, the request is sent to the server. The server then calls the music generation API based on the request, generates music with the appropriate parameters, and returns a URL for the generated music, which is then played over the store's speaker system.

[1232] Example prompt sentence:

[1233] Relaxing Jazz Music

[1234] This makes it possible to instantly adjust the atmosphere in the store according to the situation at hand.

[1235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1236] Step 1:

[1237] A user inputs a music request using the smartphone's user interface. The input request is text data such as "relaxing jazz music." This input data is sent to the server.

[1238] Step 2:

[1239] The server analyzes the received request. Keywords such as "relax" or "jazz" are extracted from the input text data, and the parameters required for music generation (e.g., genre, tempo, atmosphere) are set based on this information. The parameters set by this data analysis are passed to the next step.

[1240] Step 3:

[1241] The server calls the music generation API based on the set parameters. The music generation API uses a machine learning algorithm to generate music in real time according to the specified parameters. The generated music data, such as a music file URL, is returned to the server.

[1242] Step 4:

[1243] The server receives the URL of the generated music and sends it to the smartphone, at which point the URL for playing the generated music is displayed on the user's smartphone.

[1244] Step 5:

[1245] The device then passes the URL to the store's sound system, which then uses the URL to stream the generated music and play it through the speakers. This process allows the music to be played in real time within the physical store.

[1246] The specific operations of each step are as follows:

[1247] Step 1: The user enters "relaxing jazz music" into the smartphone UI and presses the send button.

[1248] Step 2: The server analyzes the text data and sets the genre to "jazz" and the tempo to "relaxed."

[1249] Step 3: The server calls the music generation API, requests it to generate music using the specified parameters, and the API returns the URL of the corresponding music file.

[1250] Step 4: The server sends the URL to the smartphone and notifies the user.

[1251] Step 5: The smartphone provides the URL to the store's sound system, which plays the music through the speakers.

[1252] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1253] System Overview

[1254] The present invention is a system that generates and provides music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's input and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about artists and songs previously registered by the user.

[1255] System configuration

[1256] 1. User Interface (UI)

[1257] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[1258] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[1259] 2. Emotion Recognition Engine

[1260] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1261] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., happy, sad, angry).

[1262] How it works: For example, if the user is smiling, the emotion engine recognizes the emotion "joy."

[1263] 3. Request Analysis Module

[1264] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[1265] How it works: For example, set tempo and genre parameters based on keywords like "improve concentration" and "classical music" and the emotion "joy."

[1266] 4. Music Generation Module

[1267] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1268] Operation: Generate music at a tempo of 90 BPM with classical instrument parameters.

[1269] 5. Music Provision Module

[1270] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[1271] Operation: The generated music file is sent to the device as an HTTP response and is played in real time.

[1272] 6. Storage and playback module

[1273] Users can save their favorite music, which is then stored in a database.

[1274] The server receives requests to play stored music and provides it to the user.

[1275] How it works: By selecting the generated music and pressing the save button, it will be saved in the database and can be played back later.

[1276] 7. Artist Registration Module

[1277] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1278] The server stores the registered information in a database and uses it when generating the data.

[1279] How it works: When a user registers Artist A and requests "music in Artist A's style that helps with concentration," the generative model generates music based on that information.

[1280] Specific examples

[1281] 1. Generate classical music to improve concentration

[1282] The user requests "classical music to enhance concentration" and the emotion is recognized through the camera.

[1283] The terminal transmits the request and the emotion data to the server.

[1284] The server analyzes the music and sets additional parameters based on the tempo (90 BPM), classical genre, and recognized emotions.

[1285] A generative AI model generates music based on this.

[1286] The server sends the generated music to the device so that the user can listen to it.

[1287] 2. Generate music in the style of registered artists

[1288] A user registers artist A in advance and requests "relaxing music in artist A's style."

[1289] The terminal transmits the request and the emotion data to the server.

[1290] The server refers to the registration information and analysis data and sets appropriate parameters (e.g., artist A's characteristics, emotion "relaxed").

[1291] A generative AI model generates music based on this.

[1292] The server sends the generated music to the device so that the user can listen to it.

[1293] This system allows users to easily enjoy music that best suits their situation and emotions, improving work efficiency and changing their mood.

[1294] The processing flow will be explained below.

[1295] Step 1:

[1296] The user inputs "classical music to improve concentration" through the device's dedicated interface and presses the send button.

[1297] How it works: The user types their request into a text box on the interface.

[1298] Step 2:

[1299] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[1300] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[1301] Step 3:

[1302] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1303] How it works: Video and audio data acquired through sensors is processed in real time to extract facial expressions and vocal characteristics as emotional data.

[1304] Step 4:

[1305] The device sends the acquired emotion data to the server as an HTTP request.

[1306] What it does: Sends JSON-formatted data containing emotion data to the specified API endpoint.

[1307] Step 5:

[1308] The server analyzes the received request and extracts the request content and emotion data.

[1309] What it does: Analyzes the data in the request body using JSON deserialization and extracts "Improve concentration", "Classical music", and emotional data.

[1310] Step 6:

[1311] The server sets the necessary parameters for the generative AI model based on the request content and emotion data.

[1312] What it does: Sets parameters such as tempo, genre, and instrumentation based on the request and emotion data (e.g., tempo 90 BPM, classical genre, emotion "joy").

[1313] Step 7:

[1314] The server calls the AI ​​music generation model using the set parameters.

[1315] What it does: Converts parameters to the model's input format and sends them to the model.

[1316] Step 8:

[1317] The generative AI model generates music based on the parameters it receives.

[1318] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[1319] Step 9:

[1320] The server acquires the generated music data.

[1321] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[1322] Step 10:

[1323] The server returns the generated music data to the terminal.

[1324] Operation: Sends a music file to the device as a response.

[1325] Step 11:

[1326] The device plays the received music.

[1327] Operation: Launches the device's audio player and plays the received music data.

[1328] Step 12:

[1329] The user inputs instructions into the terminal to save the music that he or she likes.

[1330] What it does: Press the Save button to send a save request.

[1331] Step 13:

[1332] The terminal sends a save request to the server.

[1333] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[1334] Step 14:

[1335] The server stores the music data in a database.

[1336] What it does: Inserts music data and related information into a database.

[1337] Step 15:

[1338] The user selects the music they want to play from the saved list.

[1339] How it works: Select a specific song from the saved list and press the play button.

[1340] Step 16:

[1341] The terminal sends a playback request to the server.

[1342] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[1343] Step 17:

[1344] The server retrieves the music data from the stored database and returns it to the terminal.

[1345] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[1346] Step 18:

[1347] The device plays the received music.

[1348] Operation: Launches the device's audio player and plays the received music data.

[1349] Example 2

[1350] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1351] Conventional music generation systems have difficulty providing music that fully reflects a user's specific needs and emotions. Furthermore, the generated music may not be saved or played smoothly. Furthermore, they lack the functionality to generate music that reflects a user's favorite artists or songs. These issues limit users' ability to enjoy music that best suits their situation and emotions.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1353] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for identifying an emotion based on the set parameters and facial expression analysis data, means for invoking a generative model that generates music based on the set parameters and the identified emotion, and means for providing the generated music to the user, thereby enabling the generation and provision of music that reflects the user's specific request and emotion.

[1354] The "means for accepting a specific request from a user" is a means for providing an interface for a user to input a specific request regarding music generation to the system.

[1355] The "means for setting parameters" is a means for determining various setting values ​​required for music generation based on the user's request and emotional data.

[1356] The "means for identifying emotions based on facial expression analysis data" is a means for analyzing sensor data such as the user's facial expressions and voice to identify the user's current emotional state.

[1357] The "means for invoking a generative model for generating music" refers to a means for invoking a machine learning model for generating music based on set parameters and identified emotions.

[1358] The "means for providing the generated music to the user" refers to a means for transmitting the generated music data to the user's terminal in an appropriate format so that the music data can be played back.

[1359] "Means for saving the generated music" refers to means for recording the generated music files in a database or storage so that they can be played back later.

[1360] "Means for playing stored music" refers to means for retrieving music files stored in a database or storage and making them playable on the user's device.

[1361] The "means of using sensors to acquire user emotional data" refers to a means of operating sensors such as cameras and microphones to collect data that grasps the user's current emotional state.

[1362] The "means for pre-registering favorite music artists and songs" is a means for providing an interface for a user to pre-record specific artists and songs into the system.

[1363] "Means for generating music based on registered information and emotional data" refers to means for setting music generation parameters and creating specific music, based on information about artists and songs registered in advance by the user and current emotional data.

[1364] MODE FOR CARRYING OUT THE INVENTION

[1365] The present invention is a system for generating and providing music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's requests and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about music artists and songs previously registered by the user.

[1366] System configuration

[1367] 1. User Interface (UI)

[1368] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[1369] Example: A user enters "Classical music to help me concentrate" as a request into the text box and clicks the submit button.

[1370] 2. Emotion Recognition Engine

[1371] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1372] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., "happy," "sad," "anger").

[1373] Example: If the user is smiling, the emotion engine will recognize the emotion "joy".

[1374] 3. Request Analysis Module

[1375] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[1376] Example: Set tempo and genre parameters based on the keywords "improves concentration" and "classical music" and the emotion "joy."

[1377] 4. Music Generation Module

[1378] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1379] Example: Generate music with a tempo of 90 BPM and classical instrument parameters.

[1380] 5. Music Provision Module

[1381] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[1382] Example: The generated music file is sent to the terminal as an HTTP response and played in real time.

[1383] 6. Storage and playback module

[1384] Users can save their favorite music, which is then stored in a database.

[1385] The server receives requests to play stored music and provides it to the user.

[1386] Example: By selecting the generated music and pressing the save button, it can be saved in the database and played back later.

[1387] 7. Artist Registration Module

[1388] Users can register their favorite music artists and songs in advance, and this information is referenced when creating music.

[1389] The server stores the registered information in a database and uses it when generating the data.

[1390] Example: If a user registers artist A and requests "music in artist A's style that helps with concentration," the generative model will generate music based on that information.

[1391] This allows users to easily enjoy music that best suits their situation and emotions. For example, if a user requests "classical music that enhances concentration through feelings of joy," the system generates music based on the user's request and emotions and provides it in real time. As another example, if a user requests "relaxing music in the style of artist A," the system can generate and provide music after setting parameters based on pre-registered artist information and the user's emotional data.

[1392] Prompt Sentence Examples

[1393] "Generate classical music that enhances concentration with feelings of joy"

[1394] "Generate relaxing music in the style of artist A"

[1395] As a result, users can easily create and enjoy music suited to a variety of situations.

[1396] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1397] Step 1: Enter and submit a user request

[1398] The user opens a dedicated interface on the device and enters a specific request for the music to be generated (e.g., "classical music to improve concentration") via text boxes and drop-down menus.

[1399] The terminal converts this input data into JSON format and sends it to the server as an HTTP request.

[1400] Input: User's music request

[1401] Output: HTTP request sent to the server

[1402] Step 2: Obtaining emotion data

[1403] The device uses sensors such as a camera and microphone to capture the user's facial expressions and tone of voice and analyze emotional data.

[1404] The server receives the emotion data sent from the terminal and sends it to an emotion recognition engine, which analyzes the data and identifies the user's current emotion.

[1405] Input: User facial and voice data

[1406] Output: Emotion data analyzed by the emotion recognition engine

[1407] Step 3: Analyzing Request and Sentiment Data

[1408] The server analyzes the request and emotion data sent by the user. The request analysis module sets parameters for music generation based on specific keywords, phrases, and emotions.

[1409] Input: User's music request and emotional data

[1410] Output: Parameters for music generation (e.g. tempo 90 BPM, classical genre)

[1411] Step 4: Invoke the model to generate music

[1412] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1413] Input: parameters for music generation

[1414] Output: Generated music file

[1415] Step 5: Submit your music

[1416] The server converts the generated music into an appropriate format (e.g., MP3) and provides it to the user's device. This module sends the generated music data as an HTTP response.

[1417] The device plays this music in real time.

[1418] Input: Generated music file

[1419] Output: Music data sent to the user's device

[1420] Step 6: Store and play your music

[1421] The user can save the generated music via a dedicated button.

[1422] The server records the saved music files in a database and provides them to the user upon receiving a playback request later.

[1423] Input: Generated music file

[1424] Output: Music files stored in the database and music files for playback

[1425] Step 7: Pre-register your artists and songs

[1426] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1427] The server stores information about registered artists and songs in a database and references it when necessary.

[1428] Input: Artist and song information registered by the user

[1429] Output: Artist and song information stored in a database

[1430] (Application example 2)

[1431] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1432] In conventional virtual stores, music generation based on user emotions was not possible, making it impossible to provide background music suited to individual users' emotions and requests in real time, which resulted in a lack of user engagement and personalized shopping experiences.

[1433] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1434] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request and emotion data acquired in real time, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, and means for playing the music generated based on the user's emotion in real time in a virtual store. This makes it possible to generate and provide music in real time that corresponds to the user's emotion.

[1435] A "specific request" is information that indicates a user's desires or requirements regarding specific music or emotions.

[1436] "Emotional data acquired in real time" refers to emotional information acquired by analyzing a user's facial expressions and tone of voice in real time using devices such as cameras and microphones.

[1437] "Parameters" are the settings and conditions used by the music generation model when generating music.

[1438] A "generative model" is a program for generating music using machine learning algorithms.

[1439] A "virtual store" is an online platform for providing products and services in a virtual space on the Internet.

[1440] "Means for providing music" refers to a method or device for allowing a user to listen to the generated music.

[1441] "Saved music" refers to music data that has been generated and stored in a database.

[1442] "Means for pre-registering favorite music artists and songs" refers to a method or interface that allows a user to pre-register specific artists or songs.

[1443] To implement this invention, it is necessary to build a system that generates music based on specific user requests and emotional data acquired in real time, and provides the music in a virtual store. This system is composed of a user interface, an emotion recognition engine, a request analysis module, a music generation module, a music provision module, a storage and playback module, and an artist registration module.

[1444] First, the user interface (UI) uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The user can then input their music request into the device's interface, for example, requesting "relaxing music."

[1445] An emotion recognition engine then analyzes the data captured by the camera and microphone to determine the user's current emotion, using OpenCV and Dlib for facial expression recognition and the Google Cloud Speech-to-Text API for voice emotion recognition.

[1446] The request analysis module then sets appropriate parameters based on the user's request text and the analyzed sentiment data, using a natural language processing library (e.g., spaCy) and TensorFlow for sentiment data analysis.

[1447] The music generation module uses a generative AI model (e.g., Magenta) based on the set parameters to generate music in real time. Here, a machine learning model is built using Python and Keras to generate music.

[1448] The music providing module plays the generated music in real time within the virtual store. The music data is sent to the terminal as an HTTP response and is played within the virtual store to which the user is connected.

[1449] Users can then use the save and play module to save the generated music. The saved music data is stored in a database such as MySQL or PostgreSQL, and can be played back from a playlist at any time.

[1450] In addition, users can use the artist registration module to pre-register their favorite artists and songs, which will allow for more personalized music to be generated based on the registered information.

[1451] Specific examples

[1452] For example, if a user requests "I want to relax" and uses the camera to recognize their emotions, the device sends the request and emotional data to the server. The server then analyzes the data to determine specific parameters for relaxation, such as tempo and music genre, and generates music in real time using a generative AI model (such as Magenta). The generated music is played in the virtual store, allowing users to enjoy shopping while listening to it. Users can also save the music if they want to play it again later.

[1453] Prompt Sentence Examples

[1454] As an example of input from the user, the following prompt sentence can be considered:

[1455] "The user is currently relaxing. The latest request is for relaxing music. Generative parameters for playing soft classical music as background noise in a virtual store setting."

[1456] As described above, the embodiments of the present invention make it possible to provide personalized music suited to the user's emotions, thereby dramatically improving the shopping experience in a virtual store.

[1457] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1458] Step 1:

[1459] Accepting user requests

[1460] The user inputs their music request through the smartphone interface, for example, requesting "relaxing music," and the input data is sent to the device in text format.

[1461] Input: User request (e.g. "relaxing music")

[1462] Output: Request text data

[1463] Step 2:

[1464] Collecting Emotional Data

[1465] The device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Image data is acquired from the camera, and audio data is acquired from the microphone.

[1466] Input: Real-time facial and voice data of the user

[1467] Output: Image data, audio data

[1468] Step 3:

[1469] Analysis by emotion recognition engine

[1470] The server analyzes the data obtained from the camera and microphone to identify the user's emotions. It uses OpenCV and Dlib for facial recognition and Google Cloud Speech-to-Text API for voice recognition. The analysis results in the user's current emotion.

[1471] Input: Image data, audio data

[1472] Data processing: facial expression analysis, voice tone analysis

[1473] Output: Emotion data (e.g. "Relaxed")

[1474] Step 4:

[1475] Request analysis and parameter setting

[1476] The server sets appropriate parameters based on the received user request text and the analyzed emotion data. This analysis uses a natural language processing library (e.g., spaCy) and TensorFlow for emotion data analysis.

[1477] Input: Request text data, emotion data

[1478] Data processing: Natural language processing analysis, emotion data integration

[1479] Output: Parameter setting data (e.g. tempo, music genre)

[1480] Step 5:

[1481] Music Generation

[1482] The server calls a generative AI model (e.g., Magenta) based on the set parameters and generates music in real time. The generative AI model uses machine learning algorithms to generate music.

[1483] Input: Parameter setting data

[1484] Data Computation: Real-time Music Generation with Generative AI Models

[1485] Output: Generated music data

[1486] Step 6:

[1487] Music provided by

[1488] The server provides the generated music to the virtual store in real time. The music data is sent to the terminal as an HTTP response, and the music is played in the virtual store to which the user is connected.

[1489] Input: Generated music data

[1490] Data processing: Music data format conversion and transmission

[1491] Output: Music playback on the user's device

[1492] Step 7:

[1493] Music Storage

[1494] Users can save their favorite music. The server accepts the save request and stores the music data in a database.

[1495] Input: User save request, generated music data

[1496] Data processing: Saving to database

[1497] Output: Data stored in the database

[1498] Step 8:

[1499] Playing music

[1500] The server receives a request to play the stored music and provides the music data to the user, which is retrieved from the playlist and played on the user's device.

[1501] Input: User playback request, data stored in database

[1502] Data processing: Retrieving and sending data from the database

[1503] Output: Music playback on the user's device

[1504] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1505] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1506] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1507] [Fourth embodiment]

[1508] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1509] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1510] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1511] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1512] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1513] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1514] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1515] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1516] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1517] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1518] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1519] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1520] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1521] System Overview

[1522] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[1523] System configuration

[1524] 1. User Interface (UI)

[1525] The user enters the music request through a dedicated interface on the terminal, which includes input fields such as text boxes and drop-down menus.

[1526] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[1527] 2. Request Analysis Module

[1528] The server analyzes the request sent by the user, and this module extracts specific keywords and phrases and sets appropriate parameters based on the content.

[1529] How it works: For example, the tempo and genre (classical) parameters are set based on the keyword "improve concentration."

[1530] 3. Music Generation Module

[1531] The server invokes a music generation model based on the configured parameters, which generates music in real time using machine learning algorithms or pre-trained models.

[1532] What it does: For example, generate classical music with a tempo of 90 BPM and classical instrument parameters.

[1533] 4. Music provision module

[1534] The server provides the generated music to the user's device. This module converts the generated music data into an appropriate format and streams it in real time.

[1535] What it does: The generated music file is sent to the user as an HTTP response and is played immediately.

[1536] 5. Storage and playback module

[1537] Users can save their favorite music, which is then stored in a database.

[1538] The server receives requests to play stored music and provides it to the user.

[1539] How it works: By selecting the music you have created and pressing the save button, it will be saved in the database. The saved music can be played back later.

[1540] 6. Artist Registration Module

[1541] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1542] The server stores the registered information in a database and uses it when generating music.

[1543] How it works: After registering Artist A, when a user requests "music in Artist A's style that helps with concentration," the generative model generates music based on this information.

[1544] Specific examples

[1545] 1. Generate classical music to improve concentration

[1546] A user requests "classical music to improve concentration."

[1547] The terminal sends this request to the server.

[1548] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[1549] The music generation model generates music based on this.

[1550] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1551] 2. Generate music in the style of registered artists

[1552] The user has registered artist A in advance.

[1553] A user requests "relaxing music in the style of artist A."

[1554] The server refers to the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[1555] The music generation model generates music based on this.

[1556] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1557] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[1558] The processing flow will be explained below.

[1559] Step 1:

[1560] The user inputs "concentration-boosting classical music" through a dedicated interface on the device.

[1561] How it works: The user enters their request into a text box on the interface and presses the submit button.

[1562] Step 2:

[1563] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[1564] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[1565] Step 3:

[1566] The server analyzes the received request and extracts the requested content.

[1567] What it does: Analyzes the data in the request body using JSON deserialization and extracts keywords such as "increases concentration" and "classical music."

[1568] Step 4:

[1569] The server sets the necessary parameters for the generative AI model based on the request.

[1570] What it does: Sets parameters such as tempo, genre, and instrumentation based on keywords (e.g., tempo 90 BPM, piano and strings).

[1571] Step 5:

[1572] The server calls the AI ​​music generation model using the set parameters.

[1573] What it does: Converts parameters to the model's input format and sends them to the model.

[1574] Step 6:

[1575] The generative AI model generates music based on the parameters it receives.

[1576] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[1577] Step 7:

[1578] The server acquires the generated music data.

[1579] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[1580] Step 8:

[1581] The server returns the generated music data to the terminal.

[1582] Operation: Sends a music file to the device as a response.

[1583] Step 9:

[1584] The device plays the received music.

[1585] Operation: Launches the device's audio player and plays the received music data.

[1586] Step 10:

[1587] The user inputs instructions into the terminal to save the music that he or she likes.

[1588] What it does: Press the Save button to send a save request.

[1589] Step 11:

[1590] The terminal sends a save request to the server.

[1591] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[1592] Step 12:

[1593] The server stores the music data in a database.

[1594] What it does: Inserts music data and related information into a database.

[1595] Step 13:

[1596] The user selects the music they want to play from the saved list.

[1597] How it works: Select a specific song from the saved list and press the play button.

[1598] Step 14:

[1599] The terminal sends a playback request to the server.

[1600] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[1601] Step 15:

[1602] The server retrieves the music data from the stored database and returns it to the terminal.

[1603] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[1604] Step 16:

[1605] The device plays the received music.

[1606] Operation: Launches the device's audio player and plays the received music data.

[1607] Example 1

[1608] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1609] Conventional music provision systems have difficulty generating music according to user requests and are unable to provide music that is optimal for a specific mood or situation in real time. This means that users have to spend time and effort finding music that suits their mood or situation. They also lack the ability to save their favorite music for easy access later. Furthermore, even if a user wanted music generated based on a specific artist or song, there was no way to achieve this.

[1610] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1611] In this invention, the server includes means for accepting specific requests from a user, means for setting parameters based on the accepted request, means for calling a generative AI model that generates music using the set parameters, means for providing the generated music to the user, means for saving the generated music, means for playing the saved music, means for pre-registering favorite music artists and songs, and means for generating music based on the registered information. This allows users to generate music in real time based on specific requests, allowing them to obtain music that best suits their mood and situation. It also allows users to easily save their favorite music and play it later. Furthermore, music can be generated based on specific artists or songs.

[1612] "User" refers to a person who utilizes the system to input specific requests and generate music.

[1613] "Request" refers to data input by a user to the system indicating specific musical generation requirements and wishes.

[1614] "Parameters" refer to the specific conditions and characteristics required for music generation that are set based on requirements, such as tempo and genre.

[1615] "Generative AI model" refers to a system component that uses machine learning algorithms or pre-trained models to generate music based on input parameters.

[1616] "Providing" refers to the act of the server sending the generated music to the user's device and the user receiving it in real time.

[1617] "Storage" refers to storing the generated music data in a storage device such as a database so that it can be accessed later.

[1618] "Playback" refers to retrieving stored music data and making it available for the user to listen to at any time.

[1619] "Artist" refers to the composer or performer of music that a user registers as a favorite.

[1620] "Song" refers to a specific piece of music that a user registers as a favorite.

[1621] MODE FOR CARRYING OUT THE INVENTION

[1622] The present invention is a system for generating and providing music based on a user's specific requests. This system involves a series of processes for accepting requests from a user, generating music based on those requests, and providing the music to the user. The system also has the ability to save the generated music and play it later. Furthermore, the system can generate music based on information about artists and songs that the user has registered in advance.

[1623] System configuration

[1624] 1. User Interface (UI)

[1625] A user accesses a dedicated interface via a terminal to input music requests, which includes input fields such as text boxes and drop-down menus.

[1626] The terminal receives a request input by a user and transmits it to the server.

[1627] 2. Request Analysis Module

[1628] The server analyzes requests sent by users. This module uses natural language processing (NLP) to extract keywords and phrases from the request and sets appropriate parameters based on the content. For example, a request for "classical music to improve concentration" would be converted to parameters of 90 BPM tempo and classical genre.

[1629] 3. Music Generation Module

[1630] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time.

[1631] The generative AI model generates music data in the form of MIDI notes or audio files based on parameters such as tempo and genre.

[1632] 4. Music provision module

[1633] The server sends the generated music data to the user's device via an HTTP response.

[1634] The device immediately plays the received music data. For example, it is possible to play streaming music on a browser using the Web Audio API.

[1635] 5. Storage and playback module

[1636] Users can use the interface to save their favorite music.

[1637] The server receives the save request and saves the generated music data in a database (e.g., MySQL). The user can then play the saved music later.

[1638] 6. Artist Registration Module

[1639] Users can register their favorite artists and songs in advance. This information is referenced when creating music, and the parameters of the music are set based on the registered information.

[1640] The server stores the registered artist information in a database and uses it when generating music. For example, if a user requests "relaxing music in the style of artist A," the generative AI model will generate music based on this information.

[1641] Examples of prompt statements

[1642] "Generate classical music to help you focus."

[1643] "Generate relaxing music in the style of artist A."

[1644] Specific examples

[1645] 1. Generate classical music to improve concentration

[1646] A user requests "classical music to improve concentration."

[1647] The terminal sends this request to the server.

[1648] The server parses the request and sets the appropriate parameters (e.g., tempo 90 BPM, classical genre).

[1649] The music generation model generates music based on this.

[1650] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1651] 2. Generate music in the style of registered artists

[1652] The user has registered artist A in advance.

[1653] A user requests "relaxing music in the style of artist A."

[1654] The server references the registration information and sets appropriate parameters (e.g., the characteristic rhythm and melody of artist A).

[1655] The music generation model generates music based on this.

[1656] The server sends the generated music to the terminal so that the user can listen to it immediately.

[1657] As described above, the system of the present invention can generate and provide music based on the user's specific requests, allowing the user to easily obtain music that best suits the situation and mood at hand, which can be useful for improving work efficiency and refreshing the mood.

[1658] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1659] Step 1:

[1660] User request input

[1661] The user accesses a dedicated interface on the device and inputs a music request, specifically, by typing "classical music to improve concentration" in the text box.

[1662] Input: Request text from the user.

[1663] Output: The data format in which the request text is sent to the server.

[1664] Step 2:

[1665] Submitting a request

[1666] The terminal sends the user's request to the server using an HTTP POST request, which includes the requested data.

[1667] Input: The request text entered by the user.

[1668] Output: An HTTP POST request containing the request text.

[1669] Step 3:

[1670] Parsing the request

[1671] The server analyzes the request received from the user. Specifically, it uses natural language processing (NLP) to extract keywords and phrases from the request. For example, from a request for "classical music that improves concentration," it recognizes the keyword "improves concentration" and sets the tempo and genre (classical) as parameters.

[1672] Input: The request data received from the user.

[1673] Data processing: Use natural language processing to extract keywords and phrases and set appropriate parameters.

[1674] Output: Parameter data (tempo 90BPM, classical genre).

[1675] Step 4:

[1676] Music Generation

[1677] The server calls a music generation model (e.g., OpenAI's MuseNet) based on the set parameters. The generative AI model uses machine learning algorithms to generate music in real time. The generated music data is in the form of MIDI notes or audio files.

[1678] Input: Set parameters (e.g. tempo 90 BPM, classical genre).

[1679] Data Computation: Generate parameter-based music using a music generation model.

[1680] Output: The generated musical data (e.g. MIDI notes or an audio file).

[1681] Step 5:

[1682] Music provided by

[1683] The server sends the generated music data to the user's device as an HTTP response.

[1684] The device immediately plays the received music data, specifically by streaming the music on the browser using the Web Audio API.

[1685] Input: Generated music data.

[1686] Output: The music stream that plays on the user's device.

[1687] Step 6:

[1688] Music Storage

[1689] The user has an interface (e.g., a save button) to save their favorite music.

[1690] The terminal sends a save request to the server.

[1691] The server receives the save request and saves the generated music data in a database (e.g., MySQL).

[1692] Input: The save request and the music data to save.

[1693] Output: Music data stored in a database.

[1694] Step 7:

[1695] Use of registered artist information

[1696] Users can register their favorite artists and songs in advance.

[1697] The terminal inputs the artist information and transmits a registration request to the server.

[1698] The server stores the registration information in a database. When generating music, parameters are set based on the registered artist information. For example, if you request "relaxing music in the style of artist A," the generative model will generate music based on this information.

[1699] Input: Registration information (artist name, etc.).

[1700] Data calculation: Set generation parameters based on artist information.

[1701] Output: The generated music data.

[1702] (Application example 1)

[1703] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1704] In brick-and-mortar stores, providing appropriate music in real time according to the situation and atmosphere of the store is important for attracting customers and improving customer satisfaction. However, conventional music playback systems only play pre-prepared playlists and lack the ability to dynamically adapt to the environment. This makes it difficult to instantly adjust the atmosphere of a brick-and-mortar store.

[1705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1706] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, means for inputting a music request based on the situation of the physical store using a user interface, and means for generating music in real time based on the input request and providing it to the store's sound system, thereby making it possible to provide optimal music in real time based on the environment and situation of the physical store.

[1707] A "means for accepting specific user requests" is an interface through which a user can input their specific musical needs or requests.

[1708] The "means for setting parameters based on the received request" is a mechanism for analyzing the request received from the user and automatically setting the parameters required for music generation.

[1709] "Means for invoking a generative model that generates music using set parameters" is a function that invokes an algorithm or generative model for automatically generating music based on set parameters.

[1710] The "means for providing the generated music to the user" is a system for converting the generated music into an appropriate format and providing it to the user in real time.

[1711] "Means for inputting music requests based on the current situation of the physical store using a user interface" refers to an interface that allows store staff to input music requests based on the current situation of the store using a smartphone or other device.

[1712] "Means for generating music in real time based on input requests and providing it to the store's sound system" refers to a system for quickly generating music based on newly input music requests and instantly playing that sound source on the store's sound equipment.

[1713] The present invention includes a system for providing optimal music in real time in a brick-and-mortar store according to the environment and situation at the time. Specific embodiments of the system will be described below.

[1714] System Overview

[1715] The system includes a series of processes for receiving a specific request from a user, setting parameters based on the request, generating music based on the set parameters, and providing the generated music in real time. The system includes a means for inputting a music request according to the situation of the physical store through a user interface, and a means for generating music based on the input request and providing the music to the store's sound system.

[1716] Hardware and software configuration

[1717] Smartphone: A device used by store staff that provides a user interface for inputting music requests.

[1718] Store sound system: A speaker system for playing the generated music.

[1719] Server: Hosts the music generation API, receives and parses music requests, and invokes the music generation model with the appropriate parameters.

[1720] Music Generation API: A web service for generating music based on user requests. It uses machine learning algorithms and pre-trained models to generate music.

[1721] Processing flow

[1722] A user (store staff member) inputs a music request through a smartphone user interface, such as "relaxing jazz music."

[1723] The server analyzes the request received from the user and sets appropriate parameters (e.g., genre, tempo, atmosphere, etc.), then executes this based on the music generation API to generate music in real time.

[1724] The generated music is sent to the store's sound system via a server and played instantly, making it possible to provide optimal music in real time according to the environment and situation of the physical store.

[1725] Specific examples

[1726] For example, if a store staff member selects "relaxing jazz music" and inputs it into their smartphone, the request is sent to the server. The server then calls the music generation API based on the request, generates music with the appropriate parameters, and returns a URL for the generated music, which is then played over the store's speaker system.

[1727] Example prompt sentence:

[1728] Relaxing Jazz Music

[1729] This makes it possible to instantly adjust the atmosphere in the store according to the situation at hand.

[1730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1731] Step 1:

[1732] A user inputs a music request using the smartphone's user interface. The input request is text data such as "relaxing jazz music." This input data is sent to the server.

[1733] Step 2:

[1734] The server analyzes the received request. Keywords such as "relax" or "jazz" are extracted from the input text data, and the parameters required for music generation (e.g., genre, tempo, atmosphere) are set based on this information. The parameters set by this data analysis are passed to the next step.

[1735] Step 3:

[1736] The server calls the music generation API based on the set parameters. The music generation API uses a machine learning algorithm to generate music in real time according to the specified parameters. The generated music data, such as a music file URL, is returned to the server.

[1737] Step 4:

[1738] The server receives the URL of the generated music and sends it to the smartphone, at which point the URL for playing the generated music is displayed on the user's smartphone.

[1739] Step 5:

[1740] The device then passes the URL to the store's sound system, which then uses the URL to stream the generated music and play it through the speakers. This process allows the music to be played in real time within the physical store.

[1741] The specific operations of each step are as follows:

[1742] Step 1: The user enters "relaxing jazz music" into the smartphone UI and presses the send button.

[1743] Step 2: The server analyzes the text data and sets the genre to "jazz" and the tempo to "relaxed."

[1744] Step 3: The server calls the music generation API, requests it to generate music using the specified parameters, and the API returns the URL of the corresponding music file.

[1745] Step 4: The server sends the URL to the smartphone and notifies the user.

[1746] Step 5: The smartphone provides the URL to the store's sound system, which plays the music through the speakers.

[1747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1748] System Overview

[1749] The present invention is a system that generates and provides music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's input and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about artists and songs previously registered by the user.

[1750] System configuration

[1751] 1. User Interface (UI)

[1752] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[1753] How it works: A user enters a request, such as "classical music to help me concentrate," and the request is sent to a server.

[1754] 2. Emotion Recognition Engine

[1755] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1756] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., happy, sad, angry).

[1757] How it works: For example, if the user is smiling, the emotion engine recognizes the emotion "joy."

[1758] 3. Request Analysis Module

[1759] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[1760] How it works: For example, set tempo and genre parameters based on keywords like "improve concentration" and "classical music" and the emotion "joy."

[1761] 4. Music Generation Module

[1762] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1763] Operation: Generate music at a tempo of 90 BPM with classical instrument parameters.

[1764] 5. Music Provision Module

[1765] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[1766] Operation: The generated music file is sent to the device as an HTTP response and is played in real time.

[1767] 6. Storage and playback module

[1768] Users can save their favorite music, which is then stored in a database.

[1769] The server receives requests to play stored music and provides it to the user.

[1770] How it works: By selecting the generated music and pressing the save button, it will be saved in the database and can be played back later.

[1771] 7. Artist Registration Module

[1772] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1773] The server stores the registered information in a database and uses it when generating the data.

[1774] How it works: When a user registers Artist A and requests "music in Artist A's style that helps with concentration," the generative model generates music based on that information.

[1775] Specific examples

[1776] 1. Generate classical music to improve concentration

[1777] The user requests "classical music to enhance concentration" and the emotion is recognized through the camera.

[1778] The terminal transmits the request and the emotion data to the server.

[1779] The server analyzes the music and sets additional parameters based on the tempo (90 BPM), classical genre, and recognized emotions.

[1780] A generative AI model generates music based on this.

[1781] The server sends the generated music to the device so that the user can listen to it.

[1782] 2. Generate music in the style of registered artists

[1783] A user registers artist A in advance and requests "relaxing music in artist A's style."

[1784] The terminal transmits the request and the emotion data to the server.

[1785] The server refers to the registration information and analysis data and sets appropriate parameters (e.g., artist A's characteristics, emotion "relaxed").

[1786] A generative AI model generates music based on this.

[1787] The server sends the generated music to the device so that the user can listen to it.

[1788] This system allows users to easily enjoy music that best suits their situation and emotions, improving work efficiency and changing their mood.

[1789] The processing flow will be explained below.

[1790] Step 1:

[1791] The user inputs "classical music to improve concentration" through the device's dedicated interface and presses the send button.

[1792] How it works: The user types their request into a text box on the interface.

[1793] Step 2:

[1794] The terminal obtains the user's request and sends the request to the server as an HTTP request.

[1795] What it does: Sends JSON-formatted data containing the request to the specified API endpoint.

[1796] Step 3:

[1797] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1798] How it works: Video and audio data acquired through sensors is processed in real time to extract facial expressions and vocal characteristics as emotional data.

[1799] Step 4:

[1800] The device sends the acquired emotion data to the server as an HTTP request.

[1801] What it does: Sends JSON-formatted data containing emotion data to the specified API endpoint.

[1802] Step 5:

[1803] The server analyzes the received request and extracts the request content and emotion data.

[1804] What it does: Analyzes the data in the request body using JSON deserialization and extracts "Improve concentration", "Classical music", and emotional data.

[1805] Step 6:

[1806] The server sets the necessary parameters for the generative AI model based on the request content and emotion data.

[1807] What it does: Sets parameters such as tempo, genre, and instrumentation based on the request and emotion data (e.g., tempo 90 BPM, classical genre, emotion "joy").

[1808] Step 7:

[1809] The server calls the AI ​​music generation model using the set parameters.

[1810] What it does: Converts parameters to the model's input format and sends them to the model.

[1811] Step 8:

[1812] The generative AI model generates music based on the parameters it receives.

[1813] What it does: Generates notes and rhythmic patterns based on parameters within the model, and ultimately outputs them as MIDI or audio files.

[1814] Step 9:

[1815] The server acquires the generated music data.

[1816] What it does: Receives generated music data (MIDI or audio files) from the model and converts it to the appropriate format.

[1817] Step 10:

[1818] The server returns the generated music data to the terminal.

[1819] Operation: Sends a music file to the device as a response.

[1820] Step 11:

[1821] The device plays the received music.

[1822] Operation: Launches the device's audio player and plays the received music data.

[1823] Step 12:

[1824] The user inputs instructions into the terminal to save the music that he or she likes.

[1825] What it does: Press the Save button to send a save request.

[1826] Step 13:

[1827] The terminal sends a save request to the server.

[1828] What it does: Sends an HTTP request to the server containing the ID and metadata of the music to be saved.

[1829] Step 14:

[1830] The server stores the music data in a database.

[1831] What it does: Inserts music data and related information into a database.

[1832] Step 15:

[1833] The user selects the music they want to play from the saved list.

[1834] How it works: Select a specific song from the saved list and press the play button.

[1835] Step 16:

[1836] The terminal sends a playback request to the server.

[1837] What it does: Sends an HTTP request to the server containing the ID of the music to play.

[1838] Step 17:

[1839] The server retrieves the music data from the stored database and returns it to the terminal.

[1840] What it does: Executes a database query to retrieve the target music data and sends it to the device.

[1841] Step 18:

[1842] The device plays the received music.

[1843] Operation: Launches the device's audio player and plays the received music data.

[1844] Example 2

[1845] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1846] Conventional music generation systems have difficulty providing music that fully reflects a user's specific needs and emotions. Furthermore, the generated music may not be saved or played smoothly. Furthermore, they lack the functionality to generate music that reflects a user's favorite artists or songs. These issues limit users' ability to enjoy music that best suits their situation and emotions.

[1847] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1848] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request, means for identifying an emotion based on the set parameters and facial expression analysis data, means for invoking a generative model that generates music based on the set parameters and the identified emotion, and means for providing the generated music to the user, thereby enabling the generation and provision of music that reflects the user's specific request and emotion.

[1849] The "means for accepting a specific request from a user" is a means for providing an interface for a user to input a specific request regarding music generation to the system.

[1850] The "means for setting parameters" is a means for determining various setting values ​​required for music generation based on the user's request and emotional data.

[1851] The "means for identifying emotions based on facial expression analysis data" is a means for analyzing sensor data such as the user's facial expressions and voice to identify the user's current emotional state.

[1852] The "means for invoking a generative model for generating music" refers to a means for invoking a machine learning model for generating music based on set parameters and identified emotions.

[1853] The "means for providing the generated music to the user" refers to a means for transmitting the generated music data to the user's terminal in an appropriate format so that the music data can be played back.

[1854] "Means for saving the generated music" refers to means for recording the generated music files in a database or storage so that they can be played back later.

[1855] "Means for playing stored music" refers to means for retrieving music files stored in a database or storage and making them playable on the user's device.

[1856] The "means of using sensors to acquire user emotional data" refers to a means of operating sensors such as cameras and microphones to collect data that grasps the user's current emotional state.

[1857] The "means for pre-registering favorite music artists and songs" is a means for providing an interface for a user to pre-record specific artists and songs into the system.

[1858] "Means for generating music based on registered information and emotional data" refers to means for setting music generation parameters and creating specific music, based on information about artists and songs registered in advance by the user and current emotional data.

[1859] MODE FOR CARRYING OUT THE INVENTION

[1860] The present invention is a system for generating and providing music based on a user's specific requests and emotions. The system generates and provides music to the user based on the user's requests and information obtained by an emotion recognition engine. The generated music can be saved and played, and can also reflect information about music artists and songs previously registered by the user.

[1861] System configuration

[1862] 1. User Interface (UI)

[1863] The user inputs music requests through a dedicated interface on the device, which includes input fields such as text boxes and drop-down menus.

[1864] Example: A user enters "Classical music to help me concentrate" as a request into the text box and clicks the submit button.

[1865] 2. Emotion Recognition Engine

[1866] The device uses sensors such as a camera and microphone to analyze the user's facial expressions and tone of voice to obtain emotional data.

[1867] The server receives the emotion data and sends it to an engine that analyzes it to identify the user's current emotion (e.g., "happy," "sad," "anger").

[1868] Example: If the user is smiling, the emotion engine will recognize the emotion "joy".

[1869] 3. Request Analysis Module

[1870] The server analyzes the request and emotion data sent by the user, and the module sets appropriate parameters based on specific keywords, phrases, and emotions.

[1871] Example: Set tempo and genre parameters based on the keywords "improves concentration" and "classical music" and the emotion "joy."

[1872] 4. Music Generation Module

[1873] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1874] Example: Generate music with a tempo of 90 BPM and classical instrument parameters.

[1875] 5. Music Provision Module

[1876] The server provides the generated music to the user's terminal. This module converts the generated music data into an appropriate format and provides it.

[1877] Example: The generated music file is sent to the terminal as an HTTP response and played in real time.

[1878] 6. Storage and playback module

[1879] Users can save their favorite music, which is then stored in a database.

[1880] The server receives requests to play stored music and provides it to the user.

[1881] Example: By selecting the generated music and pressing the save button, it can be saved in the database and played back later.

[1882] 7. Artist Registration Module

[1883] Users can register their favorite music artists and songs in advance, and this information is referenced when creating music.

[1884] The server stores the registered information in a database and uses it when generating the data.

[1885] Example: If a user registers artist A and requests "music in artist A's style that helps with concentration," the generative model will generate music based on that information.

[1886] This allows users to easily enjoy music that best suits their situation and emotions. For example, if a user requests "classical music that enhances concentration through feelings of joy," the system generates music based on the user's request and emotions and provides it in real time. As another example, if a user requests "relaxing music in the style of artist A," the system can generate and provide music after setting parameters based on pre-registered artist information and the user's emotional data.

[1887] Prompt Sentence Examples

[1888] "Generate classical music that enhances concentration with feelings of joy"

[1889] "Generate relaxing music in the style of artist A"

[1890] As a result, users can easily create and enjoy music suited to a variety of situations.

[1891] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1892] Step 1: Enter and submit a user request

[1893] The user opens a dedicated interface on the device and enters a specific request for the music to be generated (e.g., "classical music to improve concentration") via text boxes and drop-down menus.

[1894] The terminal converts this input data into JSON format and sends it to the server as an HTTP request.

[1895] Input: User's music request

[1896] Output: HTTP request sent to the server

[1897] Step 2: Obtaining emotion data

[1898] The device uses sensors such as a camera and microphone to capture the user's facial expressions and tone of voice and analyze emotional data.

[1899] The server receives the emotion data sent from the terminal and sends it to an emotion recognition engine, which analyzes the data and identifies the user's current emotion.

[1900] Input: User facial and voice data

[1901] Output: Emotion data analyzed by the emotion recognition engine

[1902] Step 3: Analyzing Request and Sentiment Data

[1903] The server analyzes the request and emotion data sent by the user. The request analysis module sets parameters for music generation based on specific keywords, phrases, and emotions.

[1904] Input: User's music request and emotional data

[1905] Output: Parameters for music generation (e.g. tempo 90 BPM, classical genre)

[1906] Step 4: Invoke the model to generate music

[1907] The server calls a generative AI model based on the set parameters, which then uses machine learning algorithms to generate music in real time.

[1908] Input: parameters for music generation

[1909] Output: Generated music file

[1910] Step 5: Submit your music

[1911] The server converts the generated music into an appropriate format (e.g., MP3) and provides it to the user's device. This module sends the generated music data as an HTTP response.

[1912] The device plays this music in real time.

[1913] Input: Generated music file

[1914] Output: Music data sent to the user's device

[1915] Step 6: Store and play your music

[1916] The user can save the generated music via a dedicated button.

[1917] The server records the saved music files in a database and provides them to the user upon receiving a playback request later.

[1918] Input: Generated music file

[1919] Output: Music files stored in the database and music files for playback

[1920] Step 7: Pre-register your artists and songs

[1921] Users can register their favorite artists and songs in advance, and this information is referenced when creating music.

[1922] The server stores information about registered artists and songs in a database and references it when necessary.

[1923] Input: Artist and song information registered by the user

[1924] Output: Artist and song information stored in a database

[1925] (Application example 2)

[1926] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1927] In conventional virtual stores, music generation based on user emotions was not possible, making it impossible to provide background music suited to individual users' emotions and requests in real time, which resulted in a lack of user engagement and personalized shopping experiences.

[1928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1929] In this invention, the server includes means for receiving a specific request from a user, means for setting parameters based on the received request and emotion data acquired in real time, means for invoking a generative model that generates music using the set parameters, means for providing the generated music to the user, and means for playing the music generated based on the user's emotion in real time in a virtual store. This makes it possible to generate and provide music in real time that corresponds to the user's emotion.

[1930] A "specific request" is information that indicates a user's desires or requirements regarding specific music or emotions.

[1931] "Emotional data acquired in real time" refers to emotional information acquired by analyzing a user's facial expressions and tone of voice in real time using devices such as cameras and microphones.

[1932] "Parameters" are the settings and conditions used by the music generation model when generating music.

[1933] A "generative model" is a program for generating music using machine learning algorithms.

[1934] A "virtual store" is an online platform for providing products and services in a virtual space on the Internet.

[1935] "Means for providing music" refers to a method or device for allowing a user to listen to the generated music.

[1936] "Saved music" refers to music data that has been generated and stored in a database.

[1937] "Means for pre-registering favorite music artists and songs" refers to a method or interface that allows a user to pre-register specific artists or songs.

[1938] To implement this invention, it is necessary to build a system that generates music based on specific user requests and emotional data acquired in real time, and provides the music in a virtual store. This system is composed of a user interface, an emotion recognition engine, a request analysis module, a music generation module, a music provision module, a storage and playback module, and an artist registration module.

[1939] First, the user interface (UI) uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice. The user can then input their music request into the device's interface, for example, requesting "relaxing music."

[1940] An emotion recognition engine then analyzes the data captured by the camera and microphone to determine the user's current emotion, using OpenCV and Dlib for facial expression recognition and the Google Cloud Speech-to-Text API for voice emotion recognition.

[1941] The request analysis module then sets appropriate parameters based on the user's request text and the analyzed sentiment data, using a natural language processing library (e.g., spaCy) and TensorFlow for sentiment data analysis.

[1942] The music generation module uses a generative AI model (e.g., Magenta) based on the set parameters to generate music in real time. Here, a machine learning model is built using Python and Keras to generate music.

[1943] The music providing module plays the generated music in real time within the virtual store. The music data is sent to the terminal as an HTTP response and is played within the virtual store to which the user is connected.

[1944] Users can then use the save and play module to save the generated music. The saved music data is stored in a database such as MySQL or PostgreSQL, and can be played back from a playlist at any time.

[1945] In addition, users can use the artist registration module to pre-register their favorite artists and songs, which will allow for more personalized music to be generated based on the registered information.

[1946] Specific examples

[1947] For example, if a user requests "I want to relax" and uses the camera to recognize their emotions, the device sends the request and emotional data to the server. The server then analyzes the data to determine specific parameters for relaxation, such as tempo and music genre, and generates music in real time using a generative AI model (such as Magenta). The generated music is played in the virtual store, allowing users to enjoy shopping while listening to it. Users can also save the music if they want to play it again later.

[1948] Prompt Sentence Examples

[1949] As an example of input from the user, the following prompt sentence can be considered:

[1950] "The user is currently relaxing. The latest request is for relaxing music. Generative parameters for playing soft classical music as background noise in a virtual store setting."

[1951] As described above, the embodiments of the present invention make it possible to provide personalized music suited to the user's emotions, thereby dramatically improving the shopping experience in a virtual store.

[1952] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1953] Step 1:

[1954] Accepting user requests

[1955] The user inputs their music request through the smartphone interface, for example, requesting "relaxing music," and the input data is sent to the device in text format.

[1956] Input: User request (e.g. "relaxing music")

[1957] Output: Request text data

[1958] Step 2:

[1959] Collecting Emotional Data

[1960] The device's camera and microphone are used to collect the user's facial expressions and tone of voice in real time. Image data is acquired from the camera, and audio data is acquired from the microphone.

[1961] Input: Real-time facial and voice data of the user

[1962] Output: Image data, audio data

[1963] Step 3:

[1964] Analysis by emotion recognition engine

[1965] The server analyzes the data obtained from the camera and microphone to identify the user's emotions. It uses OpenCV and Dlib for facial recognition and Google Cloud Speech-to-Text API for voice recognition. The analysis results in the user's current emotion.

[1966] Input: Image data, audio data

[1967] Data processing: facial expression analysis, voice tone analysis

[1968] Output: Emotion data (e.g. "Relaxed")

[1969] Step 4:

[1970] Request analysis and parameter setting

[1971] The server sets appropriate parameters based on the received user request text and the analyzed emotion data. This analysis uses a natural language processing library (e.g., spaCy) and TensorFlow for emotion data analysis.

[1972] Input: Request text data, emotion data

[1973] Data processing: Natural language processing analysis, emotion data integration

[1974] Output: Parameter setting data (e.g. tempo, music genre)

[1975] Step 5:

[1976] Music Generation

[1977] The server calls a generative AI model (e.g., Magenta) based on the set parameters and generates music in real time. The generative AI model uses machine learning algorithms to generate music.

[1978] Input: Parameter setting data

[1979] Data Computation: Real-time Music Generation with Generative AI Models

[1980] Output: Generated music data

[1981] Step 6:

[1982] Music provided by

[1983] The server provides the generated music to the virtual store in real time. The music data is sent to the terminal as an HTTP response, and the music is played in the virtual store to which the user is connected.

[1984] Input: Generated music data

[1985] Data processing: Music data format conversion and transmission

[1986] Output: Music playback on the user's device

[1987] Step 7:

[1988] Music Storage

[1989] Users can save their favorite music. The server accepts the save request and stores the music data in a database.

[1990] Input: User save request, generated music data

[1991] Data processing: Saving to database

[1992] Output: Data stored in the database

[1993] Step 8:

[1994] Playing music

[1995] The server receives a request to play the stored music and provides the music data to the user, which is retrieved from the playlist and played on the user's device.

[1996] Input: User playback request, data stored in database

[1997] Data processing: Retrieving and sending data from the database

[1998] Output: Music playback on the user's device

[1999] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2000] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2001] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2002] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2003] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2004] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2005] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2006] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2007] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2008] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2009] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2010] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2011] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2012] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2013] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2014] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2015] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2016] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2017] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2018] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2019] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2020] The following is further disclosed regarding the above embodiment.

[2021] (Claim 1)

[2022] a means for accepting specific requests from users;

[2023] means for setting parameters based on the received request;

[2024] means for invoking a generative model that generates music using the set parameters;

[2025] a means for providing the generated music to a user;

[2026] A system including:

[2027] (Claim 2)

[2028] means for storing the generated music;

[2029] means for playing the stored music;

[2030] The system of claim 1 further comprising:

[2031] (Claim 3)

[2032] A way to pre-register your favorite music artists and songs,

[2033] means for generating music based on the registered information;

[2034] 10. The system of claim 1.

[2035] "Example 1"

[2036] (Claim 1)

[2037] a means for accepting specific requests from users;

[2038] means for setting parameters based on the received request;

[2039] A means for calling a generative AI model that generates music using set parameters;

[2040] a means for providing the generated music to a user;

[2041] A system including:

[2042] (Claim 2)

[2043] means for storing the generated music;

[2044] means for playing the stored music;

[2045] The system of claim 1 further comprising:

[2046] (Claim 3)

[2047] A way to pre-register your favorite music artists and songs,

[2048] means for generating music based on the registered information;

[2049] 10. The system of claim 1.

[2050] "Application Example 1"

[2051] (Claim 1)

[2052] a means for accepting specific requests from users;

[2053] means for setting parameters based on the received request;

[2054] means for invoking a generative model that generates music using the set parameters;

[2055] a means for providing the generated music to a user;

[2056] A means for inputting music requests according to the situation of the physical store using a user interface;

[2057] means for generating music in real time based on input requests and providing the music to a store's sound system;

[2058] A system including:

[2059] (Claim 2)

[2060] means for storing the generated music;

[2061] means for playing the stored music;

[2062] The system of claim 1 further comprising:

[2063] (Claim 3)

[2064] A way to pre-register your favorite music artists and songs,

[2065] means for generating music based on the registered information;

[2066] 10. The system of claim 1.

[2067] "Example 2: Combining Emotion Engines"

[2068] (Claim 1)

[2069] means for accepting specific requests from a user;

[2070] means for setting parameters based on the received request;

[2071] A means for identifying emotions based on set parameters and facial expression analysis data;

[2072] means for invoking a generative model to generate music based on the set parameters and the identified emotions;

[2073] means for providing the generated music to a user;

[2074] A system including:

[2075] (Claim 2)

[2076] means for storing the generated music and means for playing the stored music;

[2077] further comprising means for using a sensor to obtain emotion data of the user;

[2078] 10. The system of claim 1.

[2079] (Claim 3)

[2080] A way to pre-register your favorite music artists and songs,

[2081] means for generating music based on the registered information and emotion data;

[2082] 10. The system of claim 1.

[2083] "Application example 2 when combining emotion engines"

[2084] (Claim 1)

[2085] a means for accepting specific requests from users;

[2086] means for setting parameters based on received requests and emotion data obtained in real time;

[2087] means for invoking a generative model that generates music using the set parameters;

[2088] a means for providing the generated music to a user;

[2089] A means for playing music generated based on the user's emotions in real time in a virtual store;

[2090] A system including:

[2091] (Claim 2)

[2092] means for storing the generated music;

[2093] means for playing the stored music;

[2094] The system of claim 1 further comprising:

[2095] (Claim 3)

[2096] A way to pre-register your favorite music artists and songs,

[2097] means for generating music based on the registered information and emotion data acquired in real time;

[2098] 10. The system of claim 1. [Explanation of symbols]

[2099] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for accepting specific requests from users; means for setting parameters based on the received request; means for invoking a generative model that generates music using the set parameters; a means for providing the generated music to a user; A system including:

2. means for storing the generated music; means for playing the stored music; The system of claim 1 further comprising:

3. A way to pre-register your favorite music artists and songs, means for generating music based on the registered information; The system of claim 1 .

Images