System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to provide audio input; means for a server to receive audio data and analyze music characteristics; means for the server to generate music in real time based on the analysis results; means for transmitting the generated music to the user's terminal; means for sharing the music created by the user with other users; and means for allowing other users to add new parts.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Music production requires advanced skills and knowledge, making it difficult for many individual creators to create high-quality music on their own. There is also a need to provide users with a rich musical experience even without band or session members. Furthermore, existing online platforms have limited real-time collaboration and lack effective ways for users to engage in creative collaboration. [Means for solving the problem]

[0005] The present invention provides an online platform that allows users to create music in real time and share and collaborate with other users. Specifically, the system includes a means for users to input voice, a server to receive the voice data and analyze musical characteristics, a means for the server to generate music in real time based on the analysis results, a means for transmitting the generated music to the user's device, a means for users to share the music they have created with other users, and a means for other users to add new parts. This system allows users without advanced skills to create high-quality music and engage in creative collaboration with other users online.

[0006] "User" means any person or entity that uses the System to create, share, and collaborate on music.

[0007] "Audio input" refers to the act of a user providing audio data, such as musical instrument playing or vocals, to a system through a microphone or other device.

[0008] "Server" refers to a computer system that receives, analyzes, creates, stores, and shares audio data.

[0009] "Musical characteristics" refers to musical elements such as tempo, pitch, and rhythm contained in audio data.

[0010] "Real-time" refers to a state in which a response to the user's voice input is immediate, and processing and feedback are carried out without delay.

[0011] "Accompaniment data" refers to background music data generated by AI based on the user's voice input.

[0012] "Means for generating" refers to algorithms or programs for creating new music data based on the user's voice input.

[0013] "Terminal" refers to devices used by users, such as computers, smartphones, and tablets.

[0014] "Cloud storage" refers to a remote server that stores and manages data over the Internet.

[0015] "Means of sharing" refers to the functions and platforms for sharing created music data with other users online.

[0016] "New parts" refer to the user's new performance or vocal data that is added to existing music data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides an online platform for creating music and collaborating with other users in real time. Specific embodiments for implementing the present invention are described below.

[0039] Overall system configuration

[0040] The system consists of the following components:

[0041] 1. User's Device

[0042] 2. Server

[0043] 3. Cloud Storage

[0044] Receiving voice input

[0045] Users use their own devices to input their musical instruments and vocals. A microphone is connected to the device, and the input sound is captured as digital data, allowing for real-time recording of the performance.

[0046] Sending audio data

[0047] The device then transmits the captured audio data to a server, and this transmission process occurs in real time over the Internet.

[0048] Analysis of audio data

[0049] The server analyzes the received audio data. Specifically, the data analysis module extracts musical characteristics such as tempo, pitch, and rhythm. The analysis results are used as the basis for the subsequent music generation process.

[0050] Real-time music generation

[0051] The server generates accompaniment data in real time based on the musical characteristics, and an AI algorithm generates accompaniment data that is adapted to the analysis results and the user's performance, and sends it to the device in the appropriate format.

[0052] User feedback

[0053] The device receives the generated accompaniment data and provides real-time feedback to the user, allowing the user to continue playing along with the generated music.

[0054] Recording and storage

[0055] The server simultaneously records the user's performance data and the generated accompaniment data, and stores the generated music data in cloud storage, allowing the user to access, edit, and re-share the music data later.

[0056] Music sharing and collaboration

[0057] Users have a dedicated interface to share the music they create with other users. This interface is located on the device and is easy to navigate. Other users can view and play the shared music and add their own new parts.

[0058] The server re-analyzes and generates new music data, and saves the updated music data in cloud storage, facilitating ongoing collaboration.

[0059] Specific examples

[0060] Example 1: User A holds a guitar and plays a chord progression into the device.

[0061] Device: Captures guitar audio data and sends it to the server.

[0062] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[0063] Terminal: Plays back the accompaniment data generated in real time to User A.

[0064] Example 2: User B adds a piano part.

[0065] User B inputs voice into the device and provides the piano part.

[0066] Device: Sends audio data to the server.

[0067] Server: Analyzes the piano part and integrates it into the existing music data.

[0068] This will enable advanced musical collaboration between users.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[0072] Step 2:

[0073] The device temporarily stores the captured audio data in an internal buffer and then transmits it to a server via the Internet.

[0074] Step 3:

[0075] The server receives the voice data sent from the terminal and checks the format and integrity of the received data.

[0076] Step 4:

[0077] The server uses an audio data analysis module to extract musical characteristics such as tempo, pitch, rhythm, etc. The results of this analysis are stored in a database.

[0078] Step 5:

[0079] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics, and this accompaniment data is generated in a format that adapts to the user's input.

[0080] Step 6:

[0081] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[0082] Step 7:

[0083] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0084] Step 8:

[0085] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them in cloud storage as a single music file.

[0086] Step 9:

[0087] The user has an interface that allows them to view the recorded music files on their device and edit them if necessary.

[0088] Step 10:

[0089] Users can use a dedicated function to share edited music files with other users and publish the music files online.

[0090] Step 11:

[0091] The server distributes the published music files for other users to view, play, and add their own parts to.

[0092] Step 12:

[0093] Other users input new parts and send the audio data to the server.

[0094] Step 13:

[0095] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[0096] Step 14:

[0097] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[0098] These steps facilitate real-time music creation and collaboration.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In today's world, music production requires advanced technology and equipment, making it difficult to generate music in real time and facilitate smooth collaboration between multiple users. There is a particular demand for real-time music generation and collaboration with other users while receiving immediate feedback, but few systems currently exist that enable this. Therefore, there is an urgent need to provide a system that allows users to easily generate music and collaborate with other users in real time.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for receiving audio data and analyzing musical characteristics, means for generating music in real time based on the analysis results, and means for re-analyzing and generating new music data and saving the updated music data in cloud storage. This allows users to receive real-time feedback on their voice-input performance data along with accompaniment data, enabling smooth collaboration with other users.

[0104] "User" means an individual or organization that utilizes the system to input voice and perform music, create music, and collaborate.

[0105] "Terminal" means a device used by a user to perform voice input and has the function of capturing voice data and transmitting it to a server.

[0106] A "server" is a central processing unit that receives and analyzes audio data, and generates and manages music in real time.

[0107] "Audio data" refers to digital data of musical instrument sounds or vocals played by a user on a terminal.

[0108] "Music characteristics" refers to characteristic information such as tempo, pitch, and rhythm extracted from audio data.

[0109] The "analysis results" are data on musical characteristics extracted from the audio data.

[0110] "Generating music in real time" means instantly generating accompaniment data adapted to the user's performance based on the analysis results.

[0111] "Accompaniment data" refers to auxiliary music data such as drums and bass that is generated to match the user's performance.

[0112] "Cloud storage" is a service that stores and manages data on the Internet.

[0113] "Providing feedback" means playing the generated accompaniment data to the user in real time.

[0114] "Collaboration" is the activity of multiple users working together to create, edit, and share music.

[0115] "Adding a new part" means that another user adds a new instrument sound or vocals to existing music data.

[0116] "Updated music data" refers to the latest music data that has new parts added and has been re-analyzed and re-generated.

[0117] The present invention provides a system that allows a user to input voice, generate music in real time, and collaborate with other users. Specific embodiments for carrying out the present invention are described below.

[0118] Overall system configuration

[0119] The system consists of the following components:

[0120] 1. User's Device

[0121] 2. Server

[0122] 3. Cloud Storage

[0123] Receiving voice input

[0124] Users use their own devices to input their musical instruments or vocals. A microphone is connected to the device, and the input sound is captured as digital data. This allows for real-time recording of performances. For example, if a user plays a chord progression on a guitar, the sound is captured by the microphone.

[0125] Sending audio data

[0126] The device transmits the captured audio data to the server. This transmission process is performed in real time via the Internet. For example, as soon as the user starts playing, the device collects audio data in real time and transmits it to the server.

[0127] Analysis of audio data

[0128] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. This analysis module extracts musical characteristics such as tempo (e.g., 120 BPM), pitch (e.g., key of C), and rhythm. These characteristics are the information necessary for subsequent AI music generation. For example, the server determines from the audio data that the tempo is 120 BPM and the key is C.

[0129] Real-time music generation

[0130] The server generates accompaniment data in real time based on the analysis results. Using a generative AI model, it generates accompaniment data adapted to the user's performance. For example, it generates drum and bass accompaniments based on guitar audio data. The generated data is converted into an appropriate format and sent to the device.

[0131] User feedback

[0132] The device receives the accompaniment data sent from the server. The received data is fed back to the user in real time. This allows the user to continue playing with the accompaniment added to their own performance. For example, drums and bass can be played simultaneously along with the guitar playing.

[0133] Recording and storage

[0134] The server simultaneously records the user's performance data and the generated accompaniment data. The recorded music data is stored in cloud storage, where users can access, edit, and reshare it later. This allows the music data to remain available even after the session has ended.

[0135] Music sharing and collaboration

[0136] Users have an interface to share the music they create with other users. The shared music can be viewed and played by other users, and new parts can be added. For example, User A can create a guitar accompaniment and User B can add a piano part.

[0137] Reparsing and updating

[0138] The server then re-analyzes the newly added music data and makes any necessary updates. It then integrates the new music data with the existing data using a generative AI model and saves the updated music data back to cloud storage, enabling continuous musical collaboration between users.

[0139] Specific examples

[0140] Example 1:

[0141] User A picks up a guitar and plays a chord progression into the terminal.

[0142] Device: Captures guitar audio data and sends it to the server.

[0143] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[0144] Terminal: Plays back the accompaniment data generated in real time to User A.

[0145] Example 2:

[0146] User B adds a piano part.

[0147] User B: Provides the piano part by dictating the song on the device.

[0148] Device: Sends audio data to the server.

[0149] Server: Analyzes piano parts and integrates them into existing music data, enabling advanced musical collaboration between users.

[0150] Prompt Sentence Examples

[0151] "Enter your guitar chord progression and generate accompaniment in real time."

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1:

[0154] User voice input

[0155] Users use their devices to input musical instruments or vocals. A microphone is used as an input device, and the microphone captures audio data as digital data. For example, when a user plays a guitar, the microphone collects the audio and converts it into digital data.

[0156] Input: User's voice

[0157] Output: Digitized audio data

[0158] Step 2:

[0159] Device capture and transmission of audio data

[0160] The device captures digitized audio data and transmits it over the internet to a server. This process happens in real time, with high speeds to minimize latency. For example, after a user plays the guitar, the audio data is sent instantly to the server.

[0161] Input: Digitized audio data

[0162] Output: Audio data sent to the server

[0163] Step 3:

[0164] Receiving and analyzing voice data by the server

[0165] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. Specifically, musical characteristics such as tempo, pitch, and rhythm are extracted. For example, the server analyzes the audio data and determines that the tempo is 120 BPM and the key is C.

[0166] Input: Audio data sent to the server

[0167] Output: Extracted musical characteristics data (tempo, pitch, rhythm)

[0168] Step 4:

[0169] Real-time music generation by server

[0170] The server uses a generative AI model to generate accompaniment data in real time based on the analyzed musical characteristic data. This AI model creates drum and bass accompaniment that is adapted to the user's performance. For example, it generates drum and bass accompaniment for a 120 BPM song in the key of C. The generated accompaniment data is converted into an appropriate format and sent to the device.

[0171] Input: Musical characteristics data (tempo, pitch, rhythm)

[0172] Output: Generated accompaniment data

[0173] Step 5:

[0174] Sending feedback to the device

[0175] The device receives the accompaniment data sent from the server. The received accompaniment data is played back in real time, providing feedback to the user. This allows the user to use the accompaniment generated to match their own performance. For example, a user can play the guitar while simultaneously listening to the generated drum and bass accompaniment.

[0176] Input: Generated accompaniment data

[0177] Output: Real-time feedback sound to the user

[0178] Step 6:

[0179] Server-based recording and storage

[0180] The server simultaneously records the user's performance data and the generated accompaniment data. This recording data is stored in cloud storage. Users can later access, edit, and reshare this data. For example, users can review and edit the recording data after the session has ended.

[0181] Input: User's performance data, generated accompaniment data

[0182] Output: Recording data saved in cloud storage

[0183] Step 7:

[0184] Users can share and collaborate on music

[0185] Users share their music with other users using a dedicated interface that is intuitive and easy to use. Other users can browse, play, and add new parts to the shared music. For example, user B can add a piano part to a song created by user A.

[0186] Input: User-created music data

[0187] Output: Share and collaborate with others

[0188] Step 8:

[0189] Server reparsing and updating

[0190] The server then re-analyzes the newly added music data and makes any necessary updates. It then uses a generative AI model to integrate the new music data with existing data and saves the updated music data in cloud storage. For example, it can analyze newly added parts and update the entire song, promoting collaboration.

[0191] Input: Newly added music data

[0192] Output: Updated music data (stored in cloud storage)

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] Conventional factory robot control systems require complex programming and settings, making efficient operation difficult. Furthermore, it is extremely difficult to achieve real-time collaborative work between multiple robots, often resulting in a loss of efficiency and safety. The present invention aims to solve these problems by realizing simple and efficient robot control and collaborative work using voice input.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for a user to input voice, means for the server to receive the voice data and analyze musical characteristics, means for the server to generate music in real time based on the analysis results, means for transmitting the generated music to the user's terminal, means for the user to share the music created by the user with other users, means for other users to add new parts, means for analyzing the voice input and generating operation commands for various devices, and means for transmitting the generated operation commands to industrial devices and causing them to execute them. This enables simple and efficient robot control using voice input and cooperative work between multiple robots.

[0198] "User" means any person or group of people who uses the online platform to provide voice input.

[0199] "Voice input" is the act or process of capturing user-uttered speech as digital data.

[0200] The "server" is a central processing device that receives and analyzes audio data, and generates music and motion commands.

[0201] "Voice data" refers to digitized data of voice input by a user.

[0202] "Musical characteristics" are musical elements such as tempo, pitch, and rhythm extracted from audio data.

[0203] "Real-time" means that data is sent, received, and processed nearly simultaneously.

[0204] "Music generation" is the process of creating new music based on analyzed musical characteristics.

[0205] A "terminal" is an electronic device used by a user (such as a smartphone, tablet, or PC).

[0206] "Sharing" is the process by which a user shares music or data they have created with other users.

[0207] A "robot" is an automated mechanical device that performs tasks in a factory.

[0208] An "operation command" is an instruction to operate various devices that is generated based on the analyzed voice input.

[0209] The present invention is a system that generates operational commands for music and industrial equipment in real time through user voice input, enabling efficient and rapid collaboration.

[0210] Overall system configuration

[0211] The system consists of the following components:

[0212] 1. User's Device

[0213] 2. Server

[0214] 3. Cloud Storage

[0215] 4. Industrial equipment (robots)

[0216] Voice input and analysis

[0217] The user inputs voice. The voice is captured as digital data using a microphone connected to the device. This voice data is sent from the device to a server. The server analyzes the received voice data and extracts musical characteristics such as tempo, pitch, and rhythm. The server also analyzes the instructions the user gives to industrial equipment.

[0218] Music Generation and Feedback

[0219] The server generates accompaniment data in real time based on the musical characteristics, and the generated music data is sent to the user's device and fed back in real time, allowing the user to continue playing along with the generated music.

[0220] Robot motion command generation and execution

[0221] The server analyzes the user's voice input and generates operational commands for the industrial equipment, which are then sent over the network to the industrial equipment, causing the robot to instantly perform the required action.

[0222] Recording and storage

[0223] The server simultaneously records the user's performance data, the generated accompaniment data, and the generated movement command data, and saves them in cloud storage, allowing the user to access, edit, and re-share the data later.

[0224] Music sharing and collaboration

[0225] Users can use a dedicated interface to share the music they create with other users, who can then view and play the music, add new parts, and share operating instructions for industrial equipment.

[0226] Hardware and software used

[0227] Hardware:

[0228] Microphone (for voice input)

[0229] Industrial equipment (e.g. robotic arms)

[0230] software:

[0231] Python (for processing and server communication)

[0232] sounddevice module (audio recording)

[0233] requests module (sending HTTP requests)

[0234] AI voice analysis server (voice analysis and command generation)

[0235] Specific examples

[0236] Example 1:

[0237] A worker in a factory commands a robot to "go get the next part." A microphone captures this voice and sends it as audio data to a server. The server analyzes the voice and generates an action command to "go get the part." The generated action command is sent to the robot, which then goes to get the part.

[0238] Example 2:

[0239] User A picks up a guitar and plays a chord progression into the device. This performance data is sent to the server, which analyzes it to determine that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment, which is then fed back to User A in real time.

[0240] Prompt Sentence Examples

[0241] "Voice command to the robot: Go get the next part."

[0242] "Pick up the parts from the conveyor"

[0243] "Move to the assembly area."

[0244] The above is an embodiment of the present invention.

[0245] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0246] Step 1:

[0247] The user provides voice input. The user issues a voice command using a microphone connected to the device. The device captures this voice as digital data. The input is the user's voice, and the output is digitized voice data. This digital data is analyzed in the next step.

[0248] Step 2:

[0249] The device sends the captured audio data to the server. The device transfers the audio data to the server in real time via the Internet. The input is the audio data captured by the device, and the output is the data sent to the server.

[0250] Step 3:

[0251] The server analyzes the received audio data. Using an audio analysis module, the server extracts musical characteristics such as tempo, pitch, and rhythm from the data. It also analyzes operational commands for industrial equipment. The input is the audio data sent to the server, and the output is the extracted musical characteristics and operational commands.

[0252] Step 4:

[0253] The server generates music in real time based on the analysis results. The server uses an AI algorithm to generate accompaniment data based on the extracted musical characteristics. The input is the extracted musical characteristics, and the output is the generated accompaniment data.

[0254] Step 5:

[0255] The generated music is sent to the user's device. The server sends the generated accompaniment data to the user's device in real time. The input is the generated accompaniment data, and the output is the accompaniment data sent to the user's device.

[0256] Step 6:

[0257] The server generates operation commands for industrial equipment based on the analysis results. The server generates appropriate operation commands based on the voice analysis results and sends them to the industrial equipment via the network. The input is the voice analysis results, and the output is the generated operation commands.

[0258] Step 7:

[0259] Industrial equipment performs tasks according to operation commands. Industrial equipment (such as robots) performs specified operations according to operation commands sent from the server. The input is the operation command sent from the server, and the output is the operation of the industrial equipment.

[0260] Step 8:

[0261] The server simultaneously records the user's performance data and the generated accompaniment data, and stores them in cloud storage. The input is the user's performance data and the generated accompaniment data, and the output is the data stored in cloud storage.

[0262] Step 9:

[0263] A user can share music they have created with other users. The user's device uses a dedicated interface to share the music they have created with other users. The input is data stored in cloud storage, and the output is the shared music data.

[0264] Step 10:

[0265] Another user adds a new part. Another user adds a new part to the shared music data and sends it back to the system. The input is the shared music data, and the output is the music data with the new part added.

[0266] The above are the specific processing steps of the program of the present invention.

[0267] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0268] The present invention provides an online platform that combines an emotion engine that recognizes a user's emotions to generate music in real time according to the user's emotional state, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[0269] Overall system configuration

[0270] The system consists of the following components:

[0271] 1. User's Device

[0272] 2. Server

[0273] 3. Cloud Storage

[0274] 4. Emotion Engine

[0275] Receiving voice input and analyzing emotions

[0276] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[0277] Sending voice data and emotion data

[0278] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[0279] Analysis of voice data and use of emotion data

[0280] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the AI ​​algorithm.

[0281] Real-time music generation and emotional reflection

[0282] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[0283] User feedback

[0284] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0285] Recording and storage

[0286] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0287] Music sharing and collaboration

[0288] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[0289] Specific examples

[0290] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[0291] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[0292] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[0293] Terminal: Plays back the accompaniment data generated in real time to User A.

[0294] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[0295] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[0296] Terminal: Sends voice data and emotion data to the server.

[0297] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[0298] In this way, music creation and collaboration that reflects the user's emotional state becomes possible.

[0299] The processing flow will be explained below.

[0300] Step 1:

[0301] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[0302] Step 2:

[0303] The terminal temporarily stores the voice data in a buffer and simultaneously transmits this data to the emotion engine.

[0304] Step 3:

[0305] The emotion engine analyzes the received voice data to determine the user's emotional state, for example, generating emotion categories such as "joy," "sadness," and "anger."

[0306] Step 4:

[0307] The device transmits the captured voice data and emotion data obtained from the emotion engine to a server in real time via the Internet.

[0308] Step 5:

[0309] The server receives the voice data and emotion data sent from the device and checks the format and integrity of the received data.

[0310] Step 6:

[0311] The server analyzes the audio data and extracts musical characteristics such as tempo, pitch, and rhythm, and stores the analysis results in a database.

[0312] Step 7:

[0313] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics and emotional data. The tempo, key, rhythm, and other aspects of the generated music are dynamically adjusted according to the emotional data.

[0314] Step 8:

[0315] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[0316] Step 9:

[0317] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0318] Step 10:

[0319] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them in cloud storage as a single music file.

[0320] Step 11:

[0321] The user has an interface to review the recorded music files from their device and edit them if necessary.

[0322] Step 12:

[0323] Users can use a dedicated function to share edited music files with other users and publish them online.

[0324] Step 13:

[0325] The server distributes the published music files for other users to view, play, and add their own parts to.

[0326] Step 14:

[0327] Other users input new parts and send the audio data to the server.

[0328] Step 15:

[0329] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[0330] Step 16:

[0331] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[0332] These steps facilitate real-time music generation that reflects emotional states and collaboration between users.

[0333] Example 2

[0334] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] Conventional music generation systems have difficulty generating music that reflects a user's emotional state in real time, and sharing and collaboration between users is also limited. To solve these problems, an effective method is needed to analyze emotions, generate music based on that emotion in real time, and share and collaborate with other users.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving captured audio data and emotional data and analyzing musical characteristics and emotional states, means for generating music in real time based on the analysis results and emotional data, and means for transmitting the generated music to the user's terminal. This enables the generation of music that reflects emotions in real time, as well as effective sharing and collaboration between users.

[0337] "User" refers to a person who uses a terminal to input voice data and use the system.

[0338] A "terminal" is a hardware device that allows a user to input voice information, and includes devices such as microphones, computers, and smartphones.

[0339] "Voice data" refers to data that represents in digital form the voice signal input by a user through a terminal.

[0340] "Emotional data" refers to information about the user's emotional state obtained by analyzing voice data.

[0341] "Server" refers to a computer system for receiving and analyzing audio and emotional data and generating music.

[0342] "Musical characteristics" refers to musical elements such as tempo, pitch, and rhythm obtained by analyzing audio data.

[0343] "Real-time" means that user input is processed immediately, without delay.

[0344] "Music generation" refers to the process of creating new music based on audio and emotional data.

[0345] "Sending to terminal" refers to sending the music data generated by the server to the user's terminal via the network.

[0346] "Dedicated Interface" refers to the dedicated operating screens and tools used by users to share and collaborate on music they create with other users.

[0347] "Cloud storage" refers to online storage services for storing and accessing data over the Internet.

[0348] "Adding a new part" refers to another user adding new audio data or instrument parts to an existing music file.

[0349] This invention provides an online platform that generates music in real time according to the user's emotional state by combining an emotion engine that recognizes the user's emotions, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[0350] Overall system configuration

[0351] The system consists of the following components:

[0352] 1. User's Device

[0353] 2. Server

[0354] 3. Cloud Storage

[0355] 4. Emotion Engine

[0356] Receiving voice input and analyzing emotions

[0357] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[0358] Sending voice data and emotion data

[0359] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[0360] Analysis of voice data and use of emotion data

[0361] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the generative AI model.

[0362] Real-time music generation and emotional reflection

[0363] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[0364] User feedback

[0365] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0366] Recording and storage

[0367] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0368] Music sharing and collaboration

[0369] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[0370] Specific examples

[0371] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[0372] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[0373] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[0374] Terminal: Plays back the accompaniment data generated in real time to User A.

[0375] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[0376] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[0377] Terminal: Sends voice data and emotion data to the server.

[0378] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[0379] Prompt Sentence Examples

[0380] You can check the system's operation by inputting the following prompt sentences into the generative AI model.

[0381] When a user plays the guitar and has the emotion "joy", send the audio data and emotion data to the server to generate a cheerful accompaniment in the key of C at 120 BPM in real time.

[0382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0383] Step 1: User speaks

[0384] Users input musical instruments or vocals using a microphone connected to their device. The microphone connected to the device captures the audio signal and converts it into digital data. The input is the user's voice, and the output is digitally converted audio data.

[0385] Step 2: Sentiment Analysis

[0386] The device passes the voice data captured in step 1 to the emotion engine. The emotion engine analyzes the tone, pitch, tempo, etc. of the voice data to identify the user's emotional state (e.g., "joy," "sadness," etc.). The input is the digital voice data obtained in step 1, and the output is emotion data resulting from the analysis.

[0387] Step 3: Sending voice and emotion data

[0388] The device transmits the emotion data and voice data from step 2 to the server. The transmission process is carried out in real time via the Internet. The input is the voice data and emotion data, and the output is a confirmation message that the server has received the data.

[0389] Step 4: Receiving and analyzing voice and emotion data

[0390] The server receives the audio data and emotion data sent in step 3. After receiving the audio data, it checks the format and integrity of the audio data and analyzes the musical characteristics (e.g., tempo, pitch, rhythm). The input is the received audio data and emotion data, and the output is the analyzed musical characteristic data and emotion data.

[0391] Step 5: Real-time music generation

[0392] The server uses a generative AI model to generate music in real time based on the analyzed musical characteristic data and emotional data. The tempo, key, rhythm, etc. are dynamically adjusted based on the emotional data. The input is the musical characteristic data and emotional data, and the output is the generated accompaniment data.

[0393] Step 6: Send the generated music data to your device

[0394] The server converts the generated accompaniment data into an appropriate format in real time and sends it to the user's terminal. The input is the generated accompaniment data, and the output is the accompaniment data received by the user's terminal.

[0395] Step 7: User feedback and real-time playback

[0396] The device decodes the received accompaniment data and plays it back to the user in real time. The user can then perform additional music along with the accompaniment. The input is the received accompaniment data, and the output is music played back to the user in real time.

[0397] Step 8: Record and save your performance data

[0398] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotion data, and stores them as a single music file in cloud storage. The input is the user's performance data, the generated accompaniment data, and the emotion data, and the output is the music file stored in cloud storage.

[0399] Step 9: Share and collaborate on music

[0400] Users use a dedicated interface to share music files stored in cloud storage with other users. Other users can browse, play, and add their own parts to the shared music files. The input is the music file retrieved from cloud storage, and the output is the new music file that has been shared and edited.

[0401] Example operation

[0402] As a concrete example, if User A plays the guitar and the emotion engine analyzes that his / her emotional state is "joy," the device sends this to the server. The server receives the data, generates a cheerful accompaniment with a tempo of 120 BPM and the key of C, and sends it to the device. The device plays it in real time, and User A continues playing along with the accompaniment. If User B later wants to add his / her own piano part, he / she retrieves the music file from cloud storage and plays it, reflecting the emotional data of "sadness." The server integrates the data and generates a new music file that reflects the emotion.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] In recent years, there has been a demand for music generation that reflects a user's emotions, but there is a lack of technology that can generate music in real time based on emotions and share and collaborate with other users. Therefore, providing a system that can analyze a user's emotions, generate music based on those emotions in real time, and make it easy to share with other users has become an important challenge.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0407] In this invention, the server includes means for analyzing a user's emotions and dynamically generating music based on the results, means for dynamically adjusting the tempo, key, and rhythm of the generated music, and means for providing feedback on the generated music to the user in real time and allowing the user to perform additional performances. This enables music to be generated in real time according to the user's emotions, and allows easy sharing and collaboration with other users.

[0408] A "terminal" is a device that allows a user to input voice information, including smartphones, tablets, and personal computers.

[0409] "Voice data" is digital sound information acquired when a user makes a voice input.

[0410] "Musical characteristics" refers to musical features such as tempo, pitch, and rhythm extracted from audio data.

[0411] A "server" is a high performance computing device located at a remote location that receives audio data and performs analysis and music generation.

[0412] "Real-time" refers to a time range in which processing occurs so quickly that there is almost no noticeable delay from user input.

[0413] "Accompaniment data" refers to the accompaniment portion of the music generated based on the user's voice data and emotion data.

[0414] "Cloud storage" is a remote storage service for storing data on the Internet.

[0415] "Emotion data" is information that indicates the emotional state of the user analyzed from the user's voice data.

[0416] "Generative AI model" refers to an algorithm or program for generating music using artificial intelligence techniques.

[0417] A "prompt" refers to an instruction or question that is input into an AI model.

[0418] "Collaboration" refers to multiple users working together to create a single musical work.

[0419] This invention is a system that analyzes a user's emotions, generates music in real time based on the results, and provides an online platform for sharing and collaborating with other users.

[0420] Overall system configuration

[0421] The system consists of the following components:

[0422] 1. User device: The device through which the user inputs voice and transmits emotion data. This includes smartphones, tablets, and PCs.

[0423] 2. Server: A high-performance computer device installed in a remote location that receives audio data and emotion data, analyzes them, and generates music.

[0424] 3. Cloud storage: A remote storage service on the Internet for storing generated music data.

[0425] 4. Emotion engine: An engine that contains algorithms that analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Azure Cognitive Services.

[0426] Receiving voice input and analyzing emotions

[0427] Users input musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine receives the same audio input data and analyzes the user's emotional state. For example, if the user is feeling "sad," the system will recognize that emotion.

[0428] Sending voice and emotion data

[0429] The device transmits the captured voice data and analyzed emotion data to a server. This transmission process is carried out in real time via the Internet. The voice data and emotion data are transmitted together and checked for data integrity.

[0430] Analysis of voice data and use of emotion data

[0431] The server receives the audio data and emotion data sent from the device. The received audio data is used to analyze musical characteristics, extracting information such as tempo, pitch, and rhythm. The emotion data is used as input for the AI ​​algorithm and reflected in the generated music.

[0432] Real-time music generation and emotional reflection

[0433] The server generates accompaniment data in real time based on musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. For example, if the emotional engine analyzes "joy," it will generate upbeat, fast-paced music.

[0434] User feedback

[0435] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional parts along with the accompaniment, thereby realizing a musical experience that is in line with the user's emotions.

[0436] Recording and storage

[0437] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0438] Music sharing and collaboration

[0439] Users use a dedicated interface to share the music files they create with other users. This interface is located on the device and is easy to operate. Other users can view and play the shared music files and add their own parts. For example, if user A starts playing the piano in a lonely mood, other users can add their own instruments to the music and collaborate.

[0440] Prompt example

[0441] "Use the Emotion Music Real-time Generator (EMRG) to express your lonely feelings on piano, then collaborate with other users by adding their own instruments to your music."

[0442] This concludes the detailed description of the "Mode for carrying out the invention." The entire system cooperates to automatically create, share, and collaborate on music that is in line with the user's emotions.

[0443] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0444] Step 1:

[0445] The user performs voice input. Specifically, the user inputs an instrument or vocals into a device such as a smartphone, tablet, or PC. At this time, the microphone connected to the device captures the audio data in digital format. Input: User's voice. Output: Digital audio data.

[0446] Step 2:

[0447] The device sends the captured voice data to the emotion engine, which uses algorithms such as IBM Watson Tone Analyzer and Azure Cognitive Services to analyze the voice data and identify the user's emotional state. Input: Digital voice data. Output: Analyzed emotion data.

[0448] Step 3:

[0449] The device transmits the analyzed emotion data and voice data to the server in real time via the Internet. The server checks the integrity of the received data. Input: Voice data and emotion data. Output: Data packets (voice data and emotion data).

[0450] Step 4:

[0451] The server analyzes the received audio data to extract musical characteristics. This process uses algorithms to identify information such as tempo, pitch, and rhythm. Input: Audio data. Output: Musical characteristics data.

[0452] Step 5:

[0453] The server uses a generative AI model to generate music in real time based on musical characteristic data and emotional data. The generative AI model dynamically adjusts tempo, key, rhythm, etc. to generate accompaniment data. Input: Music characteristic data and emotional data. Output: Generated accompaniment data.

[0454] Step 6:

[0455] The server sends the generated accompaniment data to the user's device. The device decodes the received accompaniment data and plays the audio in real time. This allows the user to perform additional music along with the generated music. Input: Accompaniment data. Output: Music played in real time.

[0456] Step 7:

[0457] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them as a single music file in cloud storage. Input: User's performance data and accompaniment data. Output: Music file saved in cloud storage.

[0458] Step 8:

[0459] Users can share the music files they create with other users using a dedicated interface on their device. Other users can view and play the shared music files and add their own parts. Input: Music files stored in cloud storage. Output: New music file containing the added parts.

[0460] The above is a concrete process flow for implementing this invention. Each step involves data input and output, and we provide a system that enables real-time music creation and collaboration based on the user's emotions.

[0461] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0462] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0463] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0464] [Second embodiment]

[0465] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0466] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0467] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0468] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0469] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0470] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0471] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0472] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0473] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0474] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0475] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0476] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0477] The present invention provides an online platform for creating music and collaborating with other users in real time. Specific embodiments for implementing the present invention are described below.

[0478] Overall system configuration

[0479] The system consists of the following components:

[0480] 1. User's Device

[0481] 2. Server

[0482] 3. Cloud Storage

[0483] Receiving voice input

[0484] Users use their own devices to input their musical instruments and vocals. A microphone is connected to the device, and the input sound is captured as digital data, allowing for real-time recording of the performance.

[0485] Sending audio data

[0486] The device then transmits the captured audio data to a server, and this transmission process occurs in real time over the Internet.

[0487] Analysis of audio data

[0488] The server analyzes the received audio data. Specifically, the data analysis module extracts musical characteristics such as tempo, pitch, and rhythm. The analysis results are used as the basis for the subsequent music generation process.

[0489] Real-time music generation

[0490] The server generates accompaniment data in real time based on the musical characteristics, and an AI algorithm generates accompaniment data that is adapted to the analysis results and the user's performance, and sends it to the device in the appropriate format.

[0491] User feedback

[0492] The device receives the generated accompaniment data and provides real-time feedback to the user, allowing the user to continue playing along with the generated music.

[0493] Recording and storage

[0494] The server simultaneously records the user's performance data and the generated accompaniment data, and stores the generated music data in cloud storage, allowing the user to access, edit, and re-share the music data later.

[0495] Music sharing and collaboration

[0496] Users have a dedicated interface to share the music they create with other users. This interface is located on the device and is easy to navigate. Other users can view and play the shared music and add their own new parts.

[0497] The server re-analyzes and generates new music data, and saves the updated music data in cloud storage, facilitating ongoing collaboration.

[0498] Specific examples

[0499] Example 1: User A holds a guitar and plays a chord progression into the device.

[0500] Device: Captures guitar audio data and sends it to the server.

[0501] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[0502] Terminal: Plays back the accompaniment data generated in real time to User A.

[0503] Example 2: User B adds a piano part.

[0504] User B inputs voice into the device and provides the piano part.

[0505] Device: Sends audio data to the server.

[0506] Server: Analyzes the piano part and integrates it into the existing music data.

[0507] This will enable advanced musical collaboration between users.

[0508] The processing flow will be explained below.

[0509] Step 1:

[0510] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[0511] Step 2:

[0512] The device temporarily stores the captured audio data in an internal buffer and then transmits it to a server via the Internet.

[0513] Step 3:

[0514] The server receives the voice data sent from the terminal and checks the format and integrity of the received data.

[0515] Step 4:

[0516] The server uses an audio data analysis module to extract musical characteristics such as tempo, pitch, rhythm, etc. The results of this analysis are stored in a database.

[0517] Step 5:

[0518] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics, and this accompaniment data is generated in a format that adapts to the user's input.

[0519] Step 6:

[0520] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[0521] Step 7:

[0522] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0523] Step 8:

[0524] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them in cloud storage as a single music file.

[0525] Step 9:

[0526] The user has an interface that allows them to view the recorded music files on their device and edit them if necessary.

[0527] Step 10:

[0528] Users can use a dedicated function to share edited music files with other users and publish the music files online.

[0529] Step 11:

[0530] The server distributes the published music files for other users to view, play, and add their own parts to.

[0531] Step 12:

[0532] Other users input new parts and send the audio data to the server.

[0533] Step 13:

[0534] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[0535] Step 14:

[0536] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[0537] These steps facilitate real-time music creation and collaboration.

[0538] Example 1

[0539] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0540] In today's world, music production requires advanced technology and equipment, making it difficult to generate music in real time and facilitate smooth collaboration between multiple users. There is a particular demand for real-time music generation and collaboration with other users while receiving immediate feedback, but few systems currently exist that enable this. Therefore, there is an urgent need to provide a system that allows users to easily generate music and collaborate with other users in real time.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0542] In this invention, the server includes means for receiving audio data and analyzing musical characteristics, means for generating music in real time based on the analysis results, and means for re-analyzing and generating new music data and saving the updated music data in cloud storage. This allows users to receive real-time feedback on their voice-input performance data along with accompaniment data, enabling smooth collaboration with other users.

[0543] "User" means an individual or organization that utilizes the system to input voice and perform music, create music, and collaborate.

[0544] "Terminal" means a device used by a user to perform voice input and has the function of capturing voice data and transmitting it to a server.

[0545] A "server" is a central processing unit that receives and analyzes audio data, and generates and manages music in real time.

[0546] "Audio data" refers to digital data of musical instrument sounds or vocals played by a user on a terminal.

[0547] "Music characteristics" refers to characteristic information such as tempo, pitch, and rhythm extracted from audio data.

[0548] The "analysis results" are data on musical characteristics extracted from the audio data.

[0549] "Generating music in real time" means instantly generating accompaniment data adapted to the user's performance based on the analysis results.

[0550] "Accompaniment data" refers to auxiliary music data such as drums and bass that is generated to match the user's performance.

[0551] "Cloud storage" is a service that stores and manages data on the Internet.

[0552] "Providing feedback" means playing the generated accompaniment data to the user in real time.

[0553] "Collaboration" is the activity of multiple users working together to create, edit, and share music.

[0554] "Adding a new part" means that another user adds a new instrument sound or vocals to existing music data.

[0555] "Updated music data" refers to the latest music data that has new parts added and has been re-analyzed and re-generated.

[0556] The present invention provides a system that allows a user to input voice, generate music in real time, and collaborate with other users. Specific embodiments for carrying out the present invention are described below.

[0557] Overall system configuration

[0558] The system consists of the following components:

[0559] 1. User's Device

[0560] 2. Server

[0561] 3. Cloud Storage

[0562] Receiving voice input

[0563] Users use their own devices to input their musical instruments or vocals. A microphone is connected to the device, and the input sound is captured as digital data. This allows for real-time recording of performances. For example, if a user plays a chord progression on a guitar, the sound is captured by the microphone.

[0564] Sending audio data

[0565] The device transmits the captured audio data to the server. This transmission process is performed in real time via the Internet. For example, as soon as the user starts playing, the device collects audio data in real time and transmits it to the server.

[0566] Analysis of audio data

[0567] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. This analysis module extracts musical characteristics such as tempo (e.g., 120 BPM), pitch (e.g., key of C), and rhythm. These characteristics are the information necessary for subsequent AI music generation. For example, the server determines from the audio data that the tempo is 120 BPM and the key is C.

[0568] Real-time music generation

[0569] The server generates accompaniment data in real time based on the analysis results. Using a generative AI model, it generates accompaniment data adapted to the user's performance. For example, it generates drum and bass accompaniments based on guitar audio data. The generated data is converted into an appropriate format and sent to the device.

[0570] User feedback

[0571] The device receives the accompaniment data sent from the server. The received data is fed back to the user in real time. This allows the user to continue playing with the accompaniment added to their own performance. For example, drums and bass can be played simultaneously along with the guitar playing.

[0572] Recording and storage

[0573] The server simultaneously records the user's performance data and the generated accompaniment data. The recorded music data is stored in cloud storage, where users can access, edit, and reshare it later. This allows the music data to remain available even after the session has ended.

[0574] Music sharing and collaboration

[0575] Users have an interface to share the music they create with other users. The shared music can be viewed and played by other users, and new parts can be added. For example, User A can create a guitar accompaniment and User B can add a piano part.

[0576] Reparsing and updating

[0577] The server then re-analyzes the newly added music data and makes any necessary updates. It then integrates the new music data with the existing data using a generative AI model and saves the updated music data back to cloud storage, enabling continuous musical collaboration between users.

[0578] Specific examples

[0579] Example 1:

[0580] User A picks up a guitar and plays a chord progression into the terminal.

[0581] Device: Captures guitar audio data and sends it to the server.

[0582] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[0583] Terminal: Plays back the accompaniment data generated in real time to User A.

[0584] Example 2:

[0585] User B adds a piano part.

[0586] User B: Provides the piano part by dictating the song on the device.

[0587] Device: Sends audio data to the server.

[0588] Server: Analyzes piano parts and integrates them into existing music data, enabling advanced musical collaboration between users.

[0589] Prompt Sentence Examples

[0590] "Enter your guitar chord progression and generate accompaniment in real time."

[0591] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0592] Step 1:

[0593] User voice input

[0594] Users use their devices to input musical instruments or vocals. A microphone is used as an input device, and the microphone captures audio data as digital data. For example, when a user plays a guitar, the microphone collects the audio and converts it into digital data.

[0595] Input: User's voice

[0596] Output: Digitized audio data

[0597] Step 2:

[0598] Device capture and transmission of audio data

[0599] The device captures digitized audio data and transmits it over the internet to a server. This process happens in real time, with high speeds to minimize latency. For example, after a user plays the guitar, the audio data is sent instantly to the server.

[0600] Input: Digitized audio data

[0601] Output: Audio data sent to the server

[0602] Step 3:

[0603] Receiving and analyzing voice data by the server

[0604] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. Specifically, musical characteristics such as tempo, pitch, and rhythm are extracted. For example, the server analyzes the audio data and determines that the tempo is 120 BPM and the key is C.

[0605] Input: Audio data sent to the server

[0606] Output: Extracted musical characteristics data (tempo, pitch, rhythm)

[0607] Step 4:

[0608] Real-time music generation by server

[0609] The server uses a generative AI model to generate accompaniment data in real time based on the analyzed musical characteristic data. This AI model creates drum and bass accompaniment that is adapted to the user's performance. For example, it generates drum and bass accompaniment for a 120 BPM song in the key of C. The generated accompaniment data is converted into an appropriate format and sent to the device.

[0610] Input: Musical characteristics data (tempo, pitch, rhythm)

[0611] Output: Generated accompaniment data

[0612] Step 5:

[0613] Sending feedback to the device

[0614] The device receives the accompaniment data sent from the server. The received accompaniment data is played back in real time, providing feedback to the user. This allows the user to use the accompaniment generated to match their own performance. For example, a user can play the guitar while simultaneously listening to the generated drum and bass accompaniment.

[0615] Input: Generated accompaniment data

[0616] Output: Real-time feedback sound to the user

[0617] Step 6:

[0618] Server-based recording and storage

[0619] The server simultaneously records the user's performance data and the generated accompaniment data. This recording data is stored in cloud storage. Users can later access, edit, and reshare this data. For example, users can review and edit the recording data after the session has ended.

[0620] Input: User's performance data, generated accompaniment data

[0621] Output: Recording data saved in cloud storage

[0622] Step 7:

[0623] Users can share and collaborate on music

[0624] Users share their music with other users using a dedicated interface that is intuitive and easy to use. Other users can browse, play, and add new parts to the shared music. For example, user B can add a piano part to a song created by user A.

[0625] Input: User-created music data

[0626] Output: Share and collaborate with others

[0627] Step 8:

[0628] Server reparsing and updating

[0629] The server then re-analyzes the newly added music data and makes any necessary updates. It then uses a generative AI model to integrate the new music data with existing data and saves the updated music data in cloud storage. For example, it can analyze newly added parts and update the entire song, promoting collaboration.

[0630] Input: Newly added music data

[0631] Output: Updated music data (stored in cloud storage)

[0632] (Application example 1)

[0633] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0634] Conventional factory robot control systems require complex programming and settings, making efficient operation difficult. Furthermore, it is extremely difficult to achieve real-time collaborative work between multiple robots, often resulting in a loss of efficiency and safety. The present invention aims to solve these problems by realizing simple and efficient robot control and collaborative work using voice input.

[0635] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0636] In this invention, the server includes means for a user to input voice, means for the server to receive the voice data and analyze musical characteristics, means for the server to generate music in real time based on the analysis results, means for transmitting the generated music to the user's terminal, means for the user to share the music created by the user with other users, means for other users to add new parts, means for analyzing the voice input and generating operation commands for various devices, and means for transmitting the generated operation commands to industrial devices and causing them to execute them. This enables simple and efficient robot control using voice input and cooperative work between multiple robots.

[0637] "User" means any person or group of people who uses the online platform to provide voice input.

[0638] "Voice input" is the act or process of capturing user-uttered speech as digital data.

[0639] The "server" is a central processing device that receives and analyzes audio data, and generates music and motion commands.

[0640] "Voice data" refers to digitized data of voice input by a user.

[0641] "Musical characteristics" are musical elements such as tempo, pitch, and rhythm extracted from audio data.

[0642] "Real-time" means that data is sent, received, and processed nearly simultaneously.

[0643] "Music generation" is the process of creating new music based on analyzed musical characteristics.

[0644] A "terminal" is an electronic device used by a user (such as a smartphone, tablet, or PC).

[0645] "Sharing" is the process by which a user shares music or data they have created with other users.

[0646] A "robot" is an automated mechanical device that performs tasks in a factory.

[0647] An "operation command" is an instruction to operate various devices that is generated based on the analyzed voice input.

[0648] The present invention is a system that generates operational commands for music and industrial equipment in real time through user voice input, enabling efficient and rapid collaboration.

[0649] Overall system configuration

[0650] The system consists of the following components:

[0651] 1. User's Device

[0652] 2. Server

[0653] 3. Cloud Storage

[0654] 4. Industrial equipment (robots)

[0655] Voice input and analysis

[0656] The user inputs voice. The voice is captured as digital data using a microphone connected to the device. This voice data is sent from the device to a server. The server analyzes the received voice data and extracts musical characteristics such as tempo, pitch, and rhythm. The server also analyzes the instructions the user gives to industrial equipment.

[0657] Music Generation and Feedback

[0658] The server generates accompaniment data in real time based on the musical characteristics, and the generated music data is sent to the user's device and fed back in real time, allowing the user to continue playing along with the generated music.

[0659] Robot motion command generation and execution

[0660] The server analyzes the user's voice input and generates operational commands for the industrial equipment, which are then sent over the network to the industrial equipment, causing the robot to instantly perform the required action.

[0661] Recording and storage

[0662] The server simultaneously records the user's performance data, the generated accompaniment data, and the generated movement command data, and saves them in cloud storage, allowing the user to access, edit, and re-share the data later.

[0663] Music sharing and collaboration

[0664] Users can use a dedicated interface to share the music they create with other users, who can then view and play the music, add new parts, and share operating instructions for industrial equipment.

[0665] Hardware and software used

[0666] Hardware:

[0667] Microphone (for voice input)

[0668] Industrial equipment (e.g. robotic arms)

[0669] software:

[0670] Python (for processing and server communication)

[0671] sounddevice module (audio recording)

[0672] requests module (sending HTTP requests)

[0673] AI voice analysis server (voice analysis and command generation)

[0674] Specific examples

[0675] Example 1:

[0676] A worker in a factory commands a robot to "go get the next part." A microphone captures this voice and sends it as audio data to a server. The server analyzes the voice and generates an action command to "go get the part." The generated action command is sent to the robot, which then goes to get the part.

[0677] Example 2:

[0678] User A picks up a guitar and plays a chord progression into the device. This performance data is sent to the server, which analyzes it to determine that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment, which is then fed back to User A in real time.

[0679] Prompt Sentence Examples

[0680] "Voice command to the robot: Go get the next part."

[0681] "Pick up the parts from the conveyor"

[0682] "Move to the assembly area."

[0683] The above is an embodiment of the present invention.

[0684] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0685] Step 1:

[0686] The user provides voice input. The user issues a voice command using a microphone connected to the device. The device captures this voice as digital data. The input is the user's voice, and the output is digitized voice data. This digital data is analyzed in the next step.

[0687] Step 2:

[0688] The device sends the captured audio data to the server. The device transfers the audio data to the server in real time via the Internet. The input is the audio data captured by the device, and the output is the data sent to the server.

[0689] Step 3:

[0690] The server analyzes the received audio data. Using an audio analysis module, the server extracts musical characteristics such as tempo, pitch, and rhythm from the data. It also analyzes operational commands for industrial equipment. The input is the audio data sent to the server, and the output is the extracted musical characteristics and operational commands.

[0691] Step 4:

[0692] The server generates music in real time based on the analysis results. The server uses an AI algorithm to generate accompaniment data based on the extracted musical characteristics. The input is the extracted musical characteristics, and the output is the generated accompaniment data.

[0693] Step 5:

[0694] The generated music is sent to the user's device. The server sends the generated accompaniment data to the user's device in real time. The input is the generated accompaniment data, and the output is the accompaniment data sent to the user's device.

[0695] Step 6:

[0696] The server generates operation commands for industrial equipment based on the analysis results. The server generates appropriate operation commands based on the voice analysis results and sends them to the industrial equipment via the network. The input is the voice analysis results, and the output is the generated operation commands.

[0697] Step 7:

[0698] Industrial equipment performs tasks according to operation commands. Industrial equipment (such as robots) performs specified operations according to operation commands sent from the server. The input is the operation command sent from the server, and the output is the operation of the industrial equipment.

[0699] Step 8:

[0700] The server simultaneously records the user's performance data and the generated accompaniment data, and stores them in cloud storage. The input is the user's performance data and the generated accompaniment data, and the output is the data stored in cloud storage.

[0701] Step 9:

[0702] A user can share music they have created with other users. The user's device uses a dedicated interface to share the music they have created with other users. The input is data stored in cloud storage, and the output is the shared music data.

[0703] Step 10:

[0704] Another user adds a new part. Another user adds a new part to the shared music data and sends it back to the system. The input is the shared music data, and the output is the music data with the new part added.

[0705] The above are the specific processing steps of the program of the present invention.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] The present invention provides an online platform that combines an emotion engine that recognizes a user's emotions to generate music in real time according to the user's emotional state, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[0708] Overall system configuration

[0709] The system consists of the following components:

[0710] 1. User's Device

[0711] 2. Server

[0712] 3. Cloud Storage

[0713] 4. Emotion Engine

[0714] Receiving voice input and analyzing emotions

[0715] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[0716] Sending voice data and emotion data

[0717] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[0718] Analysis of voice data and use of emotion data

[0719] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the AI ​​algorithm.

[0720] Real-time music generation and emotional reflection

[0721] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[0722] User feedback

[0723] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0724] Recording and storage

[0725] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0726] Music sharing and collaboration

[0727] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[0728] Specific examples

[0729] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[0730] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[0731] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[0732] Terminal: Plays back the accompaniment data generated in real time to User A.

[0733] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[0734] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[0735] Terminal: Sends voice data and emotion data to the server.

[0736] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[0737] In this way, music creation and collaboration that reflects the user's emotional state becomes possible.

[0738] The processing flow will be explained below.

[0739] Step 1:

[0740] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[0741] Step 2:

[0742] The terminal temporarily stores the voice data in a buffer and simultaneously transmits this data to the emotion engine.

[0743] Step 3:

[0744] The emotion engine analyzes the received voice data to determine the user's emotional state, for example, generating emotion categories such as "joy," "sadness," and "anger."

[0745] Step 4:

[0746] The device transmits the captured voice data and emotion data obtained from the emotion engine to a server in real time via the Internet.

[0747] Step 5:

[0748] The server receives the voice data and emotion data sent from the device and checks the format and integrity of the received data.

[0749] Step 6:

[0750] The server analyzes the audio data and extracts musical characteristics such as tempo, pitch, and rhythm, and stores the analysis results in a database.

[0751] Step 7:

[0752] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics and emotional data. The tempo, key, rhythm, and other aspects of the generated music are dynamically adjusted according to the emotional data.

[0753] Step 8:

[0754] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[0755] Step 9:

[0756] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0757] Step 10:

[0758] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them in cloud storage as a single music file.

[0759] Step 11:

[0760] The user has an interface to review the recorded music files from their device and edit them if necessary.

[0761] Step 12:

[0762] Users can use a dedicated function to share edited music files with other users and publish them online.

[0763] Step 13:

[0764] The server distributes the published music files for other users to view, play, and add their own parts to.

[0765] Step 14:

[0766] Other users input new parts and send the audio data to the server.

[0767] Step 15:

[0768] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[0769] Step 16:

[0770] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[0771] These steps facilitate real-time music generation that reflects emotional states and collaboration between users.

[0772] Example 2

[0773] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0774] Conventional music generation systems have difficulty generating music that reflects a user's emotional state in real time, and sharing and collaboration between users is also limited. To solve these problems, an effective method is needed to analyze emotions, generate music based on that emotion in real time, and share and collaborate with other users.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving captured audio data and emotional data and analyzing musical characteristics and emotional states, means for generating music in real time based on the analysis results and emotional data, and means for transmitting the generated music to the user's terminal. This enables the generation of music that reflects emotions in real time, as well as effective sharing and collaboration between users.

[0776] "User" refers to a person who uses a terminal to input voice data and use the system.

[0777] A "terminal" is a hardware device that allows a user to input voice information, and includes devices such as microphones, computers, and smartphones.

[0778] "Voice data" refers to data that represents in digital form the voice signal input by a user through a terminal.

[0779] "Emotional data" refers to information about the user's emotional state obtained by analyzing voice data.

[0780] "Server" refers to a computer system for receiving and analyzing audio and emotional data and generating music.

[0781] "Musical characteristics" refers to musical elements such as tempo, pitch, and rhythm obtained by analyzing audio data.

[0782] "Real-time" means that user input is processed immediately, without delay.

[0783] "Music generation" refers to the process of creating new music based on audio and emotional data.

[0784] "Sending to terminal" refers to sending the music data generated by the server to the user's terminal via the network.

[0785] "Dedicated Interface" refers to the dedicated operating screens and tools used by users to share and collaborate on music they create with other users.

[0786] "Cloud storage" refers to online storage services for storing and accessing data over the Internet.

[0787] "Adding a new part" refers to another user adding new audio data or instrument parts to an existing music file.

[0788] This invention provides an online platform that generates music in real time according to the user's emotional state by combining an emotion engine that recognizes the user's emotions, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[0789] Overall system configuration

[0790] The system consists of the following components:

[0791] 1. User's Device

[0792] 2. Server

[0793] 3. Cloud Storage

[0794] 4. Emotion Engine

[0795] Receiving voice input and analyzing emotions

[0796] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[0797] Sending voice data and emotion data

[0798] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[0799] Analysis of voice data and use of emotion data

[0800] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the generative AI model.

[0801] Real-time music generation and emotional reflection

[0802] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[0803] User feedback

[0804] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0805] Recording and storage

[0806] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0807] Music sharing and collaboration

[0808] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[0809] Specific examples

[0810] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[0811] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[0812] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[0813] Terminal: Plays back the accompaniment data generated in real time to User A.

[0814] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[0815] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[0816] Terminal: Sends voice data and emotion data to the server.

[0817] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[0818] Prompt Sentence Examples

[0819] You can check the system's operation by inputting the following prompt sentences into the generative AI model.

[0820] When a user plays the guitar and has the emotion "joy", send the audio data and emotion data to the server to generate a cheerful accompaniment in the key of C at 120 BPM in real time.

[0821] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0822] Step 1: User speaks

[0823] Users input musical instruments or vocals using a microphone connected to their device. The microphone connected to the device captures the audio signal and converts it into digital data. The input is the user's voice, and the output is digitally converted audio data.

[0824] Step 2: Sentiment Analysis

[0825] The device passes the voice data captured in step 1 to the emotion engine. The emotion engine analyzes the tone, pitch, tempo, etc. of the voice data to identify the user's emotional state (e.g., "joy," "sadness," etc.). The input is the digital voice data obtained in step 1, and the output is emotion data resulting from the analysis.

[0826] Step 3: Sending voice and emotion data

[0827] The device transmits the emotion data and voice data from step 2 to the server. The transmission process is carried out in real time via the Internet. The input is the voice data and emotion data, and the output is a confirmation message that the server has received the data.

[0828] Step 4: Receiving and analyzing voice and emotion data

[0829] The server receives the audio data and emotion data sent in step 3. After receiving the audio data, it checks the format and integrity of the audio data and analyzes the musical characteristics (e.g., tempo, pitch, rhythm). The input is the received audio data and emotion data, and the output is the analyzed musical characteristic data and emotion data.

[0830] Step 5: Real-time music generation

[0831] The server uses a generative AI model to generate music in real time based on the analyzed musical characteristic data and emotional data. The tempo, key, rhythm, etc. are dynamically adjusted based on the emotional data. The input is the musical characteristic data and emotional data, and the output is the generated accompaniment data.

[0832] Step 6: Send the generated music data to your device

[0833] The server converts the generated accompaniment data into an appropriate format in real time and sends it to the user's terminal. The input is the generated accompaniment data, and the output is the accompaniment data received by the user's terminal.

[0834] Step 7: User feedback and real-time playback

[0835] The device decodes the received accompaniment data and plays it back to the user in real time. The user can then perform additional music along with the accompaniment. The input is the received accompaniment data, and the output is music played back to the user in real time.

[0836] Step 8: Record and save your performance data

[0837] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotion data, and stores them as a single music file in cloud storage. The input is the user's performance data, the generated accompaniment data, and the emotion data, and the output is the music file stored in cloud storage.

[0838] Step 9: Share and collaborate on music

[0839] Users use a dedicated interface to share music files stored in cloud storage with other users. Other users can browse, play, and add their own parts to the shared music files. The input is the music file retrieved from cloud storage, and the output is the new music file that has been shared and edited.

[0840] Example operation

[0841] As a concrete example, if User A plays the guitar and the emotion engine analyzes that his / her emotional state is "joy," the device sends this to the server. The server receives the data, generates a cheerful accompaniment with a tempo of 120 BPM and the key of C, and sends it to the device. The device plays it in real time, and User A continues playing along with the accompaniment. If User B later wants to add his / her own piano part, he / she retrieves the music file from cloud storage and plays it, reflecting the emotional data of "sadness." The server integrates the data and generates a new music file that reflects the emotion.

[0842] (Application example 2)

[0843] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0844] In recent years, there has been a demand for music generation that reflects a user's emotions, but there is a lack of technology that can generate music in real time based on emotions and share and collaborate with other users. Therefore, providing a system that can analyze a user's emotions, generate music based on those emotions in real time, and make it easy to share with other users has become an important challenge.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0846] In this invention, the server includes means for analyzing a user's emotions and dynamically generating music based on the results, means for dynamically adjusting the tempo, key, and rhythm of the generated music, and means for providing feedback on the generated music to the user in real time and allowing the user to perform additional performances. This enables music to be generated in real time according to the user's emotions, and allows easy sharing and collaboration with other users.

[0847] A "terminal" is a device that allows a user to input voice information, including smartphones, tablets, and personal computers.

[0848] "Voice data" is digital sound information acquired when a user makes a voice input.

[0849] "Musical characteristics" refers to musical features such as tempo, pitch, and rhythm extracted from audio data.

[0850] A "server" is a high performance computing device located at a remote location that receives audio data and performs analysis and music generation.

[0851] "Real-time" refers to a time range in which processing occurs so quickly that there is almost no noticeable delay from user input.

[0852] "Accompaniment data" refers to the accompaniment portion of the music generated based on the user's voice data and emotion data.

[0853] "Cloud storage" is a remote storage service for storing data on the Internet.

[0854] "Emotion data" is information that indicates the emotional state of the user analyzed from the user's voice data.

[0855] "Generative AI model" refers to an algorithm or program for generating music using artificial intelligence techniques.

[0856] A "prompt" refers to an instruction or question that is input into an AI model.

[0857] "Collaboration" refers to multiple users working together to create a single musical work.

[0858] This invention is a system that analyzes a user's emotions, generates music in real time based on the results, and provides an online platform for sharing and collaborating with other users.

[0859] Overall system configuration

[0860] The system consists of the following components:

[0861] 1. User device: The device through which the user inputs voice and transmits emotion data. This includes smartphones, tablets, and PCs.

[0862] 2. Server: A high-performance computer device installed in a remote location that receives audio data and emotion data, analyzes them, and generates music.

[0863] 3. Cloud storage: A remote storage service on the Internet for storing generated music data.

[0864] 4. Emotion engine: An engine that contains algorithms that analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Azure Cognitive Services.

[0865] Receiving voice input and analyzing emotions

[0866] Users input musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine receives the same audio input data and analyzes the user's emotional state. For example, if the user is feeling "sad," the system will recognize that emotion.

[0867] Sending voice and emotion data

[0868] The device transmits the captured voice data and analyzed emotion data to a server. This transmission process is carried out in real time via the Internet. The voice data and emotion data are transmitted together and checked for data integrity.

[0869] Analysis of voice data and use of emotion data

[0870] The server receives the audio data and emotion data sent from the device. The received audio data is used to analyze musical characteristics, extracting information such as tempo, pitch, and rhythm. The emotion data is used as input for the AI ​​algorithm and reflected in the generated music.

[0871] Real-time music generation and emotional reflection

[0872] The server generates accompaniment data in real time based on musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. For example, if the emotional engine analyzes "joy," it will generate upbeat, fast-paced music.

[0873] User feedback

[0874] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional parts along with the accompaniment, thereby realizing a musical experience that is in line with the user's emotions.

[0875] Recording and storage

[0876] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[0877] Music sharing and collaboration

[0878] Users use a dedicated interface to share the music files they create with other users. This interface is located on the device and is easy to operate. Other users can view and play the shared music files and add their own parts. For example, if user A starts playing the piano in a lonely mood, other users can add their own instruments to the music and collaborate.

[0879] Prompt example

[0880] "Use the Emotion Music Real-time Generator (EMRG) to express your lonely feelings on piano, then collaborate with other users by adding their own instruments to your music."

[0881] This concludes the detailed description of the "Mode for carrying out the invention." The entire system cooperates to automatically create, share, and collaborate on music that is in line with the user's emotions.

[0882] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0883] Step 1:

[0884] The user performs voice input. Specifically, the user inputs an instrument or vocals into a device such as a smartphone, tablet, or PC. At this time, the microphone connected to the device captures the audio data in digital format. Input: User's voice. Output: Digital audio data.

[0885] Step 2:

[0886] The device sends the captured voice data to the emotion engine, which uses algorithms such as IBM Watson Tone Analyzer and Azure Cognitive Services to analyze the voice data and identify the user's emotional state. Input: Digital voice data. Output: Analyzed emotion data.

[0887] Step 3:

[0888] The device transmits the analyzed emotion data and voice data to the server in real time via the Internet. The server checks the integrity of the received data. Input: Voice data and emotion data. Output: Data packets (voice data and emotion data).

[0889] Step 4:

[0890] The server analyzes the received audio data to extract musical characteristics. This process uses algorithms to identify information such as tempo, pitch, and rhythm. Input: Audio data. Output: Musical characteristics data.

[0891] Step 5:

[0892] The server uses a generative AI model to generate music in real time based on musical characteristic data and emotional data. The generative AI model dynamically adjusts tempo, key, rhythm, etc. to generate accompaniment data. Input: Music characteristic data and emotional data. Output: Generated accompaniment data.

[0893] Step 6:

[0894] The server sends the generated accompaniment data to the user's device. The device decodes the received accompaniment data and plays the audio in real time. This allows the user to perform additional music along with the generated music. Input: Accompaniment data. Output: Music played in real time.

[0895] Step 7:

[0896] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them as a single music file in cloud storage. Input: User's performance data and accompaniment data. Output: Music file saved in cloud storage.

[0897] Step 8:

[0898] Users can share the music files they create with other users using a dedicated interface on their device. Other users can view and play the shared music files and add their own parts. Input: Music files stored in cloud storage. Output: New music file containing the added parts.

[0899] The above is a concrete process flow for implementing this invention. Each step involves data input and output, and we provide a system that enables real-time music creation and collaboration based on the user's emotions.

[0900] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0901] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0902] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0903] [Third embodiment]

[0904] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0905] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0906] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0907] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0908] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0909] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0910] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0911] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0912] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0913] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0914] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0915] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0916] The present invention provides an online platform for creating music and collaborating with other users in real time. Specific embodiments for implementing the present invention are described below.

[0917] Overall system configuration

[0918] The system consists of the following components:

[0919] 1. User's Device

[0920] 2. Server

[0921] 3. Cloud Storage

[0922] Receiving voice input

[0923] Users use their own devices to input their musical instruments and vocals. A microphone is connected to the device, and the input sound is captured as digital data, allowing for real-time recording of the performance.

[0924] Sending audio data

[0925] The device then transmits the captured audio data to a server, and this transmission process occurs in real time over the Internet.

[0926] Analysis of audio data

[0927] The server analyzes the received audio data. Specifically, the data analysis module extracts musical characteristics such as tempo, pitch, and rhythm. The analysis results are used as the basis for the subsequent music generation process.

[0928] Real-time music generation

[0929] The server generates accompaniment data in real time based on the musical characteristics, and an AI algorithm generates accompaniment data that is adapted to the analysis results and the user's performance, and sends it to the device in the appropriate format.

[0930] User feedback

[0931] The device receives the generated accompaniment data and provides real-time feedback to the user, allowing the user to continue playing along with the generated music.

[0932] Recording and storage

[0933] The server simultaneously records the user's performance data and the generated accompaniment data, and stores the generated music data in cloud storage, allowing the user to access, edit, and re-share the music data later.

[0934] Music sharing and collaboration

[0935] Users have a dedicated interface to share the music they create with other users. This interface is located on the device and is easy to navigate. Other users can view and play the shared music and add their own new parts.

[0936] The server re-analyzes and generates new music data, and saves the updated music data in cloud storage, facilitating ongoing collaboration.

[0937] Specific examples

[0938] Example 1: User A holds a guitar and plays a chord progression into the device.

[0939] Device: Captures guitar audio data and sends it to the server.

[0940] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[0941] Terminal: Plays back the accompaniment data generated in real time to User A.

[0942] Example 2: User B adds a piano part.

[0943] User B inputs voice into the device and provides the piano part.

[0944] Device: Sends audio data to the server.

[0945] Server: Analyzes the piano part and integrates it into the existing music data.

[0946] This will enable advanced musical collaboration between users.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[0950] Step 2:

[0951] The device temporarily stores the captured audio data in an internal buffer and then transmits it to a server via the Internet.

[0952] Step 3:

[0953] The server receives the voice data sent from the terminal and checks the format and integrity of the received data.

[0954] Step 4:

[0955] The server uses an audio data analysis module to extract musical characteristics such as tempo, pitch, rhythm, etc. The results of this analysis are stored in a database.

[0956] Step 5:

[0957] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics, and this accompaniment data is generated in a format that adapts to the user's input.

[0958] Step 6:

[0959] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[0960] Step 7:

[0961] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[0962] Step 8:

[0963] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them in cloud storage as a single music file.

[0964] Step 9:

[0965] The user has an interface that allows them to view the recorded music files on their device and edit them if necessary.

[0966] Step 10:

[0967] Users can use a dedicated function to share edited music files with other users and publish the music files online.

[0968] Step 11:

[0969] The server distributes the published music files for other users to view, play, and add their own parts to.

[0970] Step 12:

[0971] Other users input new parts and send the audio data to the server.

[0972] Step 13:

[0973] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[0974] Step 14:

[0975] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[0976] These steps facilitate real-time music creation and collaboration.

[0977] Example 1

[0978] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0979] In today's world, music production requires advanced technology and equipment, making it difficult to generate music in real time and facilitate smooth collaboration between multiple users. There is a particular demand for real-time music generation and collaboration with other users while receiving immediate feedback, but few systems currently exist that enable this. Therefore, there is an urgent need to provide a system that allows users to easily generate music and collaborate with other users in real time.

[0980] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0981] In this invention, the server includes means for receiving audio data and analyzing musical characteristics, means for generating music in real time based on the analysis results, and means for re-analyzing and generating new music data and saving the updated music data in cloud storage. This allows users to receive real-time feedback on their voice-input performance data along with accompaniment data, enabling smooth collaboration with other users.

[0982] "User" means an individual or organization that utilizes the system to input voice and perform music, create music, and collaborate.

[0983] "Terminal" means a device used by a user to perform voice input and has the function of capturing voice data and transmitting it to a server.

[0984] A "server" is a central processing unit that receives and analyzes audio data, and generates and manages music in real time.

[0985] "Audio data" refers to digital data of musical instrument sounds or vocals played by a user on a terminal.

[0986] "Music characteristics" refers to characteristic information such as tempo, pitch, and rhythm extracted from audio data.

[0987] The "analysis results" are data on musical characteristics extracted from the audio data.

[0988] "Generating music in real time" means instantly generating accompaniment data adapted to the user's performance based on the analysis results.

[0989] "Accompaniment data" refers to auxiliary music data such as drums and bass that is generated to match the user's performance.

[0990] "Cloud storage" is a service that stores and manages data on the Internet.

[0991] "Providing feedback" means playing the generated accompaniment data to the user in real time.

[0992] "Collaboration" is the activity of multiple users working together to create, edit, and share music.

[0993] "Adding a new part" means that another user adds a new instrument sound or vocals to existing music data.

[0994] "Updated music data" refers to the latest music data that has new parts added and has been re-analyzed and re-generated.

[0995] The present invention provides a system that allows a user to input voice, generate music in real time, and collaborate with other users. Specific embodiments for carrying out the present invention are described below.

[0996] Overall system configuration

[0997] The system consists of the following components:

[0998] 1. User's Device

[0999] 2. Server

[1000] 3. Cloud Storage

[1001] Receiving voice input

[1002] Users use their own devices to input their musical instruments or vocals. A microphone is connected to the device, and the input sound is captured as digital data. This allows for real-time recording of performances. For example, if a user plays a chord progression on a guitar, the sound is captured by the microphone.

[1003] Sending audio data

[1004] The device transmits the captured audio data to the server. This transmission process is performed in real time via the Internet. For example, as soon as the user starts playing, the device collects audio data in real time and transmits it to the server.

[1005] Analysis of audio data

[1006] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. This analysis module extracts musical characteristics such as tempo (e.g., 120 BPM), pitch (e.g., key of C), and rhythm. These characteristics are the information necessary for subsequent AI music generation. For example, the server determines from the audio data that the tempo is 120 BPM and the key is C.

[1007] Real-time music generation

[1008] The server generates accompaniment data in real time based on the analysis results. Using a generative AI model, it generates accompaniment data adapted to the user's performance. For example, it generates drum and bass accompaniments based on guitar audio data. The generated data is converted into an appropriate format and sent to the device.

[1009] User feedback

[1010] The device receives the accompaniment data sent from the server. The received data is fed back to the user in real time. This allows the user to continue playing with the accompaniment added to their own performance. For example, drums and bass can be played simultaneously along with the guitar playing.

[1011] Recording and storage

[1012] The server simultaneously records the user's performance data and the generated accompaniment data. The recorded music data is stored in cloud storage, where users can access, edit, and reshare it later. This allows the music data to remain available even after the session has ended.

[1013] Music sharing and collaboration

[1014] Users have an interface to share the music they create with other users. The shared music can be viewed and played by other users, and new parts can be added. For example, User A can create a guitar accompaniment and User B can add a piano part.

[1015] Reparsing and updating

[1016] The server then re-analyzes the newly added music data and makes any necessary updates. It then integrates the new music data with the existing data using a generative AI model and saves the updated music data back to cloud storage, enabling continuous musical collaboration between users.

[1017] Specific examples

[1018] Example 1:

[1019] User A picks up a guitar and plays a chord progression into the terminal.

[1020] Device: Captures guitar audio data and sends it to the server.

[1021] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[1022] Terminal: Plays back the accompaniment data generated in real time to User A.

[1023] Example 2:

[1024] User B adds a piano part.

[1025] User B: Provides the piano part by dictating the song on the device.

[1026] Device: Sends audio data to the server.

[1027] Server: Analyzes piano parts and integrates them into existing music data, enabling advanced musical collaboration between users.

[1028] Prompt Sentence Examples

[1029] "Enter your guitar chord progression and generate accompaniment in real time."

[1030] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1031] Step 1:

[1032] User voice input

[1033] Users use their devices to input musical instruments or vocals. A microphone is used as an input device, and the microphone captures audio data as digital data. For example, when a user plays a guitar, the microphone collects the audio and converts it into digital data.

[1034] Input: User's voice

[1035] Output: Digitized audio data

[1036] Step 2:

[1037] Device capture and transmission of audio data

[1038] The device captures digitized audio data and transmits it over the internet to a server. This process happens in real time, with high speeds to minimize latency. For example, after a user plays the guitar, the audio data is sent instantly to the server.

[1039] Input: Digitized audio data

[1040] Output: Audio data sent to the server

[1041] Step 3:

[1042] Receiving and analyzing voice data by the server

[1043] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. Specifically, musical characteristics such as tempo, pitch, and rhythm are extracted. For example, the server analyzes the audio data and determines that the tempo is 120 BPM and the key is C.

[1044] Input: Audio data sent to the server

[1045] Output: Extracted musical characteristics data (tempo, pitch, rhythm)

[1046] Step 4:

[1047] Real-time music generation by server

[1048] The server uses a generative AI model to generate accompaniment data in real time based on the analyzed musical characteristic data. This AI model creates drum and bass accompaniment that is adapted to the user's performance. For example, it generates drum and bass accompaniment for a 120 BPM song in the key of C. The generated accompaniment data is converted into an appropriate format and sent to the device.

[1049] Input: Musical characteristics data (tempo, pitch, rhythm)

[1050] Output: Generated accompaniment data

[1051] Step 5:

[1052] Sending feedback to the device

[1053] The device receives the accompaniment data sent from the server. The received accompaniment data is played back in real time, providing feedback to the user. This allows the user to use the accompaniment generated to match their own performance. For example, a user can play the guitar while simultaneously listening to the generated drum and bass accompaniment.

[1054] Input: Generated accompaniment data

[1055] Output: Real-time feedback sound to the user

[1056] Step 6:

[1057] Server-based recording and storage

[1058] The server simultaneously records the user's performance data and the generated accompaniment data. This recording data is stored in cloud storage. Users can later access, edit, and reshare this data. For example, users can review and edit the recording data after the session has ended.

[1059] Input: User's performance data, generated accompaniment data

[1060] Output: Recording data saved in cloud storage

[1061] Step 7:

[1062] Users can share and collaborate on music

[1063] Users share their music with other users using a dedicated interface that is intuitive and easy to use. Other users can browse, play, and add new parts to the shared music. For example, user B can add a piano part to a song created by user A.

[1064] Input: User-created music data

[1065] Output: Share and collaborate with others

[1066] Step 8:

[1067] Server reparsing and updating

[1068] The server then re-analyzes the newly added music data and makes any necessary updates. It then uses a generative AI model to integrate the new music data with existing data and saves the updated music data in cloud storage. For example, it can analyze newly added parts and update the entire song, promoting collaboration.

[1069] Input: Newly added music data

[1070] Output: Updated music data (stored in cloud storage)

[1071] (Application example 1)

[1072] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] Conventional factory robot control systems require complex programming and settings, making efficient operation difficult. Furthermore, it is extremely difficult to achieve real-time collaborative work between multiple robots, often resulting in a loss of efficiency and safety. The present invention aims to solve these problems by realizing simple and efficient robot control and collaborative work using voice input.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1075] In this invention, the server includes means for a user to input voice, means for the server to receive the voice data and analyze musical characteristics, means for the server to generate music in real time based on the analysis results, means for transmitting the generated music to the user's terminal, means for the user to share the music created by the user with other users, means for other users to add new parts, means for analyzing the voice input and generating operation commands for various devices, and means for transmitting the generated operation commands to industrial devices and causing them to execute them. This enables simple and efficient robot control using voice input and cooperative work between multiple robots.

[1076] "User" means any person or group of people who uses the online platform to provide voice input.

[1077] "Voice input" is the act or process of capturing user-uttered speech as digital data.

[1078] The "server" is a central processing device that receives and analyzes audio data, and generates music and motion commands.

[1079] "Voice data" refers to digitized data of voice input by a user.

[1080] "Musical characteristics" are musical elements such as tempo, pitch, and rhythm extracted from audio data.

[1081] "Real-time" means that data is sent, received, and processed nearly simultaneously.

[1082] "Music generation" is the process of creating new music based on analyzed musical characteristics.

[1083] A "terminal" is an electronic device used by a user (such as a smartphone, tablet, or PC).

[1084] "Sharing" is the process by which a user shares music or data they have created with other users.

[1085] A "robot" is an automated mechanical device that performs tasks in a factory.

[1086] An "operation command" is an instruction to operate various devices that is generated based on the analyzed voice input.

[1087] The present invention is a system that generates operational commands for music and industrial equipment in real time through user voice input, enabling efficient and rapid collaboration.

[1088] Overall system configuration

[1089] The system consists of the following components:

[1090] 1. User's Device

[1091] 2. Server

[1092] 3. Cloud Storage

[1093] 4. Industrial equipment (robots)

[1094] Voice input and analysis

[1095] The user inputs voice. The voice is captured as digital data using a microphone connected to the device. This voice data is sent from the device to a server. The server analyzes the received voice data and extracts musical characteristics such as tempo, pitch, and rhythm. The server also analyzes the instructions the user gives to industrial equipment.

[1096] Music Generation and Feedback

[1097] The server generates accompaniment data in real time based on the musical characteristics, and the generated music data is sent to the user's device and fed back in real time, allowing the user to continue playing along with the generated music.

[1098] Robot motion command generation and execution

[1099] The server analyzes the user's voice input and generates operational commands for the industrial equipment, which are then sent over the network to the industrial equipment, causing the robot to instantly perform the required action.

[1100] Recording and storage

[1101] The server simultaneously records the user's performance data, the generated accompaniment data, and the generated movement command data, and saves them in cloud storage, allowing the user to access, edit, and re-share the data later.

[1102] Music sharing and collaboration

[1103] Users can use a dedicated interface to share the music they create with other users, who can then view and play the music, add new parts, and share operating instructions for industrial equipment.

[1104] Hardware and software used

[1105] Hardware:

[1106] Microphone (for voice input)

[1107] Industrial equipment (e.g. robotic arms)

[1108] software:

[1109] Python (for processing and server communication)

[1110] sounddevice module (audio recording)

[1111] requests module (sending HTTP requests)

[1112] AI voice analysis server (voice analysis and command generation)

[1113] Specific examples

[1114] Example 1:

[1115] A worker in a factory commands a robot to "go get the next part." A microphone captures this voice and sends it as audio data to a server. The server analyzes the voice and generates an action command to "go get the part." The generated action command is sent to the robot, which then goes to get the part.

[1116] Example 2:

[1117] User A picks up a guitar and plays a chord progression into the device. This performance data is sent to the server, which analyzes it to determine that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment, which is then fed back to User A in real time.

[1118] Prompt Sentence Examples

[1119] "Voice command to the robot: Go get the next part."

[1120] "Pick up the parts from the conveyor"

[1121] "Move to the assembly area."

[1122] The above is an embodiment of the present invention.

[1123] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1124] Step 1:

[1125] The user provides voice input. The user issues a voice command using a microphone connected to the device. The device captures this voice as digital data. The input is the user's voice, and the output is digitized voice data. This digital data is analyzed in the next step.

[1126] Step 2:

[1127] The device sends the captured audio data to the server. The device transfers the audio data to the server in real time via the Internet. The input is the audio data captured by the device, and the output is the data sent to the server.

[1128] Step 3:

[1129] The server analyzes the received audio data. Using an audio analysis module, the server extracts musical characteristics such as tempo, pitch, and rhythm from the data. It also analyzes operational commands for industrial equipment. The input is the audio data sent to the server, and the output is the extracted musical characteristics and operational commands.

[1130] Step 4:

[1131] The server generates music in real time based on the analysis results. The server uses an AI algorithm to generate accompaniment data based on the extracted musical characteristics. The input is the extracted musical characteristics, and the output is the generated accompaniment data.

[1132] Step 5:

[1133] The generated music is sent to the user's device. The server sends the generated accompaniment data to the user's device in real time. The input is the generated accompaniment data, and the output is the accompaniment data sent to the user's device.

[1134] Step 6:

[1135] The server generates operation commands for industrial equipment based on the analysis results. The server generates appropriate operation commands based on the voice analysis results and sends them to the industrial equipment via the network. The input is the voice analysis results, and the output is the generated operation commands.

[1136] Step 7:

[1137] Industrial equipment performs tasks according to operation commands. Industrial equipment (such as robots) performs specified operations according to operation commands sent from the server. The input is the operation command sent from the server, and the output is the operation of the industrial equipment.

[1138] Step 8:

[1139] The server simultaneously records the user's performance data and the generated accompaniment data, and stores them in cloud storage. The input is the user's performance data and the generated accompaniment data, and the output is the data stored in cloud storage.

[1140] Step 9:

[1141] A user can share music they have created with other users. The user's device uses a dedicated interface to share the music they have created with other users. The input is data stored in cloud storage, and the output is the shared music data.

[1142] Step 10:

[1143] Another user adds a new part. Another user adds a new part to the shared music data and sends it back to the system. The input is the shared music data, and the output is the music data with the new part added.

[1144] The above are the specific processing steps of the program of the present invention.

[1145] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1146] The present invention provides an online platform that combines an emotion engine that recognizes a user's emotions to generate music in real time according to the user's emotional state, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[1147] Overall system configuration

[1148] The system consists of the following components:

[1149] 1. User's Device

[1150] 2. Server

[1151] 3. Cloud Storage

[1152] 4. Emotion Engine

[1153] Receiving voice input and analyzing emotions

[1154] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[1155] Sending voice data and emotion data

[1156] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[1157] Analysis of voice data and use of emotion data

[1158] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the AI ​​algorithm.

[1159] Real-time music generation and emotional reflection

[1160] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[1161] User feedback

[1162] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1163] Recording and storage

[1164] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1165] Music sharing and collaboration

[1166] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[1167] Specific examples

[1168] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[1169] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[1170] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[1171] Terminal: Plays back the accompaniment data generated in real time to User A.

[1172] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[1173] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[1174] Terminal: Sends voice data and emotion data to the server.

[1175] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[1176] In this way, music creation and collaboration that reflects the user's emotional state becomes possible.

[1177] The processing flow will be explained below.

[1178] Step 1:

[1179] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[1180] Step 2:

[1181] The terminal temporarily stores the voice data in a buffer and simultaneously transmits this data to the emotion engine.

[1182] Step 3:

[1183] The emotion engine analyzes the received voice data to determine the user's emotional state, for example, generating emotion categories such as "joy," "sadness," and "anger."

[1184] Step 4:

[1185] The device transmits the captured voice data and emotion data obtained from the emotion engine to a server in real time via the Internet.

[1186] Step 5:

[1187] The server receives the voice data and emotion data sent from the device and checks the format and integrity of the received data.

[1188] Step 6:

[1189] The server analyzes the audio data and extracts musical characteristics such as tempo, pitch, and rhythm, and stores the analysis results in a database.

[1190] Step 7:

[1191] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics and emotional data. The tempo, key, rhythm, and other aspects of the generated music are dynamically adjusted according to the emotional data.

[1192] Step 8:

[1193] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[1194] Step 9:

[1195] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1196] Step 10:

[1197] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them in cloud storage as a single music file.

[1198] Step 11:

[1199] The user has an interface to review the recorded music files from their device and edit them if necessary.

[1200] Step 12:

[1201] Users can use a dedicated function to share edited music files with other users and publish them online.

[1202] Step 13:

[1203] The server distributes the published music files for other users to view, play, and add their own parts to.

[1204] Step 14:

[1205] Other users input new parts and send the audio data to the server.

[1206] Step 15:

[1207] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[1208] Step 16:

[1209] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[1210] These steps facilitate real-time music generation that reflects emotional states and collaboration between users.

[1211] Example 2

[1212] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1213] Conventional music generation systems have difficulty generating music that reflects a user's emotional state in real time, and sharing and collaboration between users is also limited. To solve these problems, an effective method is needed to analyze emotions, generate music based on that emotion in real time, and share and collaborate with other users.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving captured audio data and emotional data and analyzing musical characteristics and emotional states, means for generating music in real time based on the analysis results and emotional data, and means for transmitting the generated music to the user's terminal. This enables the generation of music that reflects emotions in real time, as well as effective sharing and collaboration between users.

[1215] "User" refers to a person who uses a terminal to input voice data and use the system.

[1216] A "terminal" is a hardware device that allows a user to input voice information, and includes devices such as microphones, computers, and smartphones.

[1217] "Voice data" refers to data that represents in digital form the voice signal input by a user through a terminal.

[1218] "Emotional data" refers to information about the user's emotional state obtained by analyzing voice data.

[1219] "Server" refers to a computer system for receiving and analyzing audio and emotional data and generating music.

[1220] "Musical characteristics" refers to musical elements such as tempo, pitch, and rhythm obtained by analyzing audio data.

[1221] "Real-time" means that user input is processed immediately, without delay.

[1222] "Music generation" refers to the process of creating new music based on audio and emotional data.

[1223] "Sending to terminal" refers to sending the music data generated by the server to the user's terminal via the network.

[1224] "Dedicated Interface" refers to the dedicated operating screens and tools used by users to share and collaborate on music they create with other users.

[1225] "Cloud storage" refers to online storage services for storing and accessing data over the Internet.

[1226] "Adding a new part" refers to another user adding new audio data or instrument parts to an existing music file.

[1227] This invention provides an online platform that generates music in real time according to the user's emotional state by combining an emotion engine that recognizes the user's emotions, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[1228] Overall system configuration

[1229] The system consists of the following components:

[1230] 1. User's Device

[1231] 2. Server

[1232] 3. Cloud Storage

[1233] 4. Emotion Engine

[1234] Receiving voice input and analyzing emotions

[1235] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[1236] Sending voice data and emotion data

[1237] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[1238] Analysis of voice data and use of emotion data

[1239] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the generative AI model.

[1240] Real-time music generation and emotional reflection

[1241] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[1242] User feedback

[1243] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1244] Recording and storage

[1245] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1246] Music sharing and collaboration

[1247] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[1248] Specific examples

[1249] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[1250] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[1251] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[1252] Terminal: Plays back the accompaniment data generated in real time to User A.

[1253] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[1254] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[1255] Terminal: Sends voice data and emotion data to the server.

[1256] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[1257] Prompt Sentence Examples

[1258] You can check the system's operation by inputting the following prompt sentences into the generative AI model.

[1259] When a user plays the guitar and has the emotion "joy", send the audio data and emotion data to the server to generate a cheerful accompaniment in the key of C at 120 BPM in real time.

[1260] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1261] Step 1: User speaks

[1262] Users input musical instruments or vocals using a microphone connected to their device. The microphone connected to the device captures the audio signal and converts it into digital data. The input is the user's voice, and the output is digitally converted audio data.

[1263] Step 2: Sentiment Analysis

[1264] The device passes the voice data captured in step 1 to the emotion engine. The emotion engine analyzes the tone, pitch, tempo, etc. of the voice data to identify the user's emotional state (e.g., "joy," "sadness," etc.). The input is the digital voice data obtained in step 1, and the output is emotion data resulting from the analysis.

[1265] Step 3: Sending voice and emotion data

[1266] The device transmits the emotion data and voice data from step 2 to the server. The transmission process is carried out in real time via the Internet. The input is the voice data and emotion data, and the output is a confirmation message that the server has received the data.

[1267] Step 4: Receiving and analyzing voice and emotion data

[1268] The server receives the audio data and emotion data sent in step 3. After receiving the audio data, it checks the format and integrity of the audio data and analyzes the musical characteristics (e.g., tempo, pitch, rhythm). The input is the received audio data and emotion data, and the output is the analyzed musical characteristic data and emotion data.

[1269] Step 5: Real-time music generation

[1270] The server uses a generative AI model to generate music in real time based on the analyzed musical characteristic data and emotional data. The tempo, key, rhythm, etc. are dynamically adjusted based on the emotional data. The input is the musical characteristic data and emotional data, and the output is the generated accompaniment data.

[1271] Step 6: Send the generated music data to your device

[1272] The server converts the generated accompaniment data into an appropriate format in real time and sends it to the user's terminal. The input is the generated accompaniment data, and the output is the accompaniment data received by the user's terminal.

[1273] Step 7: User feedback and real-time playback

[1274] The device decodes the received accompaniment data and plays it back to the user in real time. The user can then perform additional music along with the accompaniment. The input is the received accompaniment data, and the output is music played back to the user in real time.

[1275] Step 8: Record and save your performance data

[1276] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotion data, and stores them as a single music file in cloud storage. The input is the user's performance data, the generated accompaniment data, and the emotion data, and the output is the music file stored in cloud storage.

[1277] Step 9: Share and collaborate on music

[1278] Users use a dedicated interface to share music files stored in cloud storage with other users. Other users can browse, play, and add their own parts to the shared music files. The input is the music file retrieved from cloud storage, and the output is the new music file that has been shared and edited.

[1279] Example operation

[1280] As a concrete example, if User A plays the guitar and the emotion engine analyzes that his / her emotional state is "joy," the device sends this to the server. The server receives the data, generates a cheerful accompaniment with a tempo of 120 BPM and the key of C, and sends it to the device. The device plays it in real time, and User A continues playing along with the accompaniment. If User B later wants to add his / her own piano part, he / she retrieves the music file from cloud storage and plays it, reflecting the emotional data of "sadness." The server integrates the data and generates a new music file that reflects the emotion.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] In recent years, there has been a demand for music generation that reflects a user's emotions, but there is a lack of technology that can generate music in real time based on emotions and share and collaborate with other users. Therefore, providing a system that can analyze a user's emotions, generate music based on those emotions in real time, and make it easy to share with other users has become an important challenge.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1285] In this invention, the server includes means for analyzing a user's emotions and dynamically generating music based on the results, means for dynamically adjusting the tempo, key, and rhythm of the generated music, and means for providing feedback on the generated music to the user in real time and allowing the user to perform additional performances. This enables music to be generated in real time according to the user's emotions, and allows easy sharing and collaboration with other users.

[1286] A "terminal" is a device that allows a user to input voice information, including smartphones, tablets, and personal computers.

[1287] "Voice data" is digital sound information acquired when a user makes a voice input.

[1288] "Musical characteristics" refers to musical features such as tempo, pitch, and rhythm extracted from audio data.

[1289] A "server" is a high performance computing device located at a remote location that receives audio data and performs analysis and music generation.

[1290] "Real-time" refers to a time range in which processing occurs so quickly that there is almost no noticeable delay from user input.

[1291] "Accompaniment data" refers to the accompaniment portion of the music generated based on the user's voice data and emotion data.

[1292] "Cloud storage" is a remote storage service for storing data on the Internet.

[1293] "Emotion data" is information that indicates the emotional state of the user analyzed from the user's voice data.

[1294] "Generative AI model" refers to an algorithm or program for generating music using artificial intelligence techniques.

[1295] A "prompt" refers to an instruction or question that is input into an AI model.

[1296] "Collaboration" refers to multiple users working together to create a single musical work.

[1297] This invention is a system that analyzes a user's emotions, generates music in real time based on the results, and provides an online platform for sharing and collaborating with other users.

[1298] Overall system configuration

[1299] The system consists of the following components:

[1300] 1. User device: The device through which the user inputs voice and transmits emotion data. This includes smartphones, tablets, and PCs.

[1301] 2. Server: A high-performance computer device installed in a remote location that receives audio data and emotion data, analyzes them, and generates music.

[1302] 3. Cloud storage: A remote storage service on the Internet for storing generated music data.

[1303] 4. Emotion engine: An engine that contains algorithms that analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Azure Cognitive Services.

[1304] Receiving voice input and analyzing emotions

[1305] Users input musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine receives the same audio input data and analyzes the user's emotional state. For example, if the user is feeling "sad," the system will recognize that emotion.

[1306] Sending voice and emotion data

[1307] The device transmits the captured voice data and analyzed emotion data to a server. This transmission process is carried out in real time via the Internet. The voice data and emotion data are transmitted together and checked for data integrity.

[1308] Analysis of voice data and use of emotion data

[1309] The server receives the audio data and emotion data sent from the device. The received audio data is used to analyze musical characteristics, extracting information such as tempo, pitch, and rhythm. The emotion data is used as input for the AI ​​algorithm and reflected in the generated music.

[1310] Real-time music generation and emotional reflection

[1311] The server generates accompaniment data in real time based on musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. For example, if the emotional engine analyzes "joy," it will generate upbeat, fast-paced music.

[1312] User feedback

[1313] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional parts along with the accompaniment, thereby realizing a musical experience that is in line with the user's emotions.

[1314] Recording and storage

[1315] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1316] Music sharing and collaboration

[1317] Users use a dedicated interface to share the music files they create with other users. This interface is located on the device and is easy to operate. Other users can view and play the shared music files and add their own parts. For example, if user A starts playing the piano in a lonely mood, other users can add their own instruments to the music and collaborate.

[1318] Prompt example

[1319] "Use the Emotion Music Real-time Generator (EMRG) to express your lonely feelings on piano, then collaborate with other users by adding their own instruments to your music."

[1320] This concludes the detailed description of the "Mode for carrying out the invention." The entire system cooperates to automatically create, share, and collaborate on music that is in line with the user's emotions.

[1321] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1322] Step 1:

[1323] The user performs voice input. Specifically, the user inputs an instrument or vocals into a device such as a smartphone, tablet, or PC. At this time, the microphone connected to the device captures the audio data in digital format. Input: User's voice. Output: Digital audio data.

[1324] Step 2:

[1325] The device sends the captured voice data to the emotion engine, which uses algorithms such as IBM Watson Tone Analyzer and Azure Cognitive Services to analyze the voice data and identify the user's emotional state. Input: Digital voice data. Output: Analyzed emotion data.

[1326] Step 3:

[1327] The device transmits the analyzed emotion data and voice data to the server in real time via the Internet. The server checks the integrity of the received data. Input: Voice data and emotion data. Output: Data packets (voice data and emotion data).

[1328] Step 4:

[1329] The server analyzes the received audio data to extract musical characteristics. This process uses algorithms to identify information such as tempo, pitch, and rhythm. Input: Audio data. Output: Musical characteristics data.

[1330] Step 5:

[1331] The server uses a generative AI model to generate music in real time based on musical characteristic data and emotional data. The generative AI model dynamically adjusts tempo, key, rhythm, etc. to generate accompaniment data. Input: Music characteristic data and emotional data. Output: Generated accompaniment data.

[1332] Step 6:

[1333] The server sends the generated accompaniment data to the user's device. The device decodes the received accompaniment data and plays the audio in real time. This allows the user to perform additional music along with the generated music. Input: Accompaniment data. Output: Music played in real time.

[1334] Step 7:

[1335] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them as a single music file in cloud storage. Input: User's performance data and accompaniment data. Output: Music file saved in cloud storage.

[1336] Step 8:

[1337] Users can share the music files they create with other users using a dedicated interface on their device. Other users can view and play the shared music files and add their own parts. Input: Music files stored in cloud storage. Output: New music file containing the added parts.

[1338] The above is a concrete process flow for implementing this invention. Each step involves data input and output, and we provide a system that enables real-time music creation and collaboration based on the user's emotions.

[1339] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1341] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1342] [Fourth embodiment]

[1343] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1344] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1345] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1346] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1347] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1348] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1349] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1350] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1351] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1352] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1353] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1354] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1355] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1356] The present invention provides an online platform for creating music and collaborating with other users in real time. Specific embodiments for implementing the present invention are described below.

[1357] Overall system configuration

[1358] The system consists of the following components:

[1359] 1. User's Device

[1360] 2. Server

[1361] 3. Cloud Storage

[1362] Receiving voice input

[1363] Users use their own devices to input their musical instruments and vocals. A microphone is connected to the device, and the input sound is captured as digital data, allowing for real-time recording of the performance.

[1364] Sending audio data

[1365] The device then transmits the captured audio data to a server, and this transmission process occurs in real time over the Internet.

[1366] Analysis of audio data

[1367] The server analyzes the received audio data. Specifically, the data analysis module extracts musical characteristics such as tempo, pitch, and rhythm. The analysis results are used as the basis for the subsequent music generation process.

[1368] Real-time music generation

[1369] The server generates accompaniment data in real time based on the musical characteristics, and an AI algorithm generates accompaniment data that is adapted to the analysis results and the user's performance, and sends it to the device in the appropriate format.

[1370] User feedback

[1371] The device receives the generated accompaniment data and provides real-time feedback to the user, allowing the user to continue playing along with the generated music.

[1372] Recording and storage

[1373] The server simultaneously records the user's performance data and the generated accompaniment data, and stores the generated music data in cloud storage, allowing the user to access, edit, and re-share the music data later.

[1374] Music sharing and collaboration

[1375] Users have a dedicated interface to share the music they create with other users. This interface is located on the device and is easy to navigate. Other users can view and play the shared music and add their own new parts.

[1376] The server re-analyzes and generates new music data, and saves the updated music data in cloud storage, facilitating ongoing collaboration.

[1377] Specific examples

[1378] Example 1: User A holds a guitar and plays a chord progression into the device.

[1379] Device: Captures guitar audio data and sends it to the server.

[1380] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[1381] Terminal: Plays back the accompaniment data generated in real time to User A.

[1382] Example 2: User B adds a piano part.

[1383] User B inputs voice into the device and provides the piano part.

[1384] Device: Sends audio data to the server.

[1385] Server: Analyzes the piano part and integrates it into the existing music data.

[1386] This will enable advanced musical collaboration between users.

[1387] The processing flow will be explained below.

[1388] Step 1:

[1389] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[1390] Step 2:

[1391] The device temporarily stores the captured audio data in an internal buffer and then transmits it to a server via the Internet.

[1392] Step 3:

[1393] The server receives the voice data sent from the terminal and checks the format and integrity of the received data.

[1394] Step 4:

[1395] The server uses an audio data analysis module to extract musical characteristics such as tempo, pitch, rhythm, etc. The results of this analysis are stored in a database.

[1396] Step 5:

[1397] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics, and this accompaniment data is generated in a format that adapts to the user's input.

[1398] Step 6:

[1399] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[1400] Step 7:

[1401] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1402] Step 8:

[1403] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them in cloud storage as a single music file.

[1404] Step 9:

[1405] The user has an interface that allows them to view the recorded music files on their device and edit them if necessary.

[1406] Step 10:

[1407] Users can use a dedicated function to share edited music files with other users and publish the music files online.

[1408] Step 11:

[1409] The server distributes the published music files for other users to view, play, and add their own parts to.

[1410] Step 12:

[1411] Other users input new parts and send the audio data to the server.

[1412] Step 13:

[1413] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[1414] Step 14:

[1415] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[1416] These steps facilitate real-time music creation and collaboration.

[1417] Example 1

[1418] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] In today's world, music production requires advanced technology and equipment, making it difficult to generate music in real time and facilitate smooth collaboration between multiple users. There is a particular demand for real-time music generation and collaboration with other users while receiving immediate feedback, but few systems currently exist that enable this. Therefore, there is an urgent need to provide a system that allows users to easily generate music and collaborate with other users in real time.

[1420] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1421] In this invention, the server includes means for receiving audio data and analyzing musical characteristics, means for generating music in real time based on the analysis results, and means for re-analyzing and generating new music data and saving the updated music data in cloud storage. This allows users to receive real-time feedback on their voice-input performance data along with accompaniment data, enabling smooth collaboration with other users.

[1422] "User" means an individual or organization that utilizes the system to input voice and perform music, create music, and collaborate.

[1423] "Terminal" means a device used by a user to perform voice input and has the function of capturing voice data and transmitting it to a server.

[1424] A "server" is a central processing unit that receives and analyzes audio data, and generates and manages music in real time.

[1425] "Audio data" refers to digital data of musical instrument sounds or vocals played by a user on a terminal.

[1426] "Music characteristics" refers to characteristic information such as tempo, pitch, and rhythm extracted from audio data.

[1427] The "analysis results" are data on musical characteristics extracted from the audio data.

[1428] "Generating music in real time" means instantly generating accompaniment data adapted to the user's performance based on the analysis results.

[1429] "Accompaniment data" refers to auxiliary music data such as drums and bass that is generated to match the user's performance.

[1430] "Cloud storage" is a service that stores and manages data on the Internet.

[1431] "Providing feedback" means playing the generated accompaniment data to the user in real time.

[1432] "Collaboration" is the activity of multiple users working together to create, edit, and share music.

[1433] "Adding a new part" means that another user adds a new instrument sound or vocals to existing music data.

[1434] "Updated music data" refers to the latest music data that has new parts added and has been re-analyzed and re-generated.

[1435] The present invention provides a system that allows a user to input voice, generate music in real time, and collaborate with other users. Specific embodiments for carrying out the present invention are described below.

[1436] Overall system configuration

[1437] The system consists of the following components:

[1438] 1. User's Device

[1439] 2. Server

[1440] 3. Cloud Storage

[1441] Receiving voice input

[1442] Users use their own devices to input their musical instruments or vocals. A microphone is connected to the device, and the input sound is captured as digital data. This allows for real-time recording of performances. For example, if a user plays a chord progression on a guitar, the sound is captured by the microphone.

[1443] Sending audio data

[1444] The device transmits the captured audio data to the server. This transmission process is performed in real time via the Internet. For example, as soon as the user starts playing, the device collects audio data in real time and transmits it to the server.

[1445] Analysis of audio data

[1446] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. This analysis module extracts musical characteristics such as tempo (e.g., 120 BPM), pitch (e.g., key of C), and rhythm. These characteristics are the information necessary for subsequent AI music generation. For example, the server determines from the audio data that the tempo is 120 BPM and the key is C.

[1447] Real-time music generation

[1448] The server generates accompaniment data in real time based on the analysis results. Using a generative AI model, it generates accompaniment data adapted to the user's performance. For example, it generates drum and bass accompaniments based on guitar audio data. The generated data is converted into an appropriate format and sent to the device.

[1449] User feedback

[1450] The device receives the accompaniment data sent from the server. The received data is fed back to the user in real time. This allows the user to continue playing with the accompaniment added to their own performance. For example, drums and bass can be played simultaneously along with the guitar playing.

[1451] Recording and storage

[1452] The server simultaneously records the user's performance data and the generated accompaniment data. The recorded music data is stored in cloud storage, where users can access, edit, and reshare it later. This allows the music data to remain available even after the session has ended.

[1453] Music sharing and collaboration

[1454] Users have an interface to share the music they create with other users. The shared music can be viewed and played by other users, and new parts can be added. For example, User A can create a guitar accompaniment and User B can add a piano part.

[1455] Reparsing and updating

[1456] The server then re-analyzes the newly added music data and makes any necessary updates. It then integrates the new music data with the existing data using a generative AI model and saves the updated music data back to cloud storage, enabling continuous musical collaboration between users.

[1457] Specific examples

[1458] Example 1:

[1459] User A picks up a guitar and plays a chord progression into the terminal.

[1460] Device: Captures guitar audio data and sends it to the server.

[1461] Server: Analyzes the audio data and determines that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment.

[1462] Terminal: Plays back the accompaniment data generated in real time to User A.

[1463] Example 2:

[1464] User B adds a piano part.

[1465] User B: Provides the piano part by dictating the song on the device.

[1466] Device: Sends audio data to the server.

[1467] Server: Analyzes piano parts and integrates them into existing music data, enabling advanced musical collaboration between users.

[1468] Prompt Sentence Examples

[1469] "Enter your guitar chord progression and generate accompaniment in real time."

[1470] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1471] Step 1:

[1472] User voice input

[1473] Users use their devices to input musical instruments or vocals. A microphone is used as an input device, and the microphone captures audio data as digital data. For example, when a user plays a guitar, the microphone collects the audio and converts it into digital data.

[1474] Input: User's voice

[1475] Output: Digitized audio data

[1476] Step 2:

[1477] Device capture and transmission of audio data

[1478] The device captures digitized audio data and transmits it over the internet to a server. This process happens in real time, with high speeds to minimize latency. For example, after a user plays the guitar, the audio data is sent instantly to the server.

[1479] Input: Digitized audio data

[1480] Output: Audio data sent to the server

[1481] Step 3:

[1482] Receiving and analyzing voice data by the server

[1483] The server receives the audio data sent from the device. The received audio data is analyzed using a data analysis module. Specifically, musical characteristics such as tempo, pitch, and rhythm are extracted. For example, the server analyzes the audio data and determines that the tempo is 120 BPM and the key is C.

[1484] Input: Audio data sent to the server

[1485] Output: Extracted musical characteristics data (tempo, pitch, rhythm)

[1486] Step 4:

[1487] Real-time music generation by server

[1488] The server uses a generative AI model to generate accompaniment data in real time based on the analyzed musical characteristic data. This AI model creates drum and bass accompaniment that is adapted to the user's performance. For example, it generates drum and bass accompaniment for a 120 BPM song in the key of C. The generated accompaniment data is converted into an appropriate format and sent to the device.

[1489] Input: Musical characteristics data (tempo, pitch, rhythm)

[1490] Output: Generated accompaniment data

[1491] Step 5:

[1492] Sending feedback to the device

[1493] The device receives the accompaniment data sent from the server. The received accompaniment data is played back in real time, providing feedback to the user. This allows the user to use the accompaniment generated to match their own performance. For example, a user can play the guitar while simultaneously listening to the generated drum and bass accompaniment.

[1494] Input: Generated accompaniment data

[1495] Output: Real-time feedback sound to the user

[1496] Step 6:

[1497] Server-based recording and storage

[1498] The server simultaneously records the user's performance data and the generated accompaniment data. This recording data is stored in cloud storage. Users can later access, edit, and reshare this data. For example, users can review and edit the recording data after the session has ended.

[1499] Input: User's performance data, generated accompaniment data

[1500] Output: Recording data saved in cloud storage

[1501] Step 7:

[1502] Users can share and collaborate on music

[1503] Users share their music with other users using a dedicated interface that is intuitive and easy to use. Other users can browse, play, and add new parts to the shared music. For example, user B can add a piano part to a song created by user A.

[1504] Input: User-created music data

[1505] Output: Share and collaborate with others

[1506] Step 8:

[1507] Server reparsing and updating

[1508] The server then re-analyzes the newly added music data and makes any necessary updates. It then uses a generative AI model to integrate the new music data with existing data and saves the updated music data in cloud storage. For example, it can analyze newly added parts and update the entire song, promoting collaboration.

[1509] Input: Newly added music data

[1510] Output: Updated music data (stored in cloud storage)

[1511] (Application example 1)

[1512] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1513] Conventional factory robot control systems require complex programming and settings, making efficient operation difficult. Furthermore, it is extremely difficult to achieve real-time collaborative work between multiple robots, often resulting in a loss of efficiency and safety. The present invention aims to solve these problems by realizing simple and efficient robot control and collaborative work using voice input.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1515] In this invention, the server includes means for a user to input voice, means for the server to receive the voice data and analyze musical characteristics, means for the server to generate music in real time based on the analysis results, means for transmitting the generated music to the user's terminal, means for the user to share the music created by the user with other users, means for other users to add new parts, means for analyzing the voice input and generating operation commands for various devices, and means for transmitting the generated operation commands to industrial devices and causing them to execute them. This enables simple and efficient robot control using voice input and cooperative work between multiple robots.

[1516] "User" means any person or group of people who uses the online platform to provide voice input.

[1517] "Voice input" is the act or process of capturing user-uttered speech as digital data.

[1518] The "server" is a central processing device that receives and analyzes audio data, and generates music and motion commands.

[1519] "Voice data" refers to digitized data of voice input by a user.

[1520] "Musical characteristics" are musical elements such as tempo, pitch, and rhythm extracted from audio data.

[1521] "Real-time" means that data is sent, received, and processed nearly simultaneously.

[1522] "Music generation" is the process of creating new music based on analyzed musical characteristics.

[1523] A "terminal" is an electronic device used by a user (such as a smartphone, tablet, or PC).

[1524] "Sharing" is the process by which a user shares music or data they have created with other users.

[1525] A "robot" is an automated mechanical device that performs tasks in a factory.

[1526] An "operation command" is an instruction to operate various devices that is generated based on the analyzed voice input.

[1527] The present invention is a system that generates operational commands for music and industrial equipment in real time through user voice input, enabling efficient and rapid collaboration.

[1528] Overall system configuration

[1529] The system consists of the following components:

[1530] 1. User's Device

[1531] 2. Server

[1532] 3. Cloud Storage

[1533] 4. Industrial equipment (robots)

[1534] Voice input and analysis

[1535] The user inputs voice. The voice is captured as digital data using a microphone connected to the device. This voice data is sent from the device to a server. The server analyzes the received voice data and extracts musical characteristics such as tempo, pitch, and rhythm. The server also analyzes the instructions the user gives to industrial equipment.

[1536] Music Generation and Feedback

[1537] The server generates accompaniment data in real time based on the musical characteristics, and the generated music data is sent to the user's device and fed back in real time, allowing the user to continue playing along with the generated music.

[1538] Robot motion command generation and execution

[1539] The server analyzes the user's voice input and generates operational commands for the industrial equipment, which are then sent over the network to the industrial equipment, causing the robot to instantly perform the required action.

[1540] Recording and storage

[1541] The server simultaneously records the user's performance data, the generated accompaniment data, and the generated movement command data, and saves them in cloud storage, allowing the user to access, edit, and re-share the data later.

[1542] Music sharing and collaboration

[1543] Users can use a dedicated interface to share the music they create with other users, who can then view and play the music, add new parts, and share operating instructions for industrial equipment.

[1544] Hardware and software used

[1545] Hardware:

[1546] Microphone (for voice input)

[1547] Industrial equipment (e.g. robotic arms)

[1548] software:

[1549] Python (for processing and server communication)

[1550] sounddevice module (audio recording)

[1551] requests module (sending HTTP requests)

[1552] AI voice analysis server (voice analysis and command generation)

[1553] Specific examples

[1554] Example 1:

[1555] A worker in a factory commands a robot to "go get the next part." A microphone captures this voice and sends it as audio data to a server. The server analyzes the voice and generates an action command to "go get the part." The generated action command is sent to the robot, which then goes to get the part.

[1556] Example 2:

[1557] User A picks up a guitar and plays a chord progression into the device. This performance data is sent to the server, which analyzes it to determine that the tempo is 120 BPM and the key is C. The AI ​​module uses this information to generate drum and bass accompaniment, which is then fed back to User A in real time.

[1558] Prompt Sentence Examples

[1559] "Voice command to the robot: Go get the next part."

[1560] "Pick up the parts from the conveyor"

[1561] "Move to the assembly area."

[1562] The above is an embodiment of the present invention.

[1563] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1564] Step 1:

[1565] The user provides voice input. The user issues a voice command using a microphone connected to the device. The device captures this voice as digital data. The input is the user's voice, and the output is digitized voice data. This digital data is analyzed in the next step.

[1566] Step 2:

[1567] The device sends the captured audio data to the server. The device transfers the audio data to the server in real time via the Internet. The input is the audio data captured by the device, and the output is the data sent to the server.

[1568] Step 3:

[1569] The server analyzes the received audio data. Using an audio analysis module, the server extracts musical characteristics such as tempo, pitch, and rhythm from the data. It also analyzes operational commands for industrial equipment. The input is the audio data sent to the server, and the output is the extracted musical characteristics and operational commands.

[1570] Step 4:

[1571] The server generates music in real time based on the analysis results. The server uses an AI algorithm to generate accompaniment data based on the extracted musical characteristics. The input is the extracted musical characteristics, and the output is the generated accompaniment data.

[1572] Step 5:

[1573] The generated music is sent to the user's device. The server sends the generated accompaniment data to the user's device in real time. The input is the generated accompaniment data, and the output is the accompaniment data sent to the user's device.

[1574] Step 6:

[1575] The server generates operation commands for industrial equipment based on the analysis results. The server generates appropriate operation commands based on the voice analysis results and sends them to the industrial equipment via the network. The input is the voice analysis results, and the output is the generated operation commands.

[1576] Step 7:

[1577] Industrial equipment performs tasks according to operation commands. Industrial equipment (such as robots) performs specified operations according to operation commands sent from the server. The input is the operation command sent from the server, and the output is the operation of the industrial equipment.

[1578] Step 8:

[1579] The server simultaneously records the user's performance data and the generated accompaniment data, and stores them in cloud storage. The input is the user's performance data and the generated accompaniment data, and the output is the data stored in cloud storage.

[1580] Step 9:

[1581] A user can share music they have created with other users. The user's device uses a dedicated interface to share the music they have created with other users. The input is data stored in cloud storage, and the output is the shared music data.

[1582] Step 10:

[1583] Another user adds a new part. Another user adds a new part to the shared music data and sends it back to the system. The input is the shared music data, and the output is the music data with the new part added.

[1584] The above are the specific processing steps of the program of the present invention.

[1585] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1586] The present invention provides an online platform that combines an emotion engine that recognizes a user's emotions to generate music in real time according to the user's emotional state, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[1587] Overall system configuration

[1588] The system consists of the following components:

[1589] 1. User's Device

[1590] 2. Server

[1591] 3. Cloud Storage

[1592] 4. Emotion Engine

[1593] Receiving voice input and analyzing emotions

[1594] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[1595] Sending voice data and emotion data

[1596] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[1597] Analysis of voice data and use of emotion data

[1598] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the AI ​​algorithm.

[1599] Real-time music generation and emotional reflection

[1600] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[1601] User feedback

[1602] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1603] Recording and storage

[1604] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1605] Music sharing and collaboration

[1606] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[1607] Specific examples

[1608] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[1609] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[1610] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[1611] Terminal: Plays back the accompaniment data generated in real time to User A.

[1612] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[1613] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[1614] Terminal: Sends voice data and emotion data to the server.

[1615] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[1616] In this way, music creation and collaboration that reflects the user's emotional state becomes possible.

[1617] The processing flow will be explained below.

[1618] Step 1:

[1619] Users input their musical instruments or vocals using their own devices, which are connected to a microphone and the input sound is captured as digital data.

[1620] Step 2:

[1621] The terminal temporarily stores the voice data in a buffer and simultaneously transmits this data to the emotion engine.

[1622] Step 3:

[1623] The emotion engine analyzes the received voice data to determine the user's emotional state, for example, generating emotion categories such as "joy," "sadness," and "anger."

[1624] Step 4:

[1625] The device transmits the captured voice data and emotion data obtained from the emotion engine to a server in real time via the Internet.

[1626] Step 5:

[1627] The server receives the voice data and emotion data sent from the device and checks the format and integrity of the received data.

[1628] Step 6:

[1629] The server analyzes the audio data and extracts musical characteristics such as tempo, pitch, and rhythm, and stores the analysis results in a database.

[1630] Step 7:

[1631] The server uses an AI algorithm to generate accompaniment data in real time based on the analyzed musical characteristics and emotional data. The tempo, key, rhythm, and other aspects of the generated music are dynamically adjusted according to the emotional data.

[1632] Step 8:

[1633] The server converts the generated accompaniment data into an appropriate format and transmits it to the terminal.

[1634] Step 9:

[1635] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1636] Step 10:

[1637] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them in cloud storage as a single music file.

[1638] Step 11:

[1639] The user has an interface to review the recorded music files from their device and edit them if necessary.

[1640] Step 12:

[1641] Users can use a dedicated function to share edited music files with other users and publish them online.

[1642] Step 13:

[1643] The server distributes the published music files for other users to view, play, and add their own parts to.

[1644] Step 14:

[1645] Other users input new parts and send the audio data to the server.

[1646] Step 15:

[1647] The server analyzes the new parts and integrates them into the existing music file, then stores the combined data back in cloud storage.

[1648] Step 16:

[1649] The server notifies all relevant users of the updated music file, allowing them to review and edit it again.

[1650] These steps facilitate real-time music generation that reflects emotional states and collaboration between users.

[1651] Example 2

[1652] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1653] Conventional music generation systems have difficulty generating music that reflects a user's emotional state in real time, and sharing and collaboration between users is also limited. To solve these problems, an effective method is needed to analyze emotions, generate music based on that emotion in real time, and share and collaborate with other users.

[1654] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving captured audio data and emotional data and analyzing musical characteristics and emotional states, means for generating music in real time based on the analysis results and emotional data, and means for transmitting the generated music to the user's terminal. This enables the generation of music that reflects emotions in real time, as well as effective sharing and collaboration between users.

[1655] "User" refers to a person who uses a terminal to input voice data and use the system.

[1656] A "terminal" is a hardware device that allows a user to input voice information, and includes devices such as microphones, computers, and smartphones.

[1657] "Voice data" refers to data that represents in digital form the voice signal input by a user through a terminal.

[1658] "Emotional data" refers to information about the user's emotional state obtained by analyzing voice data.

[1659] "Server" refers to a computer system for receiving and analyzing audio and emotional data and generating music.

[1660] "Musical characteristics" refers to musical elements such as tempo, pitch, and rhythm obtained by analyzing audio data.

[1661] "Real-time" means that user input is processed immediately, without delay.

[1662] "Music generation" refers to the process of creating new music based on audio and emotional data.

[1663] "Sending to terminal" refers to sending the music data generated by the server to the user's terminal via the network.

[1664] "Dedicated Interface" refers to the dedicated operating screens and tools used by users to share and collaborate on music they create with other users.

[1665] "Cloud storage" refers to online storage services for storing and accessing data over the Internet.

[1666] "Adding a new part" refers to another user adding new audio data or instrument parts to an existing music file.

[1667] This invention provides an online platform that generates music in real time according to the user's emotional state by combining an emotion engine that recognizes the user's emotions, and allows sharing and collaboration with other users. Specific embodiments for implementing the present invention are described below.

[1668] Overall system configuration

[1669] The system consists of the following components:

[1670] 1. User's Device

[1671] 2. Server

[1672] 3. Cloud Storage

[1673] 4. Emotion Engine

[1674] Receiving voice input and analyzing emotions

[1675] Users input their musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine then receives the same audio input data and analyzes the user's emotional state.

[1676] Sending voice data and emotion data

[1677] The device transmits the captured voice data and analyzed emotion data to a server, and this transmission process is carried out in real time via the Internet.

[1678] Analysis of voice data and use of emotion data

[1679] The server receives the audio data and emotion data sent from the device, checks the format and integrity of the received audio data, and analyzes the musical characteristics. The analysis results and emotion data are used as input for the generative AI model.

[1680] Real-time music generation and emotional reflection

[1681] The server generates accompaniment data in real time based on the musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. The generated accompaniment data is sent to the terminal in an appropriate format.

[1682] User feedback

[1683] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional music along with the accompaniment.

[1684] Recording and storage

[1685] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1686] Music sharing and collaboration

[1687] Users can share their music files with other users through a dedicated interface, which is easy to use and located on the device. Other users can view and play the shared music files and add their own parts.

[1688] Specific examples

[1689] Example 1: User A picks up a guitar and plays a chord progression on the device. The emotion engine determines that the emotional state is "joy."

[1690] Device: Captures guitar sound data and emotion data of "joy" and sends them to the server.

[1691] Server: Analyzes audio and emotional data and generates a drum and bass accompaniment with a tempo of 120 BPM, key of C, and a bright atmosphere.

[1692] Terminal: Plays back the accompaniment data generated in real time to User A.

[1693] Example 2: User B adds a piano part and the emotion engine determines that the emotional state is "sad."

[1694] User B: Inputs voice data into the device and provides the piano part and emotion data for "sadness."

[1695] Terminal: Sends voice data and emotion data to the server.

[1696] Server: Analyzes the piano part audio and the emotional data of "sadness" and integrates it into existing music data in a way that reflects the emotion.

[1697] Prompt Sentence Examples

[1698] You can check the system's operation by inputting the following prompt sentences into the generative AI model.

[1699] When a user plays the guitar and has the emotion "joy", send the audio data and emotion data to the server to generate a cheerful accompaniment in the key of C at 120 BPM in real time.

[1700] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1701] Step 1: User speaks

[1702] Users input musical instruments or vocals using a microphone connected to their device. The microphone connected to the device captures the audio signal and converts it into digital data. The input is the user's voice, and the output is digitally converted audio data.

[1703] Step 2: Sentiment Analysis

[1704] The device passes the voice data captured in step 1 to the emotion engine. The emotion engine analyzes the tone, pitch, tempo, etc. of the voice data to identify the user's emotional state (e.g., "joy," "sadness," etc.). The input is the digital voice data obtained in step 1, and the output is emotion data resulting from the analysis.

[1705] Step 3: Sending voice and emotion data

[1706] The device transmits the emotion data and voice data from step 2 to the server. The transmission process is carried out in real time via the Internet. The input is the voice data and emotion data, and the output is a confirmation message that the server has received the data.

[1707] Step 4: Receiving and analyzing voice and emotion data

[1708] The server receives the audio data and emotion data sent in step 3. After receiving the audio data, it checks the format and integrity of the audio data and analyzes the musical characteristics (e.g., tempo, pitch, rhythm). The input is the received audio data and emotion data, and the output is the analyzed musical characteristic data and emotion data.

[1709] Step 5: Real-time music generation

[1710] The server uses a generative AI model to generate music in real time based on the analyzed musical characteristic data and emotional data. The tempo, key, rhythm, etc. are dynamically adjusted based on the emotional data. The input is the musical characteristic data and emotional data, and the output is the generated accompaniment data.

[1711] Step 6: Send the generated music data to your device

[1712] The server converts the generated accompaniment data into an appropriate format in real time and sends it to the user's terminal. The input is the generated accompaniment data, and the output is the accompaniment data received by the user's terminal.

[1713] Step 7: User feedback and real-time playback

[1714] The device decodes the received accompaniment data and plays it back to the user in real time. The user can then perform additional music along with the accompaniment. The input is the received accompaniment data, and the output is music played back to the user in real time.

[1715] Step 8: Record and save your performance data

[1716] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotion data, and stores them as a single music file in cloud storage. The input is the user's performance data, the generated accompaniment data, and the emotion data, and the output is the music file stored in cloud storage.

[1717] Step 9: Share and collaborate on music

[1718] Users use a dedicated interface to share music files stored in cloud storage with other users. Other users can browse, play, and add their own parts to the shared music files. The input is the music file retrieved from cloud storage, and the output is the new music file that has been shared and edited.

[1719] Example operation

[1720] As a concrete example, if User A plays the guitar and the emotion engine analyzes that his / her emotional state is "joy," the device sends this to the server. The server receives the data, generates a cheerful accompaniment with a tempo of 120 BPM and the key of C, and sends it to the device. The device plays it in real time, and User A continues playing along with the accompaniment. If User B later wants to add his / her own piano part, he / she retrieves the music file from cloud storage and plays it, reflecting the emotional data of "sadness." The server integrates the data and generates a new music file that reflects the emotion.

[1721] (Application example 2)

[1722] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1723] In recent years, there has been a demand for music generation that reflects a user's emotions, but there is a lack of technology that can generate music in real time based on emotions and share and collaborate with other users. Therefore, providing a system that can analyze a user's emotions, generate music based on those emotions in real time, and make it easy to share with other users has become an important challenge.

[1724] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1725] In this invention, the server includes means for analyzing a user's emotions and dynamically generating music based on the results, means for dynamically adjusting the tempo, key, and rhythm of the generated music, and means for providing feedback on the generated music to the user in real time and allowing the user to perform additional performances. This enables music to be generated in real time according to the user's emotions, and allows easy sharing and collaboration with other users.

[1726] A "terminal" is a device that allows a user to input voice information, including smartphones, tablets, and personal computers.

[1727] "Voice data" is digital sound information acquired when a user makes a voice input.

[1728] "Musical characteristics" refers to musical features such as tempo, pitch, and rhythm extracted from audio data.

[1729] A "server" is a high performance computing device located at a remote location that receives audio data and performs analysis and music generation.

[1730] "Real-time" refers to a time range in which processing occurs so quickly that there is almost no noticeable delay from user input.

[1731] "Accompaniment data" refers to the accompaniment portion of the music generated based on the user's voice data and emotion data.

[1732] "Cloud storage" is a remote storage service for storing data on the Internet.

[1733] "Emotion data" is information that indicates the emotional state of the user analyzed from the user's voice data.

[1734] "Generative AI model" refers to an algorithm or program for generating music using artificial intelligence techniques.

[1735] A "prompt" refers to an instruction or question that is input into an AI model.

[1736] "Collaboration" refers to multiple users working together to create a single musical work.

[1737] This invention is a system that analyzes a user's emotions, generates music in real time based on the results, and provides an online platform for sharing and collaborating with other users.

[1738] Overall system configuration

[1739] The system consists of the following components:

[1740] 1. User device: The device through which the user inputs voice and transmits emotion data. This includes smartphones, tablets, and PCs.

[1741] 2. Server: A high-performance computer device installed in a remote location that receives audio data and emotion data, analyzes them, and generates music.

[1742] 3. Cloud storage: A remote storage service on the Internet for storing generated music data.

[1743] 4. Emotion engine: An engine that contains algorithms that analyze the user's emotional state. Examples include IBM Watson Tone Analyzer and Azure Cognitive Services.

[1744] Receiving voice input and analyzing emotions

[1745] Users input musical instruments or vocals using their own devices. A microphone is connected to the device, and the input audio is captured as digital data. The emotion engine receives the same audio input data and analyzes the user's emotional state. For example, if the user is feeling "sad," the system will recognize that emotion.

[1746] Sending voice and emotion data

[1747] The device transmits the captured voice data and analyzed emotion data to a server. This transmission process is carried out in real time via the Internet. The voice data and emotion data are transmitted together and checked for data integrity.

[1748] Analysis of voice data and use of emotion data

[1749] The server receives the audio data and emotion data sent from the device. The received audio data is used to analyze musical characteristics, extracting information such as tempo, pitch, and rhythm. The emotion data is used as input for the AI ​​algorithm and reflected in the generated music.

[1750] Real-time music generation and emotional reflection

[1751] The server generates accompaniment data in real time based on musical characteristics and emotional data. The tempo, key, rhythm, etc. of the generated music are dynamically adjusted according to the emotional data. For example, if the emotional engine analyzes "joy," it will generate upbeat, fast-paced music.

[1752] User feedback

[1753] The device decodes the received accompaniment data and plays it back to the user in real time, allowing the user to perform additional parts along with the accompaniment, thereby realizing a musical experience that is in line with the user's emotions.

[1754] Recording and storage

[1755] The server simultaneously records the user's performance data, the generated accompaniment data, and the emotional data, and saves them as a single music file in cloud storage, allowing the user to access, edit, and reshare the music data later.

[1756] Music sharing and collaboration

[1757] Users use a dedicated interface to share the music files they create with other users. This interface is located on the device and is easy to operate. Other users can view and play the shared music files and add their own parts. For example, if user A starts playing the piano in a lonely mood, other users can add their own instruments to the music and collaborate.

[1758] Prompt example

[1759] "Use the Emotion Music Real-time Generator (EMRG) to express your lonely feelings on piano, then collaborate with other users by adding their own instruments to your music."

[1760] This concludes the detailed description of the "Mode for carrying out the invention." The entire system cooperates to automatically create, share, and collaborate on music that is in line with the user's emotions.

[1761] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1762] Step 1:

[1763] The user performs voice input. Specifically, the user inputs an instrument or vocals into a device such as a smartphone, tablet, or PC. At this time, the microphone connected to the device captures the audio data in digital format. Input: User's voice. Output: Digital audio data.

[1764] Step 2:

[1765] The device sends the captured voice data to the emotion engine, which uses algorithms such as IBM Watson Tone Analyzer and Azure Cognitive Services to analyze the voice data and identify the user's emotional state. Input: Digital voice data. Output: Analyzed emotion data.

[1766] Step 3:

[1767] The device transmits the analyzed emotion data and voice data to the server in real time via the Internet. The server checks the integrity of the received data. Input: Voice data and emotion data. Output: Data packets (voice data and emotion data).

[1768] Step 4:

[1769] The server analyzes the received audio data to extract musical characteristics. This process uses algorithms to identify information such as tempo, pitch, and rhythm. Input: Audio data. Output: Musical characteristics data.

[1770] Step 5:

[1771] The server uses a generative AI model to generate music in real time based on musical characteristic data and emotional data. The generative AI model dynamically adjusts tempo, key, rhythm, etc. to generate accompaniment data. Input: Music characteristic data and emotional data. Output: Generated accompaniment data.

[1772] Step 6:

[1773] The server sends the generated accompaniment data to the user's device. The device decodes the received accompaniment data and plays the audio in real time. This allows the user to perform additional music along with the generated music. Input: Accompaniment data. Output: Music played in real time.

[1774] Step 7:

[1775] The server simultaneously records the user's performance data and the generated accompaniment data, and saves them as a single music file in cloud storage. Input: User's performance data and accompaniment data. Output: Music file saved in cloud storage.

[1776] Step 8:

[1777] Users can share the music files they create with other users using a dedicated interface on their device. Other users can view and play the shared music files and add their own parts. Input: Music files stored in cloud storage. Output: New music file containing the added parts.

[1778] The above is a concrete process flow for implementing this invention. Each step involves data input and output, and we provide a system that enables real-time music creation and collaboration based on the user's emotions.

[1779] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1781] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1782] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1783] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1784] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1785] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1786] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1787] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1788] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1789] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1790] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1791] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1792] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1793] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1794] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1795] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1796] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1797] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1798] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1799] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1800] The following is further disclosed regarding the above embodiment.

[1801] (Claim 1)

[1802] a means for a user to provide voice input;

[1803] a server for receiving the audio data and analyzing the musical characteristics;

[1804] The server generates music in real time based on the analysis results,

[1805] means for transmitting the generated music to a user's device;

[1806] A way for users to share the music they create with other users;

[1807] A means to allow other users to add new parts;

[1808] A system including:

[1809] (Claim 2)

[1810] 10. The system of claim 1, further comprising means for analyzing the user's voice data to extract tempo, pitch, and rhythm.

[1811] (Claim 3)

[1812] 10. The system of claim 1, further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

[1813] "Example 1"

[1814] (Claim 1)

[1815] a means for a user to provide voice input;

[1816] means for the terminal to capture and transmit audio data to a server;

[1817] a server for receiving the audio data and analyzing the musical characteristics;

[1818] The server generates music in real time based on the analysis results,

[1819] means for transmitting the generated music to a user's device;

[1820] A means for users to receive real-time feedback on the generated music data;

[1821] A means for simultaneously recording the user's performance data and the generated accompaniment data and saving them in cloud storage;

[1822] a means for users to share their music with other users and allow other users to add new parts;

[1823] The server re-analyzes and generates new music data, and stores the updated music data in cloud storage.

[1824] A system including:

[1825] (Claim 2)

[1826] 10. The system of claim 1, further comprising means for analyzing the user's voice data to extract tempo, pitch, and rhythm.

[1827] (Claim 3)

[1828] 10. The system of claim 1, further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

[1829] "Application Example 1"

[1830] (Claim 1)

[1831] a means for a user to provide voice input;

[1832] a server for receiving the audio data and analyzing the musical characteristics;

[1833] The server generates music in real time based on the analysis results,

[1834] means for transmitting the generated music to a user's device;

[1835] A way for users to share the music they create with other users;

[1836] A means to allow other users to add new parts;

[1837] A means for analyzing voice input and generating operation commands for various devices;

[1838] means for transmitting the generated operation command to an industrial device and causing the industrial device to execute the command;

[1839] A system including:

[1840] (Claim 2)

[1841] 10. The system of claim 1, further comprising means for analyzing the user's voice data to extract tempo, pitch, and rhythm.

[1842] (Claim 3)

[1843] 10. The system of claim 1, further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

[1844] "Example 2: Combining Emotion Engines"

[1845] (Claim 1)

[1846] a means for a user to provide voice input;

[1847] a server for receiving the captured audio data and emotional data and analyzing the musical characteristics and emotional state;

[1848] A means for the server to generate music in real time based on the analysis results and emotional data;

[1849] means for transmitting the generated music to a user's device;

[1850] means for providing a dedicated interface for users to share their created music with other users;

[1851] A means to allow other users to add new parts;

[1852] A system including:

[1853] (Claim 2)

[1854] 10. The system of claim 1, further comprising means for analyzing the user's audio data to extract tempo, pitch, rhythm, and emotional state.

[1855] (Claim 3)

[1856] 10. The system of claim 1, further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

[1857] "Application example 2 when combining emotion engines"

[1858] (Claim 1)

[1859] a means for a user to provide voice input;

[1860] a server for receiving the audio data and analyzing the musical characteristics;

[1861] The server generates music in real time based on the analysis results,

[1862] means for transmitting the generated music to a user's device;

[1863] A way for users to share the music they create with other users;

[1864] A means to allow other users to add new parts;

[1865] A means for analyzing a user's emotions and dynamically generating music based on the results;

[1866] A system including:

[1867] (Claim 2)

[1868] 10. The system of claim 1, further comprising means for analyzing the user's voice data to extract tempo, pitch, and rhythm.

[1869] (Claim 3)

[1870] 10. The system of claim 1, further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

[1871] (Claim 4)

[1872] 10. The system of claim 1, further comprising means for analyzing the user's emotional data and dynamically adjusting the tempo, key, and rhythm of the generated music.

[1873] (Claim 5)

[1874] 10. The system of claim 1, further comprising means for providing real-time feedback of the generated music to the user, allowing the user to perform additional performances.

[1875] (Claim 6)

[1876] 10. The system of claim 1, further comprising means for providing an interface through which a user can share a music file and through which other users can add their own parts to the music file. [Explanation of symbols]

[1877] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to provide voice input; a server for receiving the audio data and analyzing the musical characteristics; The server generates music in real time based on the analysis results, means for transmitting the generated music to a user's device; A way for users to share the music they create with other users; A means to allow other users to add new parts; A system including:

2. 10. The system of claim 1, further comprising means for analyzing the user's voice data and extracting tempo, pitch, and rhythm.

3. The system of claim 1 , further comprising means for simultaneously recording the user's music data and the generated accompaniment data and storing them in cloud storage.

Images