System

Abstract

To provide a system.SOLUTION: A system includes: a prompt generation unit that generates a prompt sentence used for inputting information on a user; a generation unit that uses a generative AI model to generate a list of at least one of musical pieces and musical instruments corresponding to the information on the user from at least one of a musical pieces database and a musical instrument database; a selection unit that displays the generated list and allows the user to select at least one of the musical pieces and the musical instruments; and a providing unit that provides means that allows the user to try to play virtual instruments and interact with characters dancing to the music.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional music learning applications, children had limited options to choose from when learning songs and instruments that matched their preferences and interests. In addition, there were few opportunities to actually try out instruments or dance to music. As a result, children were not able to develop an interest in music or a desire to learn, and a more effective music learning environment was needed. [Means for solving the problem]

[0005] A system according to one aspect of the present disclosure includes an acquisition unit that receives user information and acquires a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database, a selection unit that displays the acquired list and allows the user to select at least one of the songs and instruments, and a provision unit that provides a means for the user to try out virtual instruments and interact with characters that dance to the music. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12]11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0027] An embodiment of the present invention is embodied as a system including an acquisition unit, a selection unit, and a provision unit. This system enables exploration of a music park and learning of music and musical instruments. The acquisition unit receives information about a user and acquires a list of at least one of music and musical instruments corresponding to the user's information from at least one of a music database and a musical instrument database. For example, the list is created by a creation unit that creates a list based on at least one of the user's preferences, interests, age, and gender. The selection unit displays the acquired list and allows the user to select at least one of a music and musical instrument. The provision unit provides a means for the user to try out virtual musical instruments and interact with characters dancing to music. Specifically, the provision unit displays a trial screen corresponding to the musical instrument selected by the user, and includes an output unit that outputs sound in accordance with the performance when the user plays the musical instrument on the trial screen, and a display control unit that controls to display a character dancing to the musical instrument selected by the user. The system may also include a presentation unit that presents a purchase option for purchasing at least one of a musical instrument and a musical instrument that the user likes. Note that each functional unit of the above system is provided in a server or a terminal.

[0028] Here, the "music database" refers to a database that stores and manages information about music (such as song titles, artists, and genres). The "instrument database" refers to a database that stores and manages information about musical instruments (such as the type of instrument, its features, and how to play it). The "trial play screen" refers to a screen for playing virtual musical instruments on the terminal, providing an interactive environment for users to play musical instruments. The "dancing character" refers to a virtual character that dances to the rhythm of the music, allowing users to enjoy playing along with the music. The "purchase options" refer to options and methods for users to actually purchase their favorite musical instruments and songs, allowing them to obtain the actual instruments and songs.

[0029] According to the above system, it is possible to provide a virtual music park where children can have fun while learning songs and musical instruments. Specifically, the server receives user information and creates a list of appropriate songs and musical instruments from a song database and a musical instrument database, thereby providing options that match children's preferences and interests. In addition, the terminal displays the list of songs and musical instruments received from the server, and the user can select a learning content that matches his or her interests by making a selection. In addition, a learning environment that is close to actual performance and experience can be provided by allowing the user to try out virtual musical instruments and interact with characters dancing to music. In addition, by providing the user with the option to actually purchase musical instruments and songs that they like, it is possible to promote actual musical instrument performance and music appreciation as an extension of learning. In this way, it is possible to provide a music learning environment where children can have fun while learning songs and musical instruments that match their preferences and interests.

[0030] The functions of each element, the server, the terminal, and the user, will be described below.

[0031] 1. Server:

[0032] User information is received, and a list of at least one of songs and instruments corresponding to the user information is created from at least one of a song database and an instrument database.

[0033] An algorithm and / or database search technique is implemented to provide a list of songs and / or instruments based on the user's information.

[0034] 2. Terminal:

[0035] A list of at least one of songs and instruments received from the server is displayed, and a user interface is provided for the user to select from.

[0036] A test play screen that matches the instrument selected by the user is displayed, and a function is implemented that outputs and plays a sound in accordance with the instrument when the user plays it.

[0037] Implement a function to display a character dancing to a song selected by the user.

[0038] 3. User:

[0039] Provide the server with information about yourself (favorite music, favorite instruments, interests, age, gender, etc.).

[0040] At least one of a piece of music and an instrument is selected on the terminal, and the user tries playing or dancing to the piece of music and an instrument.

[0041] A purchase option is used to actually purchase at least one of the musical instrument and the song that the user likes.

[0042] <Example>

[0043] For example, if the user is a 5-year-old boy whose favorite instrument is the drums, the server receives the user's information and creates at least one of a song and an instrument related to drums from at least one of a song database and an instrument database. The terminal displays the list received from the server, and when the user selects the drums, a trial screen is displayed. When the user plays the drums, the terminal plays a sound in accordance with the play. Also, when the user selects a song, the terminal displays a character that dances in accordance with the song. After the user has enjoyed trying out the instruments and songs, the user is presented with a purchase option to actually purchase the instrument or song they like, and can actually purchase it.

[0044] The process flow of each embodiment will be described below.

[0045] Step 1: The server gets the user's information.

[0046] The server receives information provided by the user (favorite music, favorite instrument, interests, age, gender, etc.).

[0047] Step 2: The server creates a list of songs and instruments.

[0048] The server creates a list of at least one of songs and instruments suitable for the user from at least one of a song database and an instrument database based on the received user information. Step 3: The terminal displays the list of at least one of songs and instruments.

[0049] The terminal displays the list of at least one of the songs and the instruments received from the server.

[0050] The user can select at least one of a piece of music and an instrument on the terminal.

[0051] Step 4: The user tries out an instrument and dances.

[0052] When the user selects an instrument, the terminal displays a trial screen that matches the selected instrument.

[0053] When the user plays an instrument, the device plays sounds in sync.

[0054] Step 5: The user selects a song and dances.

[0055] When the user selects a song, the device displays a character dancing to the song.

[0056] Step 6: The user purchases an instrument and / or a song.

[0057] If a user finds an instrument or song that interests them, the device will display a purchase option.

[0058] When a user purchases an instrument or piece of music, the instrument or piece of music is actually delivered to them.

[0059] In this way, the server and the device work together to allow users to search for music and instruments, try them out, dance to them, and even purchase the instruments and music they like.

[0060] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0061] In this embodiment, the system may include an emotion engine that recognizes the user's emotion, and a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument according to the user's emotion. The system may also include an implementation unit that implements an algorithm or a data analysis method for recognizing the user's emotion using the emotion engine, and a provision unit that dynamically updates the list according to the user's emotion to provide a learning experience suitable for the user. The system may also include an adjustment unit that adjusts at least one of the playing method and difficulty of the piece of music and the instrument according to the user's emotion, and a change unit that changes at least one of the facial expression and the movement of the dancing character according to the user's emotion. Each functional unit of the above system is provided in a server or a terminal.

[0062] The functions of each element, the server, the terminal, and the user, will be described below.

[0063] 1. Server:

[0064] Implement algorithms and / or data analysis techniques to incorporate the emotion engine.

[0065] To recognize a user's emotion and to suggest at least one of a piece of music and an instrument according to the emotion.

[0066] To dynamically update a list of at least one of music pieces and instruments according to a user's emotions.

[0067] 2. Terminal:

[0068] To adjust at least one of a musical piece and a playing method of an instrument and a difficulty level according to a user's emotion.

[0069] At least one of the facial expression and the movement of the dancing character is changed according to the emotion of the user.

[0070] 3. User:

[0071] At least one of a piece of music and an instrument is selected based on the emotion recognized by the emotion engine.

[0072] Enjoy music, musical instruments, and dancing that match your emotions.

[0073] <Example>

[0074] For example, when a user is using a virtual music park app, the server utilizes an emotion engine to recognize the user's emotions. The emotion engine recognizes the emotions (e.g., joy, excitement, relaxation) that the user feels when playing a piece of music.

[0075] The server dynamically updates the list of music and instruments that match the user's emotions based on the recognized emotion information. For example, if the user is excited, the server will suggest music with a fast tempo and rhythmic instruments such as drums that will increase the excitement.

[0076] The terminal displays a list of songs and instruments corresponding to the emotion received from the server and allows the user to select one. When the user selects a song, the terminal adjusts the playing method and difficulty to match the song selected by the user. Also, when the user selects an instrument, the terminal adjusts the playing method and tone of the instrument to match the emotion of the user.

[0077] Furthermore, when a user dances to music, the device changes the facial expression and movements of the dancing character according to the user's emotions. For example, if the user is relaxed, the character will have a calm facial expression and relaxed movements.

[0078] As described above, the system that combines the emotion engine suggests songs and instruments that match the user's emotions, and adjusts the performance method and dance, providing an even more enjoyable musical experience.

[0079] The process flow of each embodiment will be described below.

[0080] Step 1: The server recognizes the user's emotion

[0081] The server utilizes an emotion engine to recognize the user's emotions.

[0082] The emotion engine recognizes the emotions (joy, excitement, relaxation, etc.) felt by the user while playing a piece of music and transmits them to the server.

[0083] Step 2: The server suggests music and / or instruments according to the emotion.

[0084] The server dynamically updates a list of at least one of music and instruments that match the user's emotion based on the received emotion information.

[0085] For example, if the user is excited, the server may suggest music with a faster tempo or rhythmic instruments such as drums that will increase excitement.

[0086] Step 3: The device displays a list of music and / or instruments corresponding to the emotion.

[0087] The terminal displays a list of at least one of music and instruments corresponding to the emotion received from the server.

[0088] The user can select at least one of a piece of music and an instrument on the terminal.

[0089] Step 4: User selects a song and plays it

[0090] When the user selects a piece of music, the terminal adjusts at least one of the playing method and the difficulty level to suit the piece of music.

[0091] For example, if the user is relaxed, the terminal will simplify the playing method and suggest playing with a relaxed atmosphere.

[0092] Step 5: The user selects and plays an instrument

[0093] When the user selects an instrument, the terminal adjusts the instrument's playing style and tone to match the user's emotions.

[0094] For example, if the user is excited, the terminal will make the playing style more rhythmic and suggest energetic tones.

[0095] Step 6: User dances to the music

[0096] When a user dances to music, the terminal changes the facial expression and movements of the dancing character according to the user's emotions.

[0097] For example, if the user is feeling happy, the character will have a bright expression and active movements.

[0098] As described above, in a system that combines an emotion engine, the server recognizes the user's emotions, and the device suggests music and instruments according to the emotions, and adjusts the playing method and dance. This allows the user to enjoy a music experience that is more tailored to their emotions.

[0099] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0100] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0101] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0104] Conventional music and instrument-related systems make it difficult for users to freely search for and try out musical pieces and instruments, and lack interactive functions to stimulate purchase motivation. In addition, it is difficult to recommend appropriate musical pieces and instruments based on the user's interests and preferences, and there is a need for a means to provide a personalized experience to each user. Another problem is that the process from intuitively expressing the intention to purchase an instrument or piece that has been tried out to actually purchasing it is complicated.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a music database and a musical instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for providing the user with a means for trying out a virtual instrument and interacting with a character dancing to music; an output means for displaying a trial play screen corresponding to the instrument selected by the user and, when the user plays the instrument on the trial play screen, outputting sound in accordance with the performance; and a display means for displaying a dance animation of a character corresponding to the song in accordance with the user's performance. This allows the user to easily search for songs and instruments and smoothly progress through the process to purchase while enjoying trial play and dance animation.

[0107] "User information" refers to personal information of individuals who use the system, including their preferences, interests, age, gender, etc.

[0108] A "music database" is a database that stores and manages information about music, such as song titles, artists, genres, and release dates.

[0109] A "database relating to musical instruments" is a database that stores and manages information relating to musical instruments, such as the type, characteristics, manufacturer, and playing method of the instrument.

[0110] A "list" is a table containing items of music and instruments selected based on user information.

[0111] The "retrieval means" is the part of the system that searches the database for appropriate information based on user input information and generates the list.

[0112] The "selection means" is an interface that displays the acquired list to the user and enables the user to select a song or instrument.

[0113] The "providing means" is a part of the system that provides functions for the user to try out virtual instruments and interact with characters.

[0114] The "test play screen" is a screen that visually displays a virtual musical instrument and allows the user to interactively operate it.

[0115] The "output means" is a part of the system that, when a user plays an instrument on the test playing screen, generates and outputs a sound corresponding to the performance.

[0116] The "display means" is a part of the system that displays a dance animation of a character that corresponds to the music played by the user.

[0117] "Purchase Options" is the part of the system that presents users with the options to select and purchase their favorite songs and instruments.

[0118] The system of the present invention allows users to interactively search for information about music and musical instruments, and to try out and purchase instruments. Specifically, the system works with a database of music and musical instruments based on user information, and generates and displays a list of songs and musical instruments. It also provides a means for users to try out virtual musical instruments and enjoy dance animations that accompany the music, while purchasing musical instruments and songs that they like.

[0119] The server provides a login screen to receive user information and displays an interface for entering email address and password. The hardware used is a general high-performance server, such as an "EC2 instance" from "Amazon Web Services (AWS (registered trademark)." "MySQL (registered trademark)" is used for database management.

[0120] After logging in, the user moves to the music search screen. The device displays a search bar, and when the user enters a song name, artist name, or genre to search, the information is sent to the server. The server queries a music database based on the received search keywords and retrieves the results. The server sends the search results in JSON format to the device, which displays the results to the user in an easy-to-read format.

[0121] For example, when a user searches for "Shape of You," the server retrieves the corresponding song information from the database and displays it on the terminal, along with the instruments the user can select.

[0122] When the user selects an instrument, the device displays a trial screen. The software used is Unity or Unreal Engine, which draws the interface of the virtual instrument. When the user operates the virtual instrument on the screen, the device receives this and generates and outputs the corresponding sound.

[0123] During the trial play process, a character dancing to the music is displayed. The server sends the character data to the device, and the device uses Unity or Unreal Engine to draw the animation. While the user is trying out the music, the character continues to dance to the music.

[0124] If a user finds a song or instrument that they like, they can select the purchase option. The terminal displays a purchase button and sends the purchase information based on the user's selection to the server. The server processes the payment through an e-commerce platform such as Shopify or WooCommerce, and uses a payment gateway such as Stripe.

[0125] Specific examples of prompt sentences are as follows:

[0126] "The system allows users to search for information in a music database and an instrument database, try out virtual instruments, enjoy characters dancing to music, and easily purchase their favorite instruments and songs. Please explain this procedure and the associated process flow in detail."

[0127] In this way, a system can be realized that allows users to easily search for, try out, enjoy, and ultimately purchase music and instruments.

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1: User authentication

[0130] Input: The user enters their email address and password on the login screen.

[0131] How it works: When the user clicks the login button, the device sends this information to the server.

[0132] Data processing: The server receives the user's email address and password and checks them against the authentication information in its database.

[0133] Output: If authentication is successful, the server grants access to the user and displays the home screen.

[0134] Step 2: Search for songs

[0135] Input: A user types a song name, artist name, or genre into the search bar and clicks the Search button.

[0136] Operation: The device sends search keywords to the server.

[0137] Data processing: The server runs a search query against a music database to retrieve song information based on the specified keywords.

[0138] Output: The server sends the search results in JSON format to the terminal, and the terminal displays the retrieved song list on the screen.

[0139] Step 3: Select your instrument

[0140] Input: The user selects the instrument they want to try out from the music search results.

[0141] Operation: The device sends the instrument selection information to the server and loads the trial screen.

[0142] Data processing: The server obtains information about the selected instrument and sends it to the terminal.

[0143] Output: The device displays the virtual instrument in a preview screen.

[0144] Step 4: Trying out the instrument

[0145] Input: The user manipulates a virtual instrument on the preview screen (e.g., pressing a piano key or plucking a guitar string).

[0146] How it works: The device processes user input in real time and generates and outputs corresponding sounds.

[0147] Data processing: Generates audio data based on user operations and plays sound.

[0148] Output: The device will output the generated sound and keep displaying it on the preview screen.

[0149] Step 5: Displaying the dancing character

[0150] Input: The user plays a song.

[0151] Operation: The server sends data of the dance character that corresponds to the song to the terminal.

[0152] Data processing: The device receives the animation data of the dancing character and displays it in time with the music.

[0153] Output: The device displays the character's dance animation on the preview screen.

[0154] Step 6: Checkout

[0155] Input: A user clicks the purchase button on the preview screen to purchase an instrument or piece of music that they like.

[0156] Operation: The device sends the purchase information to the server and displays the payment screen.

[0157] Data processing: The server processes the purchase information and processes the payment through a payment gateway.

[0158] Output: Once the payment is complete, the server sends a notification to the user that the purchase is complete and displays a purchase confirmation screen.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Modern consumers tend to obtain a lot of information through the Internet and make purchasing decisions, but there are limited ways to check the actual feel of musical instruments and music online. In addition, when providing interactive experiences of musical instruments and music, there is a challenge in how to recommend appropriate music and instruments that match the tastes and preferences of the consumer. Furthermore, a system is required that allows consumers to directly purchase musical instruments and music that they like through these experiences.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and musical instruments corresponding to the user information from at least one of a music information database and a musical instrument information database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and musical instruments, a provision means for providing a means for the user to try out virtual musical instruments and interact with virtual characters dancing to music, and a purchase means for purchasing at least one of the musical instruments and songs that the user likes. This allows the user to have an interactive experience of musical instruments and songs online, and further allows the user to purchase products that they like on the spot.

[0164] "User information" refers to personal information such as the user's preferences, hobbies, age, and gender.

[0165] The "music information database" is a database that stores and manages information about songs, such as song titles, artist names, and genres.

[0166] The "musical instrument information database" is a database that stores and manages information about musical instruments, such as the type, characteristics, and playing methods of the musical instruments.

[0167] The "acquisition means" is a function for acquiring relevant information from the music information database or the musical instrument information database based on user information.

[0168] The "selection means" is a function that displays a list of acquired songs and instruments to the user, allowing the user to select from the list.

[0169] The "means for providing" is a function that allows the user to virtually try out an instrument selected by the user, or to have a virtual character dance to the selected music piece.

[0170] "Purchase method" is a function that allows users to directly purchase their favorite instruments and music online.

[0171] The "trial playing screen" is an interactive screen for playing a virtual musical instrument.

[0172] The "virtual characters" are virtual characters that dance to the rhythm of the music.

[0173] "Purchase Options" provide users with the choice to purchase their favorite instruments and songs.

[0174] As an embodiment of the present invention, we provide a "virtual musical instrument store app" that allows users to interactively experience musical instruments and songs and even make purchases. The following is a detailed description of the main components of this app and their functions.

[0175] System Configuration

[0176] Hardware and Software

[0177] Backend: Django framework (Python) is used. The server is built on this framework to communicate with the database and process API requests.

[0178] Front-end: Using React (JavaScript (registered trademark)), we will build a user interface and provide a screen for users to search, select, and try out instruments and songs.

[0179] Database: Uses MySQL. Manages the music information database and instrument information database, and provides data according to user requests.

[0180] Dancing character animation: Using Unity, we generate animations of virtual characters dancing to music.

[0181] API integration: Uses the Spotify API (music information) and a dedicated instrument API. Acquires music information and instrument information in real time and provides it to users.

[0182] Process Overview

[0183] Getting information and displaying lists

[0184] The server first receives information entered by the user (e.g. favorite genre, artist name, type of instrument, etc.). Based on this information, it retrieves the corresponding list from the music information database and instrument information database. The retrieved list is displayed on a front-end screen using React, and is designed to allow the user to make an intuitive selection.

[0185] Virtual Trial and Dancing Characters

[0186] When a user selects a specific instrument or song from the list, that information is sent to the server. The server generates a virtual test screen based on the selected instrument, allowing the user to simulate and play the instrument. In addition, a virtual character generated using Unity dances to the selected song, allowing the user to enjoy a highly entertaining experience.

[0187] Providing purchasing options

[0188] Users are provided with the option to purchase their favorite instruments and songs on the spot. The purchase process is carried out securely through an API provided by the server. This allows users to complete the purchase process with just one click.

[0189] Examples

[0190] User Interaction

[0191] 1. A user types "guitar" into the app's search bar.

[0192] 2. The server queries the instrument information database and displays a list of matching guitars on a screen using React.

[0193] 3. The user selects a particular guitar and begins playing it virtually on the test play screen.

[0194] 4. At the same time, the virtual character will start dancing to the selected song via Unity.

[0195] 5. If the user likes it, they can tap the "Buy" button to purchase the guitar online.

[0196] Example prompts for generative AI models

[0197] "I would like to think of an application that uses a musical instrument information database that includes the types, characteristics, and playing methods of musical instruments, and a music information database that includes song titles, artists, genres, etc. In particular, please give a concrete example of a virtual music store app that integrates an interactive UI that allows users to virtually try out musical instruments on their smartphones, implements dancing characters that dance along with the music, and provides purchasing options."

[0198] This allows users to have an interactive experience with musical instruments and music online, and purchase the products they like on the spot.

[0199] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] The server receives a request from a user. The input includes information about the user's preferences and interests (e.g., favorite genres, artist names, types of instruments, etc.). Based on this information, the server retrieves a list of songs and instruments corresponding to the user from a music information database and an instrument information database. The retrieved data is output as a music list and an instrument list.

[0202] Step 2:

[0203] On the device (such as a smartphone or tablet), the acquired song and instrument lists are displayed in a user interface using React. The user browses the list and selects the song or instrument they like. This selection information is sent as input to the next processing step. The displayed list is interactive, allowing the user to make selections easily.

[0204] Step 3:

[0205] Information on the song and instrument selected by the user is sent to the server. The server receives the information and generates a virtual trial screen for the selected instrument. In addition, it uses Unity to generate a dance animation of a virtual character that matches the selected song. The generated trial screen and animation of the dancing character are output.

[0206] Step 4:

[0207] The terminal displays the generated virtual trial screen and the virtual character's dance animation. The user can interactively play the virtual instrument on the trial screen, and at the same time enjoy watching the virtual character dance to the selected music. The terminal outputs sound in real time according to the user's input (musical instrument playing motion).

[0208] Step 5:

[0209] After the user has tried out the instruments and danced to them, a purchase option is displayed on the device to allow the user to immediately purchase the instrument or song they like. When the user taps the "Purchase" button, the device sends a purchase request to the server. The server receives this request and completes the purchase procedure for the selected instrument or song. When the purchase procedure is complete, a confirmation message is displayed on the device.

[0210] Example 2

[0211] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0212] Conventional music and instrument recommendation systems do not provide appropriate recommendations based on the user's individual preferences and interests, and lack a smooth operation that allows users to immediately purchase music after enjoying playing or listening to it. Therefore, in order to improve the user experience, a new system is needed that seamlessly links music and instrument recommendations based on the user's preferences, virtual play-by-play functionality, and purchase options.

[0213] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments, a playback means for providing an interface for the user to try out a virtual instrument and playing sounds accompanying the trial of the instrument, a display means for displaying an animated character that moves in accordance with the song selected by the user, and a presentation means for presenting purchase options to the user after the user has enjoyed the trial play or animation display. This enables recommendations of songs and instruments based on the individual preferences and interests of the user, a virtual trial experience, and a smooth purchase process.

[0214] A "user" is an individual who uses the system.

[0215] "Information" is data about the user, including favorite music, favorite instruments, interests, age, gender, and so on.

[0216] A "music database" is a database in which information about songs is stored.

[0217] The "instrument database" is a database in which information about musical instruments is stored.

[0218] The "acquisition means" is a means for receiving user information and acquiring a list of songs and instruments from the song database and the instrument database based on the information.

[0219] The "selection means" is a means for displaying the obtained list and allowing the user to select at least one of a piece of music and an instrument.

[0220] A "virtual instrument" is an instrument that does not have a physical existence, but can be played virtually on an interface.

[0221] An "interface" is a means of exchanging information between a user and a system.

[0222] The "playback means" refers to a means for playing back sounds associated with the test playing of a virtual musical instrument.

[0223] An "animated character" is a virtual character that moves in sync with the music.

[0224] "Display means" refers to a means for displaying an animated character on a screen.

[0225] The "presentation means" is a means for presenting a purchase option to a user after the user has enjoyed the content.

[0226] A "list" is a list of songs and instruments corresponding to user information.

[0227] A "purchase option" is a choice that allows a user to purchase at least one of a musical instrument and a piece of music that the user likes.

[0228] In the embodiment of the present invention, the main entities are a user, a terminal, and a server. The program of this system provides recommendations of songs and instruments, trial experiences, and purchase options based on the user's information. The process flow of this system and a specific embodiment are shown below.

[0229] Entering user information

[0230] When a user starts the system, the terminal first displays a screen for inputting user information. The user enters information such as age, gender, favorite musical instrument and music into the form. The terminal receives the input information and sends it to the server.

[0231] Data processing on the server

[0232] The server analyzes the user information received from the device and accesses a music database and an instrument database (e.g., AWS's RDS or Google Cloud's Firestore). It then searches for appropriate data based on the user's preferences and generates a list of music and instruments. The generated list is returned to the device in JSON format.

[0233] Data display and user selection on the terminal

[0234] The device parses the JSON data received from the server and displays a list of songs and instruments to the user (e.g., displaying the list using HTML, CSS, and JavaScript). The user can select the song or instrument they like from the displayed list. For example, "Drum Set A" and "Drum Song B" are displayed.

[0235] Trial Play Function

[0236] Once the user selects an instrument, the device switches to a test-play screen, which displays a virtual instrument (e.g., a touch-activated drum set). When the user plays the instrument on the screen, real sounds are played using the Web Audio API. Each touch input corresponds to a specific sound, providing an interactive test-play experience.

[0237] Music playback and dance display function

[0238] When a user selects a song, the device streams the selected song (e.g., HLS protocol) and displays a screen in which an animated character dances along with the song (e.g., animation control using HTML5 and JavaScript). The character's movements are synchronized with the rhythm of the song.

[0239] Present purchasing options

[0240] After the user has finished trying out the instrument or listening to the music, the device presents them with a purchase option. If the user selects the option in a pop-up, they will be redirected to the purchase page of the EC platform (e.g., Shopify). This allows the user to smoothly purchase the instrument or music.

[0241] As a concrete example, if the user is a 5-year-old boy whose favorite instrument is the drum, the server receives the user information and searches for songs and instruments related to drums from the song database and instrument database. The terminal displays the list, and when the user selects the drum, a trial screen is displayed. When the user plays, the sound is played back, and when a song is selected, a dancing character is displayed. In addition, a purchase option is presented, and the user can purchase an item.

[0242] Here is an example of a prompt to be input to the generative AI model:

[0243] 1. "My 5-year-old boy's favorite instrument is the drums. Can you give me some information about songs and instruments related to drums?"

[0244] 2. "I want an interactive screen to try out drums. Can you tell me how to set that up?"

[0245] 3. "How can I play a song that involves drums and have a character dance along to it?"

[0246] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0247] Step 1:

[0248] The user enters information such as age, gender, favorite musical instrument and music into the input form displayed on the terminal screen. This input form is created using HTML, CSS, and JavaScript. The information entered by the user is temporarily stored on the terminal.

[0249] Input: Basic information entered by the user (age, gender, favorite instruments and songs)

[0250] Output: Get user information

[0251] Specific behavior:

[0252] The user enters information using a tablet or smartphone and taps the "Submit" button.

[0253] Step 2:

[0254] The device sends the information collected from the user to the server using the HTTPS protocol, and the data is encrypted for security reasons.

[0255] Input: User information (age, gender, favorite instruments and songs)

[0256] Output: User information sent to the server

[0257] Specific behavior:

[0258] The device monitors when the "Send" button is tapped and sends the data to the server as a POST request.

[0259] Step 3:

[0260] The server analyzes the user information received from the terminal. Based on the analysis result, it executes queries to the music database and the instrument database. This database access is performed using SQL queries, etc.

[0261] Input: User information submitted

[0262] Output: List of music and instrument data

[0263] Specific behavior:

[0264] The server executes a query and extracts, for example, for a user whose "favorite instrument = drums" data on songs and instruments related to drums.

[0265] Step 4:

[0266] The server returns the list of songs and instruments in JSON format to the terminal. This response is quick, providing data to the user without making them wait.

[0267] Input: List of music and instrument data

[0268] Output: Sending data to the terminal

[0269] Specific behavior:

[0270] The server generates JSON format data and sends it to the terminal as an HTTP response. For example, data is sent in the format {"music": ["drum song A", "drum song B"], "instrument": ["drum set A", "drum set B"]}.

[0271] Step 5:

[0272] The device parses the received JSON data and displays a list of songs and instruments to the user. To do this, it parses the received data with JavaScript and dynamically generates a screen using HTML and CSS.

[0273] Input: JSON data received from the server

[0274] Output: A list of songs and instruments displayed to the user

[0275] Specific behavior:

[0276] The device uses DOM manipulation on the browser to display a list of songs and instruments in a table format or button format, which the user can scroll through to select.

[0277] Step 6:

[0278] The user selects an instrument or a song from a list by tapping the touch screen.

[0279] Input: User selection (e.g. drums)

[0280] Output: ID of selected instrument or piece

[0281] Specific behavior:

[0282] A touch event is fired and visual feedback is provided, such as changing the background color of the selected item.

[0283] Step 7:

[0284] The device displays a trial screen based on the instrument selected by the user. A virtual version of the selected instrument is displayed on the trial screen, and the user can play it by touch operation.

[0285] Input: User's selection information (instrument ID)

[0286] Output: Display of the trial screen

[0287] Specific behavior:

[0288] The demo screen displays a virtual drum set, and sounds are played using the Web Audio API in response to touch controls.

[0289] Step 8:

[0290] When a user tries out a virtual instrument, the device outputs sounds in sync with the performance, such as specific drum sounds that correspond to the touched areas, using the Web Audio API.

[0291] Input: User touch actions

[0292] Output: Playback of the corresponding instrument sound

[0293] Specific behavior:

[0294] When the user taps on different parts of the drum set, sounds such as snare and hi-hat are produced.

[0295] Step 9:

[0296] When a user selects a song from the list, the device plays the song and displays a dancing animated character on the screen. The song data is played in streaming format, and the animation is controlled by JavaScript.

[0297] Input: ID of selected song

[0298] Output: Playing music and displaying animations

[0299] Specific behavior:

[0300] As the music plays, the characters dance to the rhythm.

[0301] Step 10:

[0302] After enjoying the trial and music playback, the device will display a pop-up with purchase options, and if the user wishes to purchase, a link to the purchase page will be generated.

[0303] Input: User action completed

[0304] Output: Purchase option popup

[0305] Specific behavior:

[0306] After the song plays, a pop-up message appears asking, "Do you want to purchase this instrument?", and when the user taps the purchase button, they are redirected to the e-commerce site.

[0307] (Application example 2)

[0308] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Conventional instrument and music selection and purchase systems lacked content that was easy for users to use and enjoy. In addition, because suggestions were not made that reflected the individual preferences and attributes of the user, the user experience was uniform and there was insufficient effort to increase purchasing motivation. Furthermore, virtual trial playing and music experience was limited, making it difficult for users to find products they liked.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0311] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for allowing the user to try out a virtual instrument, displaying a virtual character dancing to music, and presenting an option to purchase the selected instrument or song; a generation means for generating a recommendation list based on the user information using a generative AI model; and a prompt generation means for generating a prompt sentence when inputting user information. This makes it possible to provide suggestions of instruments and songs based on the individual preferences and attributes of the user, thereby improving the quality of the user experience. Furthermore, through the virtual try-out and song experience, the probability that the user will like the product increases, and it is possible to effectively increase the desire to purchase.

[0312] "User Information" is data relating to an individual user, including age, gender, preferred instrument, interests, and other demographic information.

[0313] A "music database" is an information source that stores data on multiple songs, including detailed information about each song.

[0314] An "instrument database" is an information source that stores data on multiple instruments, including detailed information about each instrument.

[0315] The "acquisition means" is a mechanism for acquiring corresponding lists from the music database and the instrument database using user information.

[0316] The "selection means" is a mechanism for displaying the acquired list and allowing the user to select at least one of a piece of music and an instrument.

[0317] A "means for providing" is a mechanism that allows a user to try out virtual instruments, displays a virtual character dancing to music, and presents purchasing options.

[0318] A "generator" is a mechanism for generating a recommendation list based on user information using a generative AI model.

[0319] A "prompt generating means" is a mechanism for generating a prompt sentence to be used when inputting user information.

[0320] A "generative AI model" is a generative model that uses artificial intelligence and makes a variety of suggestions and generates information based on input data.

[0321] A "virtual character" is a digitally generated animated character that is a visual interface through which a user can enjoy a musical experience.

[0322] In order to implement this invention, it is necessary to configure a system using a smartphone, a server, a generative AI model, a database, etc.

[0323] First, the server receives user information (age, gender, preferred instruments, interests, etc.). This information is entered through a smartphone application. At this time, to make it easier for the user to enter the information, a generative AI model is used to generate an appropriate prompt sentence. For example, a prompt may be generated that reads, "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet."

[0324] The server acquires a list of songs and musical instruments corresponding to the user information from at least one of the song database and the musical instrument database by the acquisition means. The acquired list is displayed to the user through the smartphone application, and the user selects a song or musical instrument by the selection means.

[0325] As a means of providing the service, the server provides a function for the user to try out a virtual instrument on the smartphone for the selected instrument or song, and a function for displaying a virtual character dancing to the music. For example, if the user selects a trumpet, a virtual trumpet is displayed on the smartphone screen, and a sound is made when the user taps it. Also, an animation of the character dancing to the selected song is played.

[0326] The generating means causes the server to generate a recommendation list based on the user information using the generative AI model, and the recommendation list is used to suggest musical instruments and songs that match the user's preferences.

[0327] The prompt generating means generates a prompt sentence to support input of user information. For example, if the user is a 5-year-old boy who likes drums, the prompt generating means generates a prompt such as "This 5-year-old boy likes drums. Please generate a list of musical instruments and songs related to drums."

[0328] In addition, by presenting the user with a purchasing option to actually purchase the musical instruments and music they like, the system increases the user's desire to purchase. The smartphone application is designed so that after the user has enjoyed playing and listening to the instruments, they can access the purchase page with one click.

[0329] For example, if an 8-year-old girl likes the trumpet, the application will display items and songs related to the trumpet. The user can tap the trumpet on the smartphone screen to try it out and experience its sound. When a song that accompanies the trumpet is played, the application will show a virtual character dancing happily to the song.

[0330] In this way, the user experience is improved through individual proposals and rich experiences based on user information, resulting in a natural induction into product purchases.

[0331] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0332] Step 1:

[0333] Entering user information

[0334] Users input their individual information (age, gender, favorite instrument, etc.) through a smartphone application. This provides the system with basic data to suggest the most suitable instrument and music piece for the user. The input information is sent to the server. Examples of input data include age: 8 years old, gender: female, favorite instrument: trumpet, etc.

[0335] Step 2:

[0336] Generate prompt statement

[0337] The server uses the generative AI model to generate a prompt sentence based on the user information. The prompt sentence is intended to properly guide the user information and improve the accuracy of the acquired data. It generates a prompt sentence such as "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet." This generated prompt sentence is used to query the database.

[0338] Step 3:

[0339] Obtaining information

[0340] The server queries the music database and instrument database based on the generated prompt text to obtain a list of music and instruments corresponding to the user's information. For example, a list of instruments and music related to "trumpet" is obtained. Through this process, a list suited to the user is generated on the server, and the result is sent to the terminal.

[0341] Step 4:

[0342] Viewing and selecting from the list

[0343] The terminal displays the list of songs and instruments received from the server. The user selects the song or instrument he is interested in from the list. For example, the user selects trumpet and "Trumpet Song A". This selection information is sent back to the server.

[0344] Step 5:

[0345] Try out virtual instruments and display virtual characters

[0346] The server sends the trial data for the selected instrument to the device, and the device displays the virtual instrument trial screen. When the user taps the trumpet on the screen, the sound of an actual trumpet is played. Also, an animation of a virtual character dancing to the selected music piece is displayed. For example, tapping the key of a trumpet produces a "C" sound.

[0347] Step 6:

[0348] Present purchasing options

[0349] After the user has finished playing or listening to the instrument, the server sends the purchase options for the selected instrument or piece to the terminal, and the terminal displays a link that allows the user to purchase the instrument or piece they like with one click. For example, the options "Purchase Trumpet" or "Purchase Piece A" may be presented.

[0350] In this way, a series of processes are carried out, from inputting user information, generating prompts using the generative AI model, retrieving information from the database, playing and displaying the products, and presenting purchasing options. This system allows users to easily select and purchase products while having fun.

[0351] [Second embodiment]

[0352] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0353] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0354] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0355] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0356] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0357] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0358] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0359] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0360] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0361] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0362] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0363] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0364] An embodiment of the present invention is embodied as a system including an acquisition unit, a selection unit, and a provision unit. This system enables exploration of a music park and learning of music and musical instruments. The acquisition unit receives information about a user and acquires a list of at least one of music and musical instruments corresponding to the user's information from at least one of a music database and a musical instrument database. For example, the list is created by a creation unit that creates a list based on at least one of the user's preferences, interests, age, and gender. The selection unit displays the acquired list and allows the user to select at least one of a music and musical instrument. The provision unit provides a means for the user to try out virtual musical instruments and interact with characters dancing to music. Specifically, the provision unit displays a trial screen corresponding to the musical instrument selected by the user, and includes an output unit that outputs sound in accordance with the performance when the user plays the musical instrument on the trial screen, and a display control unit that controls to display a character dancing to the musical instrument selected by the user. The system may also include a presentation unit that presents a purchase option for purchasing at least one of a musical instrument and a musical instrument that the user likes. Note that each functional unit of the above system is provided in a server or a terminal.

[0365] Here, the "music database" refers to a database that stores and manages information about music (such as song titles, artists, and genres). The "instrument database" refers to a database that stores and manages information about musical instruments (such as the type of instrument, its features, and how to play it). The "trial play screen" refers to a screen for playing virtual musical instruments on the terminal, providing an interactive environment for users to play musical instruments. The "dancing character" refers to a virtual character that dances to the rhythm of the music, allowing users to enjoy playing along with the music. The "purchase options" refer to options and methods for users to actually purchase their favorite musical instruments and songs, allowing them to obtain the actual instruments and songs.

[0366] According to the above system, it is possible to provide a virtual music park where children can have fun while learning songs and musical instruments. Specifically, the server receives user information and creates a list of appropriate songs and musical instruments from a song database and a musical instrument database, thereby providing options that match children's preferences and interests. In addition, the terminal displays the list of songs and musical instruments received from the server, and the user can select a learning content that matches his or her interests by making a selection. In addition, a learning environment that is close to actual performance and experience can be provided by allowing the user to try out virtual musical instruments and interact with characters dancing to music. In addition, by providing the user with the option to actually purchase musical instruments and songs that they like, it is possible to promote actual musical instrument performance and music appreciation as an extension of learning. In this way, it is possible to provide a music learning environment where children can have fun while learning songs and musical instruments that match their preferences and interests.

[0367] The functions of each element, the server, the terminal, and the user, will be described below.

[0368] 1. Server:

[0369] User information is received, and a list of at least one of songs and instruments corresponding to the user information is created from at least one of a song database and an instrument database.

[0370] An algorithm and / or database search technique is implemented to provide a list of songs and / or instruments based on the user's information.

[0371] 2. Terminal:

[0372] A list of at least one of songs and instruments received from the server is displayed, and a user interface is provided for the user to select from.

[0373] A test play screen that matches the instrument selected by the user is displayed, and a function is implemented that outputs and plays a sound in accordance with the instrument when the user plays it.

[0374] Implement a function to display a character dancing to a song selected by the user.

[0375] 3. User:

[0376] Provide the server with information about yourself (favorite music, favorite instruments, interests, age, gender, etc.).

[0377] At least one of a piece of music and an instrument is selected on the terminal, and the user tries playing or dancing to the piece of music and an instrument.

[0378] A purchase option is used to actually purchase at least one of the musical instrument and the song that the user likes.

[0379] <Example>

[0380] For example, if the user is a 5-year-old boy whose favorite instrument is the drums, the server receives the user's information and creates at least one of a song and an instrument related to drums from at least one of a song database and an instrument database. The terminal displays the list received from the server, and when the user selects the drums, a trial screen is displayed. When the user plays the drums, the terminal plays a sound in accordance with the play. Also, when the user selects a song, the terminal displays a character that dances in accordance with the song. After the user has enjoyed trying out the instruments and songs, the user is presented with a purchase option to actually purchase the instrument or song they like, and can actually purchase it.

[0381] The process flow of each embodiment will be described below.

[0382] Step 1: The server gets the user's information.

[0383] The server receives information provided by the user (favorite music, favorite instrument, interests, age, gender, etc.).

[0384] Step 2: The server creates a list of songs and instruments.

[0385] The server creates a list of at least one of songs and instruments suitable for the user from at least one of a song database and an instrument database based on the received user information. Step 3: The terminal displays the list of at least one of songs and instruments.

[0386] The terminal displays the list of at least one of the songs and the instruments received from the server.

[0387] The user can select at least one of a piece of music and an instrument on the terminal.

[0388] Step 4: The user tries out an instrument and dances.

[0389] When the user selects an instrument, the terminal displays a trial screen that matches the selected instrument.

[0390] When the user plays an instrument, the device plays sounds in sync.

[0391] Step 5: The user selects a song and dances.

[0392] When the user selects a song, the device displays a character dancing to the song.

[0393] Step 6: The user purchases an instrument and / or a song.

[0394] If a user finds an instrument or song that interests them, the device will display a purchase option.

[0395] When a user purchases an instrument or piece of music, the instrument or piece of music is actually delivered to them.

[0396] In this way, the server and the device work together to allow users to search for music and instruments, try them out, dance to them, and even purchase the instruments and music they like.

[0397] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0398] In this embodiment, the system may include an emotion engine that recognizes the user's emotion, and a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument according to the user's emotion. The system may also include an implementation unit that implements an algorithm or a data analysis method for recognizing the user's emotion using the emotion engine, and a provision unit that dynamically updates the list according to the user's emotion to provide a learning experience suitable for the user. The system may also include an adjustment unit that adjusts at least one of the playing method and difficulty of the piece of music and the instrument according to the user's emotion, and a change unit that changes at least one of the facial expression and the movement of the dancing character according to the user's emotion. Each functional unit of the above system is provided in a server or a terminal.

[0399] The functions of each element, the server, the terminal, and the user, will be described below.

[0400] 1. Server:

[0401] Implement algorithms and / or data analysis techniques to incorporate the emotion engine.

[0402] To recognize a user's emotion and to suggest at least one of a piece of music and an instrument according to the emotion.

[0403] To dynamically update a list of at least one of music pieces and instruments according to a user's emotions.

[0404] 2. Terminal:

[0405] To adjust at least one of a musical piece and a playing method of an instrument and a difficulty level according to a user's emotion.

[0406] At least one of the facial expression and the movement of the dancing character is changed according to the emotion of the user.

[0407] 3. User:

[0408] At least one of a piece of music and an instrument is selected based on the emotion recognized by the emotion engine.

[0409] Enjoy music, musical instruments, and dancing that match your emotions.

[0410] <Example>

[0411] For example, when a user is using a virtual music park app, the server utilizes an emotion engine to recognize the user's emotions. The emotion engine recognizes the emotions (e.g., joy, excitement, relaxation) that the user feels when playing a piece of music.

[0412] The server dynamically updates the list of music and instruments that match the user's emotions based on the recognized emotion information. For example, if the user is excited, the server will suggest music with a fast tempo and rhythmic instruments such as drums that will increase the excitement.

[0413] The terminal displays a list of songs and instruments corresponding to the emotion received from the server and allows the user to select one. When the user selects a song, the terminal adjusts the playing method and difficulty to match the song selected by the user. Also, when the user selects an instrument, the terminal adjusts the playing method and tone of the instrument to match the emotion of the user.

[0414] Furthermore, when a user dances to music, the device changes the facial expression and movements of the dancing character according to the user's emotions. For example, if the user is relaxed, the character will have a calm facial expression and relaxed movements.

[0415] As described above, the system that combines the emotion engine suggests songs and instruments that match the user's emotions, and adjusts the performance method and dance, providing an even more enjoyable musical experience.

[0416] The process flow of each embodiment will be described below.

[0417] Step 1: The server recognizes the user's emotion

[0418] The server utilizes an emotion engine to recognize the user's emotions.

[0419] The emotion engine recognizes the emotions (joy, excitement, relaxation, etc.) felt by the user while playing a piece of music and transmits them to the server.

[0420] Step 2: The server suggests music and / or instruments according to the emotion.

[0421] The server dynamically updates a list of at least one of music and instruments that match the user's emotion based on the received emotion information.

[0422] For example, if the user is excited, the server may suggest music with a faster tempo or rhythmic instruments such as drums that will increase excitement.

[0423] Step 3: The device displays a list of music and / or instruments corresponding to the emotion.

[0424] The terminal displays a list of at least one of music and instruments corresponding to the emotion received from the server.

[0425] The user can select at least one of a piece of music and an instrument on the terminal.

[0426] Step 4: User selects a song and plays it

[0427] When the user selects a piece of music, the terminal adjusts at least one of the playing method and the difficulty level to suit the piece of music.

[0428] For example, if the user is relaxed, the terminal will simplify the playing method and suggest playing with a relaxed atmosphere.

[0429] Step 5: The user selects and plays an instrument

[0430] When the user selects an instrument, the terminal adjusts the instrument's playing style and tone to match the user's emotions.

[0431] For example, if the user is excited, the terminal will make the playing style more rhythmic and suggest energetic tones.

[0432] Step 6: User dances to the music

[0433] When a user dances to music, the terminal changes the facial expression and movements of the dancing character according to the user's emotions.

[0434] For example, if the user is feeling happy, the character will have a bright expression and active movements.

[0435] As described above, in a system that combines an emotion engine, the server recognizes the user's emotions, and the device suggests music and instruments according to the emotions, and adjusts the playing method and dance. This allows the user to enjoy a music experience that is more tailored to their emotions.

[0436] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0437] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0438] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0439] Example 1

[0440] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0441] Conventional music and instrument-related systems make it difficult for users to freely search for and try out musical pieces and instruments, and lack interactive functions to stimulate purchase motivation. In addition, it is difficult to recommend appropriate musical pieces and instruments based on the user's interests and preferences, and there is a need for a means to provide a personalized experience to each user. Another problem is that the process from intuitively expressing the intention to purchase an instrument or piece that has been tried out to actually purchasing it is complicated.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0443] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a music database and a musical instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for providing the user with a means for trying out a virtual instrument and interacting with a character dancing to music; an output means for displaying a trial play screen corresponding to the instrument selected by the user and, when the user plays the instrument on the trial play screen, outputting sound in accordance with the performance; and a display means for displaying a dance animation of a character corresponding to the song in accordance with the user's performance. This allows the user to easily search for songs and instruments and smoothly progress through the process to purchase while enjoying trial play and dance animation.

[0444] "User information" refers to personal information of individuals who use the system, including their preferences, interests, age, gender, etc.

[0445] A "music database" is a database that stores and manages information about music, such as song titles, artists, genres, and release dates.

[0446] A "database relating to musical instruments" is a database that stores and manages information relating to musical instruments, such as the type, characteristics, manufacturer, and playing method of the instrument.

[0447] A "list" is a table containing items of music and instruments selected based on user information.

[0448] The "retrieval means" is the part of the system that searches the database for appropriate information based on user input information and generates the list.

[0449] The "selection means" is an interface that displays the acquired list to the user and enables the user to select a song or instrument.

[0450] The "providing means" is a part of the system that provides functions for the user to try out virtual instruments and interact with characters.

[0451] The "test play screen" is a screen that visually displays a virtual musical instrument and allows the user to interactively operate it.

[0452] The "output means" is a part of the system that, when a user plays an instrument on the test playing screen, generates and outputs a sound corresponding to the performance.

[0453] The "display means" is a part of the system that displays a dance animation of a character that corresponds to the music played by the user.

[0454] "Purchase Options" is the part of the system that presents users with the options to select and purchase their favorite songs and instruments.

[0455] The system of the present invention allows users to interactively search for information about music and musical instruments, and to try out and purchase instruments. Specifically, the system works with a database of music and musical instruments based on user information, and generates and displays a list of songs and musical instruments. It also provides a means for users to try out virtual musical instruments and enjoy dance animations that accompany the music, while purchasing musical instruments and songs that they like.

[0456] The server provides a login screen to receive user information and displays an interface for entering email address and password. The hardware used is a general high-performance server, such as an "EC2 instance" from "Amazon Web Services (AWS)". "MySQL" is used for database management.

[0457] After logging in, the user moves to the music search screen. The device displays a search bar, and when the user enters a song name, artist name, or genre to search, the information is sent to the server. The server queries a music database based on the received search keywords and retrieves the results. The server sends the search results in JSON format to the device, which displays the results to the user in an easy-to-read format.

[0458] For example, when a user searches for "Shape of You," the server retrieves the corresponding song information from the database and displays it on the terminal, along with the instruments the user can select.

[0459] When the user selects an instrument, the device displays a trial screen. The software used is Unity or Unreal Engine, which draws the interface of the virtual instrument. When the user operates the virtual instrument on the screen, the device receives this and generates and outputs the corresponding sound.

[0460] During the trial play process, a character dancing to the music is displayed. The server sends the character data to the device, and the device uses Unity or Unreal Engine to draw the animation. While the user is trying out the music, the character continues to dance to the music.

[0461] If a user finds a song or instrument that they like, they can select the purchase option. The terminal displays a purchase button and sends the purchase information based on the user's selection to the server. The server processes the payment through an e-commerce platform such as Shopify or WooCommerce, and uses a payment gateway such as Stripe.

[0462] Specific examples of prompt sentences are as follows:

[0463] "The system allows users to search for information in a music database and an instrument database, try out virtual instruments, enjoy characters dancing to music, and easily purchase their favorite instruments and songs. Please explain this procedure and the associated process flow in detail."

[0464] In this way, a system can be realized that allows users to easily search for, try out, enjoy, and ultimately purchase music and instruments.

[0465] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0466] Step 1: User authentication

[0467] Input: The user enters their email address and password on the login screen.

[0468] How it works: When the user clicks the login button, the device sends this information to the server.

[0469] Data processing: The server receives the user's email address and password and checks them against the authentication information in its database.

[0470] Output: If authentication is successful, the server grants access to the user and displays the home screen.

[0471] Step 2: Search for songs

[0472] Input: A user types a song name, artist name, or genre into the search bar and clicks the Search button.

[0473] Operation: The device sends search keywords to the server.

[0474] Data processing: The server runs a search query against a music database to retrieve song information based on the specified keywords.

[0475] Output: The server sends the search results in JSON format to the terminal, and the terminal displays the retrieved song list on the screen.

[0476] Step 3: Select your instrument

[0477] Input: The user selects the instrument they want to try out from the music search results.

[0478] Operation: The device sends the instrument selection information to the server and loads the trial screen.

[0479] Data processing: The server obtains information about the selected instrument and sends it to the terminal.

[0480] Output: The device displays the virtual instrument in a preview screen.

[0481] Step 4: Trying out the instrument

[0482] Input: The user manipulates a virtual instrument on the preview screen (e.g., pressing a piano key or plucking a guitar string).

[0483] How it works: The device processes user input in real time and generates and outputs corresponding sounds.

[0484] Data processing: Generates audio data based on user operations and plays sound.

[0485] Output: The device will output the generated sound and keep displaying it on the preview screen.

[0486] Step 5: Displaying the dancing character

[0487] Input: The user plays a song.

[0488] Operation: The server sends data of the dance character that corresponds to the song to the terminal.

[0489] Data processing: The device receives the animation data of the dancing character and displays it in time with the music.

[0490] Output: The device displays the character's dance animation on the preview screen.

[0491] Step 6: Checkout

[0492] Input: A user clicks the purchase button on the preview screen to purchase an instrument or piece of music that they like.

[0493] Operation: The device sends the purchase information to the server and displays the payment screen.

[0494] Data processing: The server processes the purchase information and processes the payment through a payment gateway.

[0495] Output: Once the payment is complete, the server sends a notification to the user that the purchase is complete and displays a purchase confirmation screen.

[0496] (Application example 1)

[0497] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0498] Modern consumers tend to obtain a lot of information through the Internet and make purchasing decisions, but there are limited ways to check the actual feel of musical instruments and music online. In addition, when providing interactive experiences of musical instruments and music, there is a challenge in how to recommend appropriate music and instruments that match the tastes and preferences of the consumer. Furthermore, a system is required that allows consumers to directly purchase musical instruments and music that they like through these experiences.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0500] In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and musical instruments corresponding to the user information from at least one of a music information database and a musical instrument information database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and musical instruments, a provision means for providing a means for the user to try out virtual musical instruments and interact with virtual characters dancing to music, and a purchase means for purchasing at least one of the musical instruments and songs that the user likes. This allows the user to have an interactive experience of musical instruments and songs online, and further allows the user to purchase products that they like on the spot.

[0501] "User information" refers to personal information such as the user's preferences, hobbies, age, and gender.

[0502] The "music information database" is a database that stores and manages information about songs, such as song titles, artist names, and genres.

[0503] The "musical instrument information database" is a database that stores and manages information about musical instruments, such as the type, characteristics, and playing methods of the musical instruments.

[0504] The "acquisition means" is a function for acquiring relevant information from the music information database or the musical instrument information database based on user information.

[0505] The "selection means" is a function that displays a list of acquired songs and instruments to the user, allowing the user to select from the list.

[0506] The "means for providing" is a function that allows the user to virtually try out an instrument selected by the user, or to have a virtual character dance to the selected music piece.

[0507] "Purchase method" is a function that allows users to directly purchase their favorite instruments and music online.

[0508] The "trial playing screen" is an interactive screen for playing a virtual musical instrument.

[0509] The "virtual characters" are virtual characters that dance to the rhythm of the music.

[0510] "Purchase Options" provide users with the choice to purchase their favorite instruments and songs.

[0511] As an embodiment of the present invention, we provide a "virtual musical instrument store app" that allows users to interactively experience musical instruments and songs and even make purchases. The following is a detailed description of the main components of this app and their functions.

[0512] System Configuration

[0513] Hardware and Software

[0514] Backend: Django framework (Python) is used. The server is built on this framework to communicate with the database and process API requests.

[0515] Frontend: Using React (JavaScript), we will build the user interface and provide a screen for users to search, select, and try out instruments and songs.

[0516] Database: Uses MySQL. Manages the music information database and instrument information database, and provides data according to user requests.

[0517] Dancing character animation: Using Unity, we generate animations of virtual characters dancing to music.

[0518] API integration: Uses the Spotify API (music information) and a dedicated instrument API. Acquires music information and instrument information in real time and provides it to users.

[0519] Process Overview

[0520] Getting information and displaying lists

[0521] The server first receives information entered by the user (e.g. favorite genre, artist name, type of instrument, etc.). Based on this information, it retrieves the corresponding list from the music information database and instrument information database. The retrieved list is displayed on a front-end screen using React, and is designed to allow the user to make an intuitive selection.

[0522] Virtual Trial and Dancing Characters

[0523] When a user selects a specific instrument or song from the list, that information is sent to the server. The server generates a virtual test screen based on the selected instrument, allowing the user to simulate and play the instrument. In addition, a virtual character generated using Unity dances to the selected song, allowing the user to enjoy a highly entertaining experience.

[0524] Providing purchasing options

[0525] Users are provided with the option to purchase their favorite instruments and songs on the spot. The purchase process is carried out securely through an API provided by the server. This allows users to complete the purchase process with just one click.

[0526] Examples

[0527] User Interaction

[0528] 1. A user types "guitar" into the app's search bar.

[0529] 2. The server queries the instrument information database and displays a list of matching guitars on a screen using React.

[0530] 3. The user selects a particular guitar and begins playing it virtually on the test play screen.

[0531] 4. At the same time, the virtual character will start dancing to the selected song via Unity.

[0532] 5. If the user likes it, they can tap the "Buy" button to purchase the guitar online.

[0533] Example prompts for generative AI models

[0534] "I would like to think of an application that uses a musical instrument information database that includes the types, characteristics, and playing methods of musical instruments, and a music information database that includes song titles, artists, genres, etc. In particular, please give a concrete example of a virtual music store app that integrates an interactive UI that allows users to virtually try out musical instruments on their smartphones, implements dancing characters that dance along with the music, and provides purchasing options."

[0535] This allows users to have an interactive experience with musical instruments and music online, and purchase the products they like on the spot.

[0536] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0537] Step 1:

[0538] The server receives a request from a user. The input includes information about the user's preferences and interests (e.g., favorite genres, artist names, types of instruments, etc.). Based on this information, the server retrieves a list of songs and instruments corresponding to the user from a music information database and an instrument information database. The retrieved data is output as a music list and an instrument list.

[0539] Step 2:

[0540] On the device (such as a smartphone or tablet), the acquired song and instrument lists are displayed in a user interface using React. The user browses the list and selects the song or instrument they like. This selection information is sent as input to the next processing step. The displayed list is interactive, allowing the user to make selections easily.

[0541] Step 3:

[0542] Information on the song and instrument selected by the user is sent to the server. The server receives the information and generates a virtual trial screen for the selected instrument. In addition, it uses Unity to generate a dance animation of a virtual character that matches the selected song. The generated trial screen and animation of the dancing character are output.

[0543] Step 4:

[0544] The terminal displays the generated virtual trial screen and the virtual character's dance animation. The user can interactively play the virtual instrument on the trial screen, and at the same time enjoy watching the virtual character dance to the selected music. The terminal outputs sound in real time according to the user's input (musical instrument playing motion).

[0545] Step 5:

[0546] After the user has tried out the instruments and danced to them, a purchase option is displayed on the device to allow the user to immediately purchase the instrument or song they like. When the user taps the "Purchase" button, the device sends a purchase request to the server. The server receives this request and completes the purchase procedure for the selected instrument or song. When the purchase procedure is complete, a confirmation message is displayed on the device.

[0547] Example 2

[0548] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0549] Conventional music and instrument recommendation systems do not provide appropriate recommendations based on the user's individual preferences and interests, and lack a smooth operation that allows users to immediately purchase music after enjoying playing or listening to it. Therefore, in order to improve the user experience, a new system is needed that seamlessly links music and instrument recommendations based on the user's preferences, virtual play-by-play functionality, and purchase options.

[0550] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments, a playback means for providing an interface for the user to try out a virtual instrument and playing sounds accompanying the trial of the instrument, a display means for displaying an animated character that moves in accordance with the song selected by the user, and a presentation means for presenting purchase options to the user after the user has enjoyed the trial play or animation display. This enables recommendations of songs and instruments based on the individual preferences and interests of the user, a virtual trial experience, and a smooth purchase process.

[0551] A "user" is an individual who uses the system.

[0552] "Information" is data about the user, including favorite music, favorite instruments, interests, age, gender, and so on.

[0553] A "music database" is a database in which information about songs is stored.

[0554] The "instrument database" is a database in which information about musical instruments is stored.

[0555] The "acquisition means" is a means for receiving user information and acquiring a list of songs and instruments from the song database and the instrument database based on the information.

[0556] The "selection means" is a means for displaying the obtained list and allowing the user to select at least one of a piece of music and an instrument.

[0557] A "virtual instrument" is an instrument that does not have a physical existence, but can be played virtually on an interface.

[0558] An "interface" is a means of exchanging information between a user and a system.

[0559] The "playback means" refers to a means for playing back sounds associated with the test playing of a virtual musical instrument.

[0560] An "animated character" is a virtual character that moves in sync with the music.

[0561] "Display means" refers to a means for displaying an animated character on a screen.

[0562] The "presentation means" is a means for presenting a purchase option to a user after the user has enjoyed the content.

[0563] A "list" is a list of songs and instruments corresponding to user information.

[0564] A "purchase option" is a choice that allows a user to purchase at least one of a musical instrument and a piece of music that the user likes.

[0565] In the embodiment of the present invention, the main entities are a user, a terminal, and a server. The program of this system provides recommendations of songs and instruments, trial experiences, and purchase options based on the user's information. The process flow of this system and a specific embodiment are shown below.

[0566] Entering user information

[0567] When a user starts the system, the terminal first displays a screen for inputting user information. The user enters information such as age, gender, favorite musical instrument and music into the form. The terminal receives the input information and sends it to the server.

[0568] Data processing on the server

[0569] The server analyzes the user information received from the device and accesses a music database and an instrument database (e.g., AWS's RDS or Google Cloud's Firestore). It then searches for the appropriate data based on the user's preferences and generates a list of songs and instruments. The generated list is returned to the device in JSON format.

[0570] Data display and user selection on the terminal

[0571] The device parses the JSON data received from the server and displays a list of songs and instruments to the user (e.g., displaying the list using HTML, CSS, and JavaScript). The user can select the song or instrument they like from the displayed list. For example, "Drum Set A" and "Drum Song B" are displayed.

[0572] Trial Play Function

[0573] Once the user selects an instrument, the device switches to a test-play screen, which displays a virtual instrument (e.g., a touch-activated drum set). When the user plays the instrument on the screen, real sounds are played using the Web Audio API. Each touch input corresponds to a specific sound, providing an interactive test-play experience.

[0574] Music playback and dance display function

[0575] When a user selects a song, the device streams the selected song (e.g., HLS protocol) and displays a screen in which an animated character dances along with the song (e.g., animation control using HTML5 and JavaScript). The character's movements are synchronized with the rhythm of the song.

[0576] Present purchasing options

[0577] After the user has finished trying out the instrument or listening to the music, the device presents them with a purchase option. If the user selects the option in a pop-up, they will be redirected to the purchase page of the EC platform (e.g., Shopify). This allows the user to smoothly purchase the instrument or music.

[0578] As a concrete example, if the user is a 5-year-old boy whose favorite instrument is the drum, the server receives the user information and searches for songs and instruments related to drums from the song database and instrument database. The terminal displays the list, and when the user selects the drum, a trial screen is displayed. When the user plays, the sound is played back, and when a song is selected, a dancing character is displayed. In addition, a purchase option is presented, and the user can purchase an item.

[0579] Here is an example of a prompt to be input to the generative AI model:

[0580] 1. "My 5-year-old boy's favorite instrument is the drums. Can you give me some information about songs and instruments related to drums?"

[0581] 2. "I want an interactive screen to try out drums. Can you tell me how to set that up?"

[0582] 3. "How can I play a song that involves drums and have a character dance along to it?"

[0583] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0584] Step 1:

[0585] The user enters information such as age, gender, favorite musical instrument and music into the input form displayed on the terminal screen. This input form is created using HTML, CSS, and JavaScript. The information entered by the user is temporarily stored on the terminal.

[0586] Input: Basic information entered by the user (age, gender, favorite instruments and songs)

[0587] Output: Get user information

[0588] Specific behavior:

[0589] The user enters information using a tablet or smartphone and taps the "Submit" button.

[0590] Step 2:

[0591] The device sends the information collected from the user to the server using the HTTPS protocol, and the data is encrypted for security reasons.

[0592] Input: User information (age, gender, favorite instruments and songs)

[0593] Output: User information sent to the server

[0594] Specific behavior:

[0595] The device monitors when the "Send" button is tapped and sends the data to the server as a POST request.

[0596] Step 3:

[0597] The server analyzes the user information received from the terminal. Based on the analysis result, it executes queries to the music database and the instrument database. This database access is performed using SQL queries, etc.

[0598] Input: User information submitted

[0599] Output: List of music and instrument data

[0600] Specific behavior:

[0601] The server executes a query and extracts, for example, for a user whose "favorite instrument = drums" data on songs and instruments related to drums.

[0602] Step 4:

[0603] The server returns the list of songs and instruments in JSON format to the terminal. This response is quick, providing data to the user without making them wait.

[0604] Input: List of music and instrument data

[0605] Output: Sending data to the terminal

[0606] Specific behavior:

[0607] The server generates JSON format data and sends it to the terminal as an HTTP response. For example, data is sent in the format {"music": ["drum song A", "drum song B"], "instrument": ["drum set A", "drum set B"]}.

[0608] Step 5:

[0609] The device parses the received JSON data and displays a list of songs and instruments to the user. To do this, it parses the received data with JavaScript and dynamically generates a screen using HTML and CSS.

[0610] Input: JSON data received from the server

[0611] Output: A list of songs and instruments displayed to the user

[0612] Specific behavior:

[0613] The device uses DOM manipulation on the browser to display a list of songs and instruments in a table format or button format, which the user can scroll through to select.

[0614] Step 6:

[0615] The user selects an instrument or a song from a list by tapping the touch screen.

[0616] Input: User selection (e.g. drums)

[0617] Output: ID of selected instrument or piece

[0618] Specific behavior:

[0619] A touch event is fired and visual feedback is provided, such as changing the background color of the selected item.

[0620] Step 7:

[0621] The device displays a trial screen based on the instrument selected by the user. A virtual version of the selected instrument is displayed on the trial screen, and the user can play it by touch operation.

[0622] Input: User's selection information (instrument ID)

[0623] Output: Display of the trial screen

[0624] Specific behavior:

[0625] The demo screen displays a virtual drum set, and sounds are played using the Web Audio API in response to touch controls.

[0626] Step 8:

[0627] When a user tries out a virtual instrument, the device outputs sounds in sync with the performance, such as specific drum sounds that correspond to the touched areas, using the Web Audio API.

[0628] Input: User touch actions

[0629] Output: Playback of the corresponding instrument sound

[0630] Specific behavior:

[0631] When the user taps on different parts of the drum set, sounds such as snare and hi-hat are produced.

[0632] Step 9:

[0633] When a user selects a song from the list, the device plays the song and displays a dancing animated character on the screen. The song data is played in streaming format, and the animation is controlled by JavaScript.

[0634] Input: ID of selected song

[0635] Output: Playing music and displaying animations

[0636] Specific behavior:

[0637] As the music plays, the characters dance to the rhythm.

[0638] Step 10:

[0639] After enjoying the trial and music playback, the device will display a pop-up with purchase options, and if the user wishes to purchase, a link to the purchase page will be generated.

[0640] Input: User action completed

[0641] Output: Purchase option popup

[0642] Specific behavior:

[0643] After the song plays, a pop-up message appears asking, "Do you want to purchase this instrument?", and when the user taps the purchase button, they are redirected to the e-commerce site.

[0644] (Application example 2)

[0645] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0646] Conventional instrument and music selection and purchase systems lacked content that was easy for users to use and enjoy. In addition, because suggestions were not made that reflected the individual preferences and attributes of the user, the user experience was uniform and there was insufficient effort to increase purchasing motivation. Furthermore, virtual trial playing and music experience was limited, making it difficult for users to find products they liked.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0648] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for allowing the user to try out a virtual instrument, displaying a virtual character dancing to music, and presenting an option to purchase the selected instrument or song; a generation means for generating a recommendation list based on the user information using a generative AI model; and a prompt generation means for generating a prompt sentence when inputting user information. This makes it possible to provide suggestions of instruments and songs based on the individual preferences and attributes of the user, thereby improving the quality of the user experience. Furthermore, through the virtual try-out and song experience, the probability that the user will like the product increases, and it is possible to effectively increase the desire to purchase.

[0649] "User Information" is data relating to an individual user, including age, gender, preferred instrument, interests, and other demographic information.

[0650] A "music database" is an information source that stores data on multiple songs, including detailed information about each song.

[0651] An "instrument database" is an information source that stores data on multiple instruments, including detailed information about each instrument.

[0652] The "acquisition means" is a mechanism for acquiring corresponding lists from the music database and the instrument database using user information.

[0653] The "selection means" is a mechanism for displaying the acquired list and allowing the user to select at least one of a piece of music and an instrument.

[0654] A "means for providing" is a mechanism that allows a user to try out virtual instruments, displays a virtual character dancing to music, and presents purchasing options.

[0655] A "generator" is a mechanism for generating a recommendation list based on user information using a generative AI model.

[0656] A "prompt generating means" is a mechanism for generating a prompt sentence to be used when inputting user information.

[0657] A "generative AI model" is a generative model that uses artificial intelligence and makes a variety of suggestions and generates information based on input data.

[0658] A "virtual character" is a digitally generated animated character that is a visual interface through which a user can enjoy a musical experience.

[0659] In order to implement this invention, it is necessary to configure a system using a smartphone, a server, a generative AI model, a database, etc.

[0660] First, the server receives user information (age, gender, preferred instruments, interests, etc.). This information is entered through a smartphone application. At this time, to make it easier for the user to enter the information, a generative AI model is used to generate an appropriate prompt sentence. For example, a prompt may be generated that reads, "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet."

[0661] The server acquires a list of songs and musical instruments corresponding to the user information from at least one of the song database and the musical instrument database by the acquisition means. The acquired list is displayed to the user through the smartphone application, and the user selects a song or musical instrument by the selection means.

[0662] As a means of providing the service, the server provides a function for the user to try out a virtual instrument on the smartphone for the selected instrument or song, and a function for displaying a virtual character dancing to the music. For example, if the user selects a trumpet, a virtual trumpet is displayed on the smartphone screen, and a sound is made when the user taps it. Also, an animation of the character dancing to the selected song is played.

[0663] The generating means causes the server to generate a recommendation list based on the user information using the generative AI model, and the recommendation list is used to suggest musical instruments and songs that match the user's preferences.

[0664] The prompt generating means generates a prompt sentence to support input of user information. For example, if the user is a 5-year-old boy who likes drums, the prompt generating means generates a prompt such as "This 5-year-old boy likes drums. Please generate a list of musical instruments and songs related to drums."

[0665] In addition, by presenting the user with a purchasing option to actually purchase the musical instruments and music they like, the system increases the user's desire to purchase. The smartphone application is designed so that after the user has enjoyed playing and listening to the instruments, they can access the purchase page with one click.

[0666] For example, if an 8-year-old girl likes the trumpet, the application will display items and songs related to the trumpet. The user can tap the trumpet on the smartphone screen to try it out and experience its sound. When a song that accompanies the trumpet is played, the application will show a virtual character dancing happily to the song.

[0667] In this way, the user experience is improved through individual proposals and rich experiences based on user information, resulting in a natural induction into product purchases.

[0668] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0669] Step 1:

[0670] Entering user information

[0671] Users input their individual information (age, gender, favorite instrument, etc.) through a smartphone application. This provides the system with basic data to suggest the most suitable instrument and music piece for the user. The input information is sent to the server. Examples of input data include age: 8 years old, gender: female, favorite instrument: trumpet, etc.

[0672] Step 2:

[0673] Generate prompt statement

[0674] The server uses the generative AI model to generate a prompt sentence based on the user information. The prompt sentence is intended to properly guide the user information and improve the accuracy of the acquired data. It generates a prompt sentence such as "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet." This generated prompt sentence is used to query the database.

[0675] Step 3:

[0676] Obtaining information

[0677] The server queries the music database and instrument database based on the generated prompt text to obtain a list of music and instruments corresponding to the user's information. For example, a list of instruments and music related to "trumpet" is obtained. Through this process, a list suited to the user is generated on the server, and the result is sent to the terminal.

[0678] Step 4:

[0679] Viewing and selecting from the list

[0680] The terminal displays the list of songs and instruments received from the server. The user selects the song or instrument he is interested in from the list. For example, the user selects trumpet and "Trumpet Song A". This selection information is sent back to the server.

[0681] Step 5:

[0682] Try out virtual instruments and display virtual characters

[0683] The server sends the trial data for the selected instrument to the device, and the device displays the virtual instrument trial screen. When the user taps the trumpet on the screen, the sound of an actual trumpet is played. Also, an animation of a virtual character dancing to the selected music piece is displayed. For example, tapping the key of a trumpet produces a "C" sound.

[0684] Step 6:

[0685] Present purchasing options

[0686] After the user has finished playing or listening to the instrument, the server sends the purchase options for the selected instrument or piece to the terminal, and the terminal displays a link that allows the user to purchase the instrument or piece they like with one click. For example, the options "Purchase Trumpet" or "Purchase Piece A" may be presented.

[0687] In this way, a series of processes are carried out, from inputting user information, generating prompts using the generative AI model, retrieving information from the database, playing and displaying the products, and presenting purchasing options. This system allows users to easily select and purchase products while having fun.

[0688] [Third embodiment]

[0689] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0690] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0691] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0692] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0693] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0694] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0695] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0696] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0697] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0698] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0699] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0700] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0701] An embodiment of the present invention is embodied as a system including an acquisition unit, a selection unit, and a provision unit. This system enables exploration of a music park and learning of music and musical instruments. The acquisition unit receives information about a user and acquires a list of at least one of music and musical instruments corresponding to the user's information from at least one of a music database and a musical instrument database. For example, the list is created by a creation unit that creates a list based on at least one of the user's preferences, interests, age, and gender. The selection unit displays the acquired list and allows the user to select at least one of a music and musical instrument. The provision unit provides a means for the user to try out virtual musical instruments and interact with characters dancing to music. Specifically, the provision unit displays a trial screen corresponding to the musical instrument selected by the user, and includes an output unit that outputs sound in accordance with the performance when the user plays the musical instrument on the trial screen, and a display control unit that controls to display a character dancing to the musical instrument selected by the user. The system may also include a presentation unit that presents a purchase option for purchasing at least one of a musical instrument and a musical instrument that the user likes. Note that each functional unit of the above system is provided in a server or a terminal.

[0702] Here, the "music database" refers to a database that stores and manages information about music (such as song titles, artists, and genres). The "instrument database" refers to a database that stores and manages information about musical instruments (such as the type of instrument, its features, and how to play it). The "trial play screen" refers to a screen for playing virtual musical instruments on the terminal, providing an interactive environment for users to play musical instruments. The "dancing character" refers to a virtual character that dances to the rhythm of the music, allowing users to enjoy playing along with the music. The "purchase options" refer to options and methods for users to actually purchase their favorite musical instruments and songs, allowing them to obtain the actual instruments and songs.

[0703] According to the above system, it is possible to provide a virtual music park where children can have fun while learning songs and musical instruments. Specifically, the server receives user information and creates a list of appropriate songs and musical instruments from a song database and a musical instrument database, thereby providing options that match children's preferences and interests. In addition, the terminal displays the list of songs and musical instruments received from the server, and the user can select a learning content that matches his or her interests by making a selection. In addition, a learning environment that is close to actual performance and experience can be provided by allowing the user to try out virtual musical instruments and interact with characters dancing to music. In addition, by providing the user with the option to actually purchase musical instruments and songs that they like, it is possible to promote actual musical instrument performance and music appreciation as an extension of learning. In this way, it is possible to provide a music learning environment where children can have fun while learning songs and musical instruments that match their preferences and interests.

[0704] The functions of each element, the server, the terminal, and the user, will be described below.

[0705] 1. Server:

[0706] User information is received, and a list of at least one of songs and instruments corresponding to the user information is created from at least one of a song database and an instrument database.

[0707] An algorithm and / or database search technique is implemented to provide a list of songs and / or instruments based on the user's information.

[0708] 2. Terminal:

[0709] A list of at least one of songs and instruments received from the server is displayed, and a user interface is provided for the user to select from.

[0710] A test play screen that matches the instrument selected by the user is displayed, and a function is implemented that outputs and plays a sound in accordance with the instrument when the user plays it.

[0711] Implement a function to display a character dancing to a song selected by the user.

[0712] 3. User:

[0713] Provide the server with information about yourself (favorite music, favorite instruments, interests, age, gender, etc.).

[0714] At least one of a piece of music and an instrument is selected on the terminal, and the user tries playing or dancing to the piece of music and an instrument.

[0715] A purchase option is used to actually purchase at least one of the musical instrument and the song that the user likes.

[0716] <Example>

[0717] For example, if the user is a 5-year-old boy whose favorite instrument is the drums, the server receives the user's information and creates at least one of a song and an instrument related to drums from at least one of a song database and an instrument database. The terminal displays the list received from the server, and when the user selects the drums, a trial screen is displayed. When the user plays the drums, the terminal plays a sound in accordance with the play. Also, when the user selects a song, the terminal displays a character that dances in accordance with the song. After the user has enjoyed trying out the instruments and songs, the user is presented with a purchase option to actually purchase the instrument or song they like, and can actually purchase it.

[0718] The process flow of each embodiment will be described below.

[0719] Step 1: The server gets the user's information.

[0720] The server receives information provided by the user (favorite music, favorite instrument, interests, age, gender, etc.).

[0721] Step 2: The server creates a list of songs and instruments.

[0722] The server creates a list of at least one of songs and instruments suitable for the user from at least one of a song database and an instrument database based on the received user information. Step 3: The terminal displays the list of at least one of songs and instruments.

[0723] The terminal displays the list of at least one of the songs and the instruments received from the server.

[0724] The user can select at least one of a piece of music and an instrument on the terminal.

[0725] Step 4: The user tries out an instrument and dances.

[0726] When the user selects an instrument, the terminal displays a trial screen that matches the selected instrument.

[0727] When the user plays an instrument, the device plays sounds in sync.

[0728] Step 5: The user selects a song and dances.

[0729] When the user selects a song, the device displays a character dancing to the song.

[0730] Step 6: The user purchases an instrument and / or a song.

[0731] If a user finds an instrument or song that interests them, the device will display a purchase option.

[0732] When a user purchases an instrument or piece of music, the instrument or piece of music is actually delivered to them.

[0733] In this way, the server and the device work together to allow users to search for music and instruments, try them out, dance to them, and even purchase the instruments and music they like.

[0734] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0735] In this embodiment, the system may include an emotion engine that recognizes the user's emotion, and a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument according to the user's emotion. The system may also include an implementation unit that implements an algorithm or a data analysis method for recognizing the user's emotion using the emotion engine, and a provision unit that dynamically updates the list according to the user's emotion to provide a learning experience suitable for the user. The system may also include an adjustment unit that adjusts at least one of the playing method and difficulty of the piece of music and the instrument according to the user's emotion, and a change unit that changes at least one of the facial expression and the movement of the dancing character according to the user's emotion. Each functional unit of the above system is provided in a server or a terminal.

[0736] The functions of each element, the server, the terminal, and the user, will be described below.

[0737] 1. Server:

[0738] Implement algorithms and / or data analysis techniques to incorporate the emotion engine.

[0739] To recognize a user's emotion and to suggest at least one of a piece of music and an instrument according to the emotion.

[0740] To dynamically update a list of at least one of music pieces and instruments according to a user's emotions.

[0741] 2. Terminal:

[0742] To adjust at least one of a musical piece and a playing method of an instrument and a difficulty level according to a user's emotion.

[0743] At least one of the facial expression and the movement of the dancing character is changed according to the emotion of the user.

[0744] 3. User:

[0745] At least one of a piece of music and an instrument is selected based on the emotion recognized by the emotion engine.

[0746] Enjoy music, musical instruments, and dancing that match your emotions.

[0747] <Example>

[0748] For example, when a user is using a virtual music park app, the server utilizes an emotion engine to recognize the user's emotions. The emotion engine recognizes the emotions (e.g., joy, excitement, relaxation) that the user feels when playing a piece of music.

[0749] The server dynamically updates the list of music and instruments that match the user's emotions based on the recognized emotion information. For example, if the user is excited, the server will suggest music with a fast tempo and rhythmic instruments such as drums that will increase the excitement.

[0750] The terminal displays a list of songs and instruments corresponding to the emotion received from the server and allows the user to select one. When the user selects a song, the terminal adjusts the playing method and difficulty to match the song selected by the user. Also, when the user selects an instrument, the terminal adjusts the playing method and tone of the instrument to match the emotion of the user.

[0751] Furthermore, when a user dances to music, the device changes the facial expression and movements of the dancing character according to the user's emotions. For example, if the user is relaxed, the character will have a calm facial expression and relaxed movements.

[0752] As described above, the system that combines the emotion engine suggests songs and instruments that match the user's emotions, and adjusts the performance method and dance, providing an even more enjoyable musical experience.

[0753] The process flow of each embodiment will be described below.

[0754] Step 1: The server recognizes the user's emotion

[0755] The server utilizes an emotion engine to recognize the user's emotions.

[0756] The emotion engine recognizes the emotions (joy, excitement, relaxation, etc.) felt by the user while playing a piece of music and transmits them to the server.

[0757] Step 2: The server suggests music and / or instruments according to the emotion.

[0758] The server dynamically updates a list of at least one of music and instruments that match the user's emotion based on the received emotion information.

[0759] For example, if the user is excited, the server may suggest music with a faster tempo or rhythmic instruments such as drums that will increase excitement.

[0760] Step 3: The device displays a list of music and / or instruments corresponding to the emotion.

[0761] The terminal displays a list of at least one of music and instruments corresponding to the emotion received from the server.

[0762] The user can select at least one of a piece of music and an instrument on the terminal.

[0763] Step 4: User selects a song and plays it

[0764] When the user selects a piece of music, the terminal adjusts at least one of the playing method and the difficulty level to suit the piece of music.

[0765] For example, if the user is relaxed, the terminal will simplify the playing method and suggest playing with a relaxed atmosphere.

[0766] Step 5: The user selects and plays an instrument

[0767] When the user selects an instrument, the terminal adjusts the instrument's playing style and tone to match the user's emotions.

[0768] For example, if the user is excited, the terminal will make the playing style more rhythmic and suggest energetic tones.

[0769] Step 6: User dances to the music

[0770] When a user dances to music, the terminal changes the facial expression and movements of the dancing character according to the user's emotions.

[0771] For example, if the user is feeling happy, the character will have a bright expression and active movements.

[0772] As described above, in a system that combines an emotion engine, the server recognizes the user's emotions, and the device suggests music and instruments according to the emotions, and adjusts the playing method and dance. This allows the user to enjoy a music experience that is more tailored to their emotions.

[0773] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0774] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0775] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0776] Example 1

[0777] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0778] Conventional music and instrument-related systems make it difficult for users to freely search for and try out musical pieces and instruments, and lack interactive functions to stimulate purchase motivation. In addition, it is difficult to recommend appropriate musical pieces and instruments based on the user's interests and preferences, and there is a need for a means to provide a personalized experience to each user. Another problem is that the process from intuitively expressing the intention to purchase an instrument or piece that has been tried out to actually purchasing it is complicated.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0780] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a music database and a musical instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for providing the user with a means for trying out a virtual instrument and interacting with a character dancing to music; an output means for displaying a trial play screen corresponding to the instrument selected by the user and, when the user plays the instrument on the trial play screen, outputting sound in accordance with the performance; and a display means for displaying a dance animation of a character corresponding to the song in accordance with the user's performance. This allows the user to easily search for songs and instruments and smoothly progress through the process to purchase while enjoying trial play and dance animation.

[0781] "User information" refers to personal information of individuals who use the system, including their preferences, interests, age, gender, etc.

[0782] A "music database" is a database that stores and manages information about music, such as song titles, artists, genres, and release dates.

[0783] A "database relating to musical instruments" is a database that stores and manages information relating to musical instruments, such as the type, characteristics, manufacturer, and playing method of the instrument.

[0784] A "list" is a table containing items of music and instruments selected based on user information.

[0785] The "retrieval means" is the part of the system that searches the database for appropriate information based on user input information and generates the list.

[0786] The "selection means" is an interface that displays the acquired list to the user and enables the user to select a song or instrument.

[0787] The "providing means" is a part of the system that provides functions for the user to try out virtual instruments and interact with characters.

[0788] The "test play screen" is a screen that visually displays a virtual musical instrument and allows the user to interactively operate it.

[0789] The "output means" is a part of the system that, when a user plays an instrument on the test playing screen, generates and outputs a sound corresponding to the performance.

[0790] The "display means" is a part of the system that displays a dance animation of a character that corresponds to the music played by the user.

[0791] "Purchase Options" is the part of the system that presents users with the options to select and purchase their favorite songs and instruments.

[0792] The system of the present invention allows users to interactively search for information about music and musical instruments, and to try out and purchase instruments. Specifically, the system works with a database of music and musical instruments based on user information, and generates and displays a list of songs and musical instruments. It also provides a means for users to try out virtual musical instruments and enjoy dance animations that accompany the music, while purchasing musical instruments and songs that they like.

[0793] The server provides a login screen to receive user information and displays an interface for entering email address and password. The hardware used is a general high-performance server, such as an "EC2 instance" from "Amazon Web Services (AWS)". "MySQL" is used for database management.

[0794] After logging in, the user moves to the music search screen. The device displays a search bar, and when the user enters a song name, artist name, or genre to search, the information is sent to the server. The server queries a music database based on the received search keywords and retrieves the results. The server sends the search results in JSON format to the device, which displays the results to the user in an easy-to-read format.

[0795] For example, when a user searches for "Shape of You," the server retrieves the corresponding song information from the database and displays it on the terminal, along with the instruments the user can select.

[0796] When the user selects an instrument, the device displays a trial screen. The software used is Unity or Unreal Engine, which draws the interface of the virtual instrument. When the user operates the virtual instrument on the screen, the device receives this and generates and outputs the corresponding sound.

[0797] During the trial play process, a character dancing to the music is displayed. The server sends the character data to the device, and the device uses Unity or Unreal Engine to draw the animation. While the user is trying out the music, the character continues to dance to the music.

[0798] If a user finds a song or instrument that they like, they can select the purchase option. The terminal displays a purchase button and sends the purchase information based on the user's selection to the server. The server processes the payment through an e-commerce platform such as Shopify or WooCommerce, and uses a payment gateway such as Stripe.

[0799] Specific examples of prompt sentences are as follows:

[0800] "The system allows users to search for information in a music database and an instrument database, try out virtual instruments, enjoy characters dancing to music, and easily purchase their favorite instruments and songs. Please explain this procedure and the associated process flow in detail."

[0801] In this way, a system can be realized that allows users to easily search for, try out, enjoy, and ultimately purchase music and instruments.

[0802] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0803] Step 1: User authentication

[0804] Input: The user enters their email address and password on the login screen.

[0805] How it works: When the user clicks the login button, the device sends this information to the server.

[0806] Data processing: The server receives the user's email address and password and checks them against the authentication information in its database.

[0807] Output: If authentication is successful, the server grants access to the user and displays the home screen.

[0808] Step 2: Search for songs

[0809] Input: A user types a song name, artist name, or genre into the search bar and clicks the Search button.

[0810] Operation: The device sends search keywords to the server.

[0811] Data processing: The server runs a search query against a music database to retrieve song information based on the specified keywords.

[0812] Output: The server sends the search results in JSON format to the terminal, and the terminal displays the retrieved song list on the screen.

[0813] Step 3: Select your instrument

[0814] Input: The user selects the instrument they want to try out from the music search results.

[0815] Operation: The device sends the instrument selection information to the server and loads the trial screen.

[0816] Data processing: The server obtains information about the selected instrument and sends it to the terminal.

[0817] Output: The device displays the virtual instrument in a preview screen.

[0818] Step 4: Trying out the instrument

[0819] Input: The user manipulates a virtual instrument on the preview screen (e.g., pressing a piano key or plucking a guitar string).

[0820] How it works: The device processes user input in real time and generates and outputs corresponding sounds.

[0821] Data processing: Generates audio data based on user operations and plays sound.

[0822] Output: The device will output the generated sound and keep displaying it on the preview screen.

[0823] Step 5: Displaying the dancing character

[0824] Input: The user plays a song.

[0825] Operation: The server sends data of the dance character that corresponds to the song to the terminal.

[0826] Data processing: The device receives the animation data of the dancing character and displays it in time with the music.

[0827] Output: The device displays the character's dance animation on the preview screen.

[0828] Step 6: Checkout

[0829] Input: A user clicks the purchase button on the preview screen to purchase an instrument or piece of music that they like.

[0830] Operation: The device sends the purchase information to the server and displays the payment screen.

[0831] Data processing: The server processes the purchase information and processes the payment through a payment gateway.

[0832] Output: Once the payment is complete, the server sends a notification to the user that the purchase is complete and displays a purchase confirmation screen.

[0833] (Application example 1)

[0834] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0835] Modern consumers tend to obtain a lot of information through the Internet and make purchasing decisions, but there are limited ways to check the actual feel of musical instruments and music online. In addition, when providing interactive experiences of musical instruments and music, there is a challenge in how to recommend appropriate music and instruments that match the tastes and preferences of the consumer. Furthermore, a system is required that allows consumers to directly purchase musical instruments and music that they like through these experiences.

[0836] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0837] In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and musical instruments corresponding to the user information from at least one of a music information database and a musical instrument information database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and musical instruments, a provision means for providing a means for the user to try out virtual musical instruments and interact with virtual characters dancing to music, and a purchase means for purchasing at least one of the musical instruments and songs that the user likes. This allows the user to have an interactive experience of musical instruments and songs online, and further allows the user to purchase products that they like on the spot.

[0838] "User information" refers to personal information such as the user's preferences, hobbies, age, and gender.

[0839] The "music information database" is a database that stores and manages information about songs, such as song titles, artist names, and genres.

[0840] The "musical instrument information database" is a database that stores and manages information about musical instruments, such as the type, characteristics, and playing methods of the musical instruments.

[0841] The "acquisition means" is a function for acquiring relevant information from the music information database or the musical instrument information database based on user information.

[0842] The "selection means" is a function that displays a list of acquired songs and instruments to the user, allowing the user to select from the list.

[0843] The "means for providing" is a function that allows the user to virtually try out an instrument selected by the user, or to have a virtual character dance to the selected music piece.

[0844] "Purchase method" is a function that allows users to directly purchase their favorite instruments and music online.

[0845] The "trial playing screen" is an interactive screen for playing a virtual musical instrument.

[0846] The "virtual characters" are virtual characters that dance to the rhythm of the music.

[0847] "Purchase Options" provide users with the choice to purchase their favorite instruments and songs.

[0848] As an embodiment of the present invention, we provide a "virtual musical instrument store app" that allows users to interactively experience musical instruments and songs and even make purchases. The following is a detailed description of the main components of this app and their functions.

[0849] System Configuration

[0850] Hardware and Software

[0851] Backend: Django framework (Python) is used. The server is built on this framework to communicate with the database and process API requests.

[0852] Frontend: Using React (JavaScript), we will build the user interface and provide a screen for users to search, select, and try out instruments and songs.

[0853] Database: Uses MySQL. Manages the music information database and instrument information database, and provides data according to user requests.

[0854] Dancing character animation: Using Unity, we generate animations of virtual characters dancing to music.

[0855] API integration: Uses the Spotify API (music information) and a dedicated instrument API. Acquires music information and instrument information in real time and provides it to users.

[0856] Process Overview

[0857] Getting information and displaying lists

[0858] The server first receives information entered by the user (e.g. favorite genre, artist name, type of instrument, etc.). Based on this information, it retrieves the corresponding list from the music information database and instrument information database. The retrieved list is displayed on a front-end screen using React, and is designed to allow the user to make an intuitive selection.

[0859] Virtual Trial and Dancing Characters

[0860] When a user selects a specific instrument or song from the list, that information is sent to the server. The server generates a virtual test screen based on the selected instrument, allowing the user to simulate and play the instrument. In addition, a virtual character generated using Unity dances to the selected song, allowing the user to enjoy a highly entertaining experience.

[0861] Providing purchasing options

[0862] Users are provided with the option to purchase their favorite instruments and songs on the spot. The purchase process is carried out securely through an API provided by the server. This allows users to complete the purchase process with just one click.

[0863] Examples

[0864] User Interaction

[0865] 1. A user types "guitar" into the app's search bar.

[0866] 2. The server queries the instrument information database and displays a list of matching guitars on a screen using React.

[0867] 3. The user selects a particular guitar and begins playing it virtually on the test play screen.

[0868] 4. At the same time, the virtual character will start dancing to the selected song via Unity.

[0869] 5. If the user likes it, they can tap the "Buy" button to purchase the guitar online.

[0870] Example prompts for generative AI models

[0871] "I would like to think of an application that uses a musical instrument information database that includes the types, characteristics, and playing methods of musical instruments, and a music information database that includes song titles, artists, genres, etc. In particular, please give a concrete example of a virtual music store app that integrates an interactive UI that allows users to virtually try out musical instruments on their smartphones, implements dancing characters that dance along with the music, and provides purchasing options."

[0872] This allows users to have an interactive experience with musical instruments and music online, and purchase the products they like on the spot.

[0873] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0874] Step 1:

[0875] The server receives a request from a user. The input includes information about the user's preferences and interests (e.g., favorite genres, artist names, types of instruments, etc.). Based on this information, the server retrieves a list of songs and instruments corresponding to the user from a music information database and an instrument information database. The retrieved data is output as a music list and an instrument list.

[0876] Step 2:

[0877] On the device (such as a smartphone or tablet), the acquired song and instrument lists are displayed in a user interface using React. The user browses the list and selects the song or instrument they like. This selection information is sent as input to the next processing step. The displayed list is interactive, allowing the user to make selections easily.

[0878] Step 3:

[0879] Information on the song and instrument selected by the user is sent to the server. The server receives the information and generates a virtual trial screen for the selected instrument. In addition, it uses Unity to generate a dance animation of a virtual character that matches the selected song. The generated trial screen and animation of the dancing character are output.

[0880] Step 4:

[0881] The terminal displays the generated virtual trial screen and the virtual character's dance animation. The user can interactively play the virtual instrument on the trial screen, and at the same time enjoy watching the virtual character dance to the selected music. The terminal outputs sound in real time according to the user's input (musical instrument playing motion).

[0882] Step 5:

[0883] After the user has tried out the instruments and danced to them, a purchase option is displayed on the device to allow the user to immediately purchase the instrument or song they like. When the user taps the "Purchase" button, the device sends a purchase request to the server. The server receives this request and completes the purchase procedure for the selected instrument or song. When the purchase procedure is complete, a confirmation message is displayed on the device.

[0884] Example 2

[0885] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0886] Conventional music and instrument recommendation systems do not provide appropriate recommendations based on the user's individual preferences and interests, and lack a smooth operation that allows users to immediately purchase music after enjoying playing or listening to it. Therefore, in order to improve the user experience, a new system is needed that seamlessly links music and instrument recommendations based on the user's preferences, virtual play-by-play functionality, and purchase options.

[0887] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments, a playback means for providing an interface for the user to try out a virtual instrument and playing sounds accompanying the trial of the instrument, a display means for displaying an animated character that moves in accordance with the song selected by the user, and a presentation means for presenting purchase options to the user after the user has enjoyed the trial play or animation display. This enables recommendations of songs and instruments based on the individual preferences and interests of the user, a virtual trial experience, and a smooth purchase process.

[0888] A "user" is an individual who uses the system.

[0889] "Information" is data about the user, including favorite music, favorite instruments, interests, age, gender, and so on.

[0890] A "music database" is a database in which information about songs is stored.

[0891] The "instrument database" is a database in which information about musical instruments is stored.

[0892] The "acquisition means" is a means for receiving user information and acquiring a list of songs and instruments from the song database and the instrument database based on the information.

[0893] The "selection means" is a means for displaying the obtained list and allowing the user to select at least one of a piece of music and an instrument.

[0894] A "virtual instrument" is an instrument that does not have a physical existence, but can be played virtually on an interface.

[0895] An "interface" is a means of exchanging information between a user and a system.

[0896] The "playback means" refers to a means for playing back sounds associated with the test playing of a virtual musical instrument.

[0897] An "animated character" is a virtual character that moves in sync with the music.

[0898] "Display means" refers to a means for displaying an animated character on a screen.

[0899] The "presentation means" is a means for presenting a purchase option to a user after the user has enjoyed the content.

[0900] A "list" is a list of songs and instruments corresponding to user information.

[0901] A "purchase option" is a choice that allows a user to purchase at least one of a musical instrument and a piece of music that the user likes.

[0902] In the embodiment of the present invention, the main entities are a user, a terminal, and a server. The program of this system provides recommendations of songs and instruments, trial experiences, and purchase options based on the user's information. The process flow of this system and a specific embodiment are shown below.

[0903] Entering user information

[0904] When a user starts the system, the terminal first displays a screen for inputting user information. The user enters information such as age, gender, favorite musical instrument and music into the form. The terminal receives the input information and sends it to the server.

[0905] Data processing on the server

[0906] The server analyzes the user information received from the device and accesses a music database and an instrument database (e.g., AWS's RDS or Google Cloud's Firestore). It then searches for the appropriate data based on the user's preferences and generates a list of songs and instruments. The generated list is returned to the device in JSON format.

[0907] Data display and user selection on the terminal

[0908] The device parses the JSON data received from the server and displays a list of songs and instruments to the user (e.g., displaying the list using HTML, CSS, and JavaScript). The user can select the song or instrument they like from the displayed list. For example, "Drum Set A" and "Drum Song B" are displayed.

[0909] Trial Play Function

[0910] Once the user selects an instrument, the device switches to a test-play screen, which displays a virtual instrument (e.g., a touch-activated drum set). When the user plays the instrument on the screen, real sounds are played using the Web Audio API. Each touch input corresponds to a specific sound, providing an interactive test-play experience.

[0911] Music playback and dance display function

[0912] When a user selects a song, the device streams the selected song (e.g., HLS protocol) and displays a screen in which an animated character dances along with the song (e.g., animation control using HTML5 and JavaScript). The character's movements are synchronized with the rhythm of the song.

[0913] Present purchasing options

[0914] After the user has finished trying out the instrument or listening to the music, the device presents them with a purchase option. If the user selects the option in a pop-up, they will be redirected to the purchase page of the EC platform (e.g., Shopify). This allows the user to smoothly purchase the instrument or music.

[0915] As a concrete example, if the user is a 5-year-old boy whose favorite instrument is the drum, the server receives the user information and searches for songs and instruments related to drums from the song database and instrument database. The terminal displays the list, and when the user selects the drum, a trial screen is displayed. When the user plays, the sound is played back, and when a song is selected, a dancing character is displayed. In addition, a purchase option is presented, and the user can purchase an item.

[0916] Here is an example of a prompt to be input to the generative AI model:

[0917] 1. "My 5-year-old boy's favorite instrument is the drums. Can you give me some information about songs and instruments related to drums?"

[0918] 2. "I want an interactive screen to try out drums. Can you tell me how to set that up?"

[0919] 3. "How can I play a song that involves drums and have a character dance along to it?"

[0920] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0921] Step 1:

[0922] The user enters information such as age, gender, favorite musical instrument and music into the input form displayed on the terminal screen. This input form is created using HTML, CSS, and JavaScript. The information entered by the user is temporarily stored on the terminal.

[0923] Input: Basic information entered by the user (age, gender, favorite instruments and songs)

[0924] Output: Get user information

[0925] Specific behavior:

[0926] The user enters information using a tablet or smartphone and taps the "Submit" button.

[0927] Step 2:

[0928] The device sends the information collected from the user to the server using the HTTPS protocol, and the data is encrypted for security reasons.

[0929] Input: User information (age, gender, favorite instruments and songs)

[0930] Output: User information sent to the server

[0931] Specific behavior:

[0932] The device monitors when the "Send" button is tapped and sends the data to the server as a POST request.

[0933] Step 3:

[0934] The server analyzes the user information received from the terminal. Based on the analysis result, it executes queries to the music database and the instrument database. This database access is performed using SQL queries, etc.

[0935] Input: User information submitted

[0936] Output: List of music and instrument data

[0937] Specific behavior:

[0938] The server executes a query and extracts, for example, for a user whose "favorite instrument = drums" data on songs and instruments related to drums.

[0939] Step 4:

[0940] The server returns the list of songs and instruments in JSON format to the terminal. This response is quick, providing data to the user without making them wait.

[0941] Input: List of music and instrument data

[0942] Output: Sending data to the terminal

[0943] Specific behavior:

[0944] The server generates JSON format data and sends it to the terminal as an HTTP response. For example, data is sent in the format {"music": ["drum song A", "drum song B"], "instrument": ["drum set A", "drum set B"]}.

[0945] Step 5:

[0946] The device parses the received JSON data and displays a list of songs and instruments to the user. To do this, it parses the received data with JavaScript and dynamically generates a screen using HTML and CSS.

[0947] Input: JSON data received from the server

[0948] Output: A list of songs and instruments displayed to the user

[0949] Specific behavior:

[0950] The device uses DOM manipulation on the browser to display a list of songs and instruments in a table format or button format, which the user can scroll through to select.

[0951] Step 6:

[0952] The user selects an instrument or a song from a list by tapping the touch screen.

[0953] Input: User selection (e.g. drums)

[0954] Output: ID of selected instrument or piece

[0955] Specific behavior:

[0956] A touch event is fired and visual feedback is provided, such as changing the background color of the selected item.

[0957] Step 7:

[0958] The device displays a trial screen based on the instrument selected by the user. A virtual version of the selected instrument is displayed on the trial screen, and the user can play it by touch operation.

[0959] Input: User's selection information (instrument ID)

[0960] Output: Display of the trial screen

[0961] Specific behavior:

[0962] The demo screen displays a virtual drum set, and sounds are played using the Web Audio API in response to touch controls.

[0963] Step 8:

[0964] When a user tries out a virtual instrument, the device outputs sounds in sync with the performance, such as specific drum sounds that correspond to the touched areas, using the Web Audio API.

[0965] Input: User touch actions

[0966] Output: Playback of the corresponding instrument sound

[0967] Specific behavior:

[0968] When the user taps on different parts of the drum set, sounds such as snare and hi-hat are produced.

[0969] Step 9:

[0970] When a user selects a song from the list, the device plays the song and displays a dancing animated character on the screen. The song data is played in streaming format, and the animation is controlled by JavaScript.

[0971] Input: ID of selected song

[0972] Output: Playing music and displaying animations

[0973] Specific behavior:

[0974] As the music plays, the characters dance to the rhythm.

[0975] Step 10:

[0976] After enjoying the trial and music playback, the device will display a pop-up with purchase options, and if the user wishes to purchase, a link to the purchase page will be generated.

[0977] Input: User action completed

[0978] Output: Purchase option popup

[0979] Specific behavior:

[0980] After the song plays, a pop-up message appears asking, "Do you want to purchase this instrument?", and when the user taps the purchase button, they are redirected to the e-commerce site.

[0981] (Application example 2)

[0982] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0983] Conventional instrument and music selection and purchase systems lacked content that was easy for users to use and enjoy. In addition, because suggestions were not made that reflected the individual preferences and attributes of the user, the user experience was uniform and there was insufficient effort to increase purchasing motivation. Furthermore, virtual trial playing and music experience was limited, making it difficult for users to find products they liked.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0985] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for allowing the user to try out a virtual instrument, displaying a virtual character dancing to music, and presenting an option to purchase the selected instrument or song; a generation means for generating a recommendation list based on the user information using a generative AI model; and a prompt generation means for generating a prompt sentence when inputting user information. This makes it possible to provide suggestions of instruments and songs based on the individual preferences and attributes of the user, thereby improving the quality of the user experience. Furthermore, through the virtual try-out and song experience, the probability that the user will like the product increases, and it is possible to effectively increase the desire to purchase.

[0986] "User Information" is data relating to an individual user, including age, gender, preferred instrument, interests, and other demographic information.

[0987] A "music database" is an information source that stores data on multiple songs, including detailed information about each song.

[0988] An "instrument database" is an information source that stores data on multiple instruments, including detailed information about each instrument.

[0989] The "acquisition means" is a mechanism for acquiring corresponding lists from the music database and the instrument database using user information.

[0990] The "selection means" is a mechanism for displaying the acquired list and allowing the user to select at least one of a piece of music and an instrument.

[0991] A "means for providing" is a mechanism that allows a user to try out virtual instruments, displays a virtual character dancing to music, and presents purchasing options.

[0992] "Generation means" refers to a mechanism for generating a recommendation list based on user information using a generative AI model.

[0993] A "prompt generating means" is a mechanism for generating a prompt sentence to be used when inputting user information.

[0994] A "generative AI model" is a generative model that uses artificial intelligence and makes a variety of suggestions and generates information based on input data.

[0995] A "virtual character" is a digitally generated animated character that is a visual interface through which a user can enjoy a musical experience.

[0996] In order to implement this invention, it is necessary to configure a system using a smartphone, a server, a generative AI model, a database, etc.

[0997] First, the server receives user information (age, gender, preferred instruments, interests, etc.). This information is entered through a smartphone application. At this time, to make it easier for the user to enter the information, a generative AI model is used to generate an appropriate prompt sentence. For example, a prompt may be generated that reads, "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet."

[0998] The server acquires a list of songs and musical instruments corresponding to the user information from at least one of the song database and the musical instrument database by the acquisition means. The acquired list is displayed to the user through the smartphone application, and the user selects a song or musical instrument by the selection means.

[0999] As a means of providing the service, the server provides a function for the user to try out a virtual instrument on the smartphone for the selected instrument or song, and a function for displaying a virtual character dancing to the music. For example, if the user selects a trumpet, a virtual trumpet is displayed on the smartphone screen, and a sound is made when the user taps it. Also, an animation of the character dancing to the selected song is played.

[1000] The generating means causes the server to generate a recommendation list based on the user information using the generative AI model, and the recommendation list is used to suggest musical instruments and songs that match the user's preferences.

[1001] The prompt generating means generates a prompt sentence to support input of user information. For example, if the user is a 5-year-old boy who likes drums, the prompt generating means generates a prompt such as "This 5-year-old boy likes drums. Please generate a list of musical instruments and songs related to drums."

[1002] In addition, by presenting the user with a purchasing option to actually purchase the musical instruments and music they like, the system increases the user's desire to purchase. The smartphone application is designed so that after the user has enjoyed playing and listening to the instruments, they can access the purchase page with one click.

[1003] For example, if an 8-year-old girl likes the trumpet, the application will display items and songs related to the trumpet. The user can tap the trumpet on the smartphone screen to try it out and experience its sound. When a song that accompanies the trumpet is played, the application will show a virtual character dancing happily to the song.

[1004] In this way, the user experience is improved through individual proposals and rich experiences based on user information, resulting in a natural induction into product purchases.

[1005] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1006] Step 1:

[1007] Entering user information

[1008] Users input their individual information (age, gender, favorite instrument, etc.) through a smartphone application. This provides the system with basic data to suggest the most suitable instrument and music piece for the user. The input information is sent to the server. Examples of input data include age: 8 years old, gender: female, favorite instrument: trumpet, etc.

[1009] Step 2:

[1010] Generate prompt statement

[1011] The server uses the generative AI model to generate a prompt sentence based on the user information. The prompt sentence is intended to properly guide the user information and improve the accuracy of the acquired data. It generates a prompt sentence such as "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet." This generated prompt sentence is used to query the database.

[1012] Step 3:

[1013] Obtaining information

[1014] The server queries the music database and instrument database based on the generated prompt text to obtain a list of music and instruments corresponding to the user's information. For example, a list of instruments and music related to "trumpet" is obtained. Through this process, a list suited to the user is generated on the server, and the result is sent to the terminal.

[1015] Step 4:

[1016] Viewing and selecting from the list

[1017] The terminal displays the list of songs and instruments received from the server. The user selects the song or instrument he is interested in from the list. For example, the user selects trumpet and "Trumpet Song A". This selection information is sent back to the server.

[1018] Step 5:

[1019] Try out virtual instruments and display virtual characters

[1020] The server sends the trial data for the selected instrument to the device, and the device displays the virtual instrument trial screen. When the user taps the trumpet on the screen, the sound of an actual trumpet is played. Also, an animation of a virtual character dancing to the selected music piece is displayed. For example, tapping the key of a trumpet produces a "C" sound.

[1021] Step 6:

[1022] Present purchasing options

[1023] After the user has finished playing or listening to the instrument, the server sends the purchase options for the selected instrument or piece to the terminal, and the terminal displays a link that allows the user to purchase the instrument or piece they like with one click. For example, the options "Purchase Trumpet" or "Purchase Piece A" may be presented.

[1024] In this way, a series of processes are carried out, from inputting user information, generating prompts using the generative AI model, retrieving information from the database, playing and displaying the products, and presenting purchasing options. This system allows users to easily select and purchase products while having fun.

[1025] [Fourth embodiment]

[1026] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1027] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1029] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1030] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1031] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1033] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1034] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1035] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1037] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1038] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1039] An embodiment of the present invention is embodied as a system including an acquisition unit, a selection unit, and a provision unit. This system enables exploration of a music park and learning of music and musical instruments. The acquisition unit receives information about a user and acquires a list of at least one of music and musical instruments corresponding to the user's information from at least one of a music database and a musical instrument database. For example, the list is created by a creation unit that creates a list based on at least one of the user's preferences, interests, age, and gender. The selection unit displays the acquired list and allows the user to select at least one of a music and musical instrument. The provision unit provides a means for the user to try out virtual musical instruments and interact with characters dancing to music. Specifically, the provision unit displays a trial screen corresponding to the musical instrument selected by the user, and includes an output unit that outputs sound in accordance with the performance when the user plays the musical instrument on the trial screen, and a display control unit that controls to display a character dancing to the musical instrument selected by the user. The system may also include a presentation unit that presents a purchase option for purchasing at least one of a musical instrument and a musical instrument that the user likes. Note that each functional unit of the above system is provided in a server or a terminal.

[1040] Here, the "music database" refers to a database that stores and manages information about music (such as song titles, artists, and genres). The "instrument database" refers to a database that stores and manages information about musical instruments (such as the type of instrument, its features, and how to play it). The "trial play screen" refers to a screen for playing virtual musical instruments on the terminal, providing an interactive environment for users to play musical instruments. The "dancing character" refers to a virtual character that dances to the rhythm of the music, allowing users to enjoy playing along with the music. The "purchase options" refer to options and methods for users to actually purchase their favorite musical instruments and songs, allowing them to obtain the actual instruments and songs.

[1041] According to the above system, it is possible to provide a virtual music park where children can have fun while learning songs and musical instruments. Specifically, the server receives user information and creates a list of appropriate songs and musical instruments from a song database and a musical instrument database, thereby providing options that match children's preferences and interests. In addition, the terminal displays the list of songs and musical instruments received from the server, and the user can select a learning content that matches his or her interests by making a selection. In addition, a learning environment that is close to actual performance and experience can be provided by allowing the user to try out virtual musical instruments and interact with characters dancing to music. In addition, by providing the user with the option to actually purchase musical instruments and songs that they like, it is possible to promote actual musical instrument performance and music appreciation as an extension of learning. In this way, it is possible to provide a music learning environment where children can have fun while learning songs and musical instruments that match their preferences and interests.

[1042] The functions of each element, the server, the terminal, and the user, will be described below.

[1043] 1. Server:

[1044] User information is received, and a list of at least one of songs and instruments corresponding to the user information is created from at least one of a song database and an instrument database.

[1045] An algorithm and / or database search technique is implemented to provide a list of songs and / or instruments based on the user's information.

[1046] 2. Terminal:

[1047] A list of at least one of songs and instruments received from the server is displayed, and a user interface is provided for the user to select from.

[1048] A test play screen that matches the instrument selected by the user is displayed, and a function is implemented that outputs and plays a sound in accordance with the instrument when the user plays it.

[1049] Implement a function to display a character dancing to a song selected by the user.

[1050] 3. User:

[1051] Provide the server with information about yourself (favorite music, favorite instruments, interests, age, gender, etc.).

[1052] At least one of a piece of music and an instrument is selected on the terminal, and the user tries playing or dancing to the piece of music and an instrument.

[1053] A purchase option is used to actually purchase at least one of the musical instrument and the song that the user likes.

[1054] <Example>

[1055] For example, if the user is a 5-year-old boy whose favorite instrument is the drums, the server receives the user's information and creates at least one of a song and an instrument related to drums from at least one of a song database and an instrument database. The terminal displays the list received from the server, and when the user selects the drums, a trial screen is displayed. When the user plays the drums, the terminal plays a sound in accordance with the play. Also, when the user selects a song, the terminal displays a character that dances in accordance with the song. After the user has enjoyed trying out the instruments and songs, the user is presented with a purchase option to actually purchase the instrument or song they like, and can actually purchase it.

[1056] The process flow of each embodiment will be described below.

[1057] Step 1: The server gets the user's information.

[1058] The server receives information provided by the user (favorite music, favorite instrument, interests, age, gender, etc.).

[1059] Step 2: The server creates a list of songs and instruments.

[1060] The server creates a list of at least one of songs and instruments suitable for the user from at least one of a song database and an instrument database based on the received user information. Step 3: The terminal displays the list of at least one of songs and instruments.

[1061] The terminal displays the list of at least one of the songs and the instruments received from the server.

[1062] The user can select at least one of a piece of music and an instrument on the terminal.

[1063] Step 4: The user tries out an instrument and dances.

[1064] When the user selects an instrument, the terminal displays a trial screen that matches the selected instrument.

[1065] When the user plays an instrument, the device plays sounds in sync.

[1066] Step 5: The user selects a song and dances.

[1067] When the user selects a song, the device displays a character dancing to the song.

[1068] Step 6: The user purchases an instrument and / or a song.

[1069] If a user finds an instrument or song that interests them, the device will display a purchase option.

[1070] When a user purchases an instrument or piece of music, the instrument or piece of music is actually delivered to them.

[1071] In this way, the server and the device work together to allow users to search for music and instruments, try them out, dance to them, and even purchase the instruments and music they like.

[1072] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1073] In this embodiment, the system may include an emotion engine that recognizes the user's emotion, and a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument according to the user's emotion. The system may also include an implementation unit that implements an algorithm or a data analysis method for recognizing the user's emotion using the emotion engine, and a provision unit that dynamically updates the list according to the user's emotion to provide a learning experience suitable for the user. The system may also include an adjustment unit that adjusts at least one of the playing method and difficulty of the piece of music and the instrument according to the user's emotion, and a change unit that changes at least one of the facial expression and the movement of the dancing character according to the user's emotion. Each functional unit of the above system is provided in a server or a terminal.

[1074] The functions of each element, the server, the terminal, and the user, will be described below.

[1075] 1. Server:

[1076] Implement algorithms and / or data analysis techniques to incorporate the emotion engine.

[1077] To recognize a user's emotion and to suggest at least one of a piece of music and an instrument according to the emotion.

[1078] To dynamically update a list of at least one of music pieces and instruments according to a user's emotions.

[1079] 2. Terminal:

[1080] To adjust at least one of a musical piece and a playing method of an instrument and a difficulty level according to a user's emotion.

[1081] At least one of the facial expression and the movement of the dancing character is changed according to the emotion of the user.

[1082] 3. User:

[1083] At least one of a piece of music and an instrument is selected based on the emotion recognized by the emotion engine.

[1084] Enjoy music, musical instruments, and dancing that match your emotions.

[1085] <Example>

[1086] For example, when a user is using a virtual music park app, the server utilizes an emotion engine to recognize the user's emotions. The emotion engine recognizes the emotions (e.g., joy, excitement, relaxation) that the user feels when playing a piece of music.

[1087] The server dynamically updates the list of music and instruments that match the user's emotions based on the recognized emotion information. For example, if the user is excited, the server will suggest music with a fast tempo and rhythmic instruments such as drums that will increase the excitement.

[1088] The terminal displays a list of songs and instruments corresponding to the emotion received from the server and allows the user to select one. When the user selects a song, the terminal adjusts the playing method and difficulty to match the song selected by the user. Also, when the user selects an instrument, the terminal adjusts the playing method and tone of the instrument to match the emotion of the user.

[1089] Furthermore, when a user dances to music, the device changes the facial expression and movements of the dancing character according to the user's emotions. For example, if the user is relaxed, the character will have a calm facial expression and relaxed movements.

[1090] As described above, the system that combines the emotion engine suggests songs and instruments that match the user's emotions, and adjusts the performance method and dance, providing an even more enjoyable musical experience.

[1091] The process flow of each embodiment will be described below.

[1092] Step 1: The server recognizes the user's emotion

[1093] The server utilizes an emotion engine to recognize the user's emotions.

[1094] The emotion engine recognizes the emotions (joy, excitement, relaxation, etc.) felt by the user while playing a piece of music and transmits them to the server.

[1095] Step 2: The server suggests music and / or instruments according to the emotion.

[1096] The server dynamically updates a list of at least one of music and instruments that match the user's emotion based on the received emotion information.

[1097] For example, if the user is excited, the server may suggest music with a faster tempo or rhythmic instruments such as drums that will increase excitement.

[1098] Step 3: The device displays a list of music and / or instruments corresponding to the emotion.

[1099] The terminal displays a list of at least one of music and instruments corresponding to the emotion received from the server.

[1100] The user can select at least one of a piece of music and an instrument on the terminal.

[1101] Step 4: User selects a song and plays it

[1102] When the user selects a piece of music, the terminal adjusts at least one of the playing method and the difficulty level to suit the piece of music.

[1103] For example, when the user is relaxed, the terminal simplifies the playing method and suggests playing with a relaxed atmosphere.

[1104] Step 5: The user selects and plays an instrument

[1105] When the user selects an instrument, the terminal adjusts the instrument's playing style and tone to match the user's emotions.

[1106] For example, if the user is excited, the terminal will make the playing style more rhythmic and suggest energetic tones.

[1107] Step 6: User dances to the music

[1108] When a user dances to music, the terminal changes the facial expression and movements of the dancing character according to the user's emotions.

[1109] For example, if the user is feeling happy, the character will have a bright expression and active movements.

[1110] As described above, in a system that combines an emotion engine, the server recognizes the user's emotions, and the device suggests music and instruments according to the emotions, and adjusts the playing method and dance. This allows the user to enjoy a music experience that is more tailored to their emotions.

[1111] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1112] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1113] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1114] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1115] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1116] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1117] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1118] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1119] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1120] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1121] Example 1

[1122] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1123] Conventional music and instrument-related systems make it difficult for users to freely search for and try out musical pieces and instruments, and lack interactive functions to stimulate purchase motivation. In addition, it is difficult to recommend appropriate musical pieces and instruments based on the user's interests and preferences, and there is a need for a means to provide a personalized experience to each user. Another problem is that the process from intuitively expressing the intention to purchase an instrument or piece that has been tried out to actually purchasing it is complicated.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1125] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a music database and a musical instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for providing the user with a means for trying out a virtual instrument and interacting with a character dancing to music; an output means for displaying a trial play screen corresponding to the instrument selected by the user and, when the user plays the instrument on the trial play screen, outputting sound in accordance with the performance; and a display means for displaying a dance animation of a character corresponding to the song in accordance with the user's performance. This allows the user to easily search for songs and instruments and smoothly progress through the process to purchase while enjoying trial play and dance animation.

[1126] "User information" refers to personal information of individuals who use the system, including their preferences, interests, age, gender, etc.

[1127] A "music database" is a database that stores and manages information about music, such as song titles, artists, genres, and release dates.

[1128] A "database relating to musical instruments" is a database that stores and manages information relating to musical instruments, such as the type, characteristics, manufacturer, and playing method of the instrument.

[1129] A "list" is a table containing items of music and instruments selected based on user information.

[1130] The "retrieval means" is the part of the system that searches the database for appropriate information based on user input information and generates the list.

[1131] The "selection means" is an interface that displays the acquired list to the user and enables the user to select a song or instrument.

[1132] The "providing means" is a part of the system that provides functions for the user to try out virtual instruments and interact with characters.

[1133] The "test play screen" is a screen that visually displays a virtual musical instrument and allows the user to interactively operate it.

[1134] The "output means" is a part of the system that, when a user plays an instrument on the test playing screen, generates and outputs a sound corresponding to the performance.

[1135] The "display means" is a part of the system that displays a dance animation of a character that corresponds to the music played by the user.

[1136] "Purchase Options" is the part of the system that presents users with the options to select and purchase their favorite songs and instruments.

[1137] The system of the present invention allows users to interactively search for information about music and musical instruments, and to try out and purchase instruments. Specifically, the system works with a database of music and musical instruments based on user information, and generates and displays a list of songs and musical instruments. It also provides a means for users to try out virtual musical instruments and enjoy dance animations that accompany the music, while purchasing musical instruments and songs that they like.

[1138] The server provides a login screen to receive user information and displays an interface for entering email address and password. The hardware used is a general high-performance server, such as an "EC2 instance" from "Amazon Web Services (AWS)". "MySQL" is used for database management.

[1139] After logging in, the user moves to the music search screen. The device displays a search bar, and when the user enters a song name, artist name, or genre to search, the information is sent to the server. The server queries a music database based on the received search keywords and retrieves the results. The server sends the search results in JSON format to the device, which displays the results to the user in an easy-to-read format.

[1140] For example, when a user searches for "Shape of You," the server retrieves the corresponding song information from the database and displays it on the terminal, along with the instruments the user can select.

[1141] When the user selects an instrument, the device displays a trial screen. The software used is Unity or Unreal Engine, which draws the interface of the virtual instrument. When the user operates the virtual instrument on the screen, the device receives this and generates and outputs the corresponding sound.

[1142] During the trial play process, a character dancing to the music is displayed. The server sends the character data to the device, and the device uses Unity or Unreal Engine to draw the animation. While the user is trying out the music, the character continues to dance to the music.

[1143] If a user finds a song or instrument that they like, they can select the purchase option. The terminal displays a purchase button and sends the purchase information based on the user's selection to the server. The server processes the payment through an e-commerce platform such as Shopify or WooCommerce, and uses a payment gateway such as Stripe.

[1144] Specific examples of prompt sentences are as follows:

[1145] "The system allows users to search for information in a music database and an instrument database, try out virtual instruments, enjoy characters dancing to music, and easily purchase their favorite instruments and songs. Please explain this procedure and the associated process flow in detail."

[1146] In this way, a system can be realized that allows users to easily search for, try out, enjoy, and ultimately purchase music and instruments.

[1147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1148] Step 1: User authentication

[1149] Input: The user enters their email address and password on the login screen.

[1150] How it works: When the user clicks the login button, the device sends this information to the server.

[1151] Data processing: The server receives the user's email address and password and checks them against the authentication information in its database.

[1152] Output: If authentication is successful, the server grants access to the user and displays the home screen.

[1153] Step 2: Search for songs

[1154] Input: A user types a song name, artist name, or genre into the search bar and clicks the Search button.

[1155] Operation: The device sends search keywords to the server.

[1156] Data processing: The server runs a search query against a music database to retrieve song information based on the specified keywords.

[1157] Output: The server sends the search results in JSON format to the terminal, and the terminal displays the retrieved song list on the screen.

[1158] Step 3: Select your instrument

[1159] Input: The user selects the instrument they want to try out from the music search results.

[1160] Operation: The device sends the instrument selection information to the server and loads the trial screen.

[1161] Data processing: The server obtains information about the selected instrument and sends it to the terminal.

[1162] Output: The device displays the virtual instrument in a preview screen.

[1163] Step 4: Trying out the instrument

[1164] Input: The user manipulates a virtual instrument on the preview screen (e.g., pressing a piano key or plucking a guitar string).

[1165] How it works: The device processes user input in real time and generates and outputs corresponding sounds.

[1166] Data processing: Generates audio data based on user operations and plays sound.

[1167] Output: The device will output the generated sound and keep displaying it on the preview screen.

[1168] Step 5: Displaying the dancing character

[1169] Input: The user plays a song.

[1170] Operation: The server sends data of the dance character that corresponds to the song to the terminal.

[1171] Data processing: The device receives the animation data of the dancing character and displays it in time with the music.

[1172] Output: The device displays the character's dance animation on the preview screen.

[1173] Step 6: Checkout

[1174] Input: A user clicks the purchase button on the preview screen to purchase an instrument or piece of music that they like.

[1175] Operation: The device sends the purchase information to the server and displays the payment screen.

[1176] Data processing: The server processes the purchase information and processes the payment through a payment gateway.

[1177] Output: Once the payment is complete, the server sends a notification to the user that the purchase is complete and displays a purchase confirmation screen.

[1178] (Application example 1)

[1179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1180] Modern consumers tend to obtain a lot of information through the Internet and make purchasing decisions, but there are limited ways to check the actual feel of musical instruments and music online. In addition, when providing interactive experiences of musical instruments and music, there is a challenge in how to recommend appropriate music and instruments that match the tastes and preferences of the consumer. Furthermore, a system is required that allows consumers to directly purchase musical instruments and music that they like through these experiences.

[1181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1182] In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and musical instruments corresponding to the user information from at least one of a music information database and a musical instrument information database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and musical instruments, a provision means for providing a means for the user to try out virtual musical instruments and interact with virtual characters dancing to music, and a purchase means for purchasing at least one of the musical instruments and songs that the user likes. This allows the user to have an interactive experience of musical instruments and songs online, and further allows the user to purchase products that they like on the spot.

[1183] "User information" refers to personal information such as the user's preferences, hobbies, age, and gender.

[1184] The "music information database" is a database that stores and manages information about songs, such as song titles, artist names, and genres.

[1185] The "musical instrument information database" is a database that stores and manages information about musical instruments, such as the type, characteristics, and playing methods of the musical instruments.

[1186] The "acquisition means" is a function for acquiring relevant information from the music information database or the musical instrument information database based on user information.

[1187] The "selection means" is a function that displays a list of acquired songs and instruments to the user, allowing the user to select from the list.

[1188] The "means for providing" is a function that allows the user to virtually try out an instrument selected by the user, or to have a virtual character dance to the selected music piece.

[1189] "Purchase method" is a function that allows users to directly purchase their favorite instruments and music online.

[1190] The "trial playing screen" is an interactive screen for playing a virtual musical instrument.

[1191] The "virtual characters" are virtual characters that dance to the rhythm of the music.

[1192] "Purchase Options" provide users with the choice to purchase their favorite instruments and songs.

[1193] As an embodiment of the present invention, we provide a "virtual musical instrument store app" that allows users to interactively experience musical instruments and songs and even make purchases. The following is a detailed description of the main components of this app and their functions.

[1194] System Configuration

[1195] Hardware and Software

[1196] Backend: Django framework (Python) is used. The server is built on this framework to communicate with the database and process API requests.

[1197] Frontend: Using React (JavaScript), we will build the user interface and provide a screen for users to search, select, and try out instruments and songs.

[1198] Database: Uses MySQL. Manages the music information database and instrument information database, and provides data according to user requests.

[1199] Dancing character animation: Using Unity, we generate animations of virtual characters dancing to music.

[1200] API integration: Uses the Spotify API (music information) and a dedicated instrument API. Acquires music information and instrument information in real time and provides it to users.

[1201] Process Overview

[1202] Getting information and displaying lists

[1203] The server first receives information entered by the user (e.g. favorite genre, artist name, type of instrument, etc.). Based on this information, it retrieves the corresponding list from the music information database and instrument information database. The retrieved list is displayed on a front-end screen using React, and is designed to allow the user to make an intuitive selection.

[1204] Virtual Trial and Dancing Characters

[1205] When a user selects a specific instrument or song from the list, that information is sent to the server. The server generates a virtual test screen based on the selected instrument, allowing the user to simulate and play the instrument. In addition, a virtual character generated using Unity dances to the selected song, allowing the user to enjoy a highly entertaining experience.

[1206] Providing purchasing options

[1207] Users are provided with the option to purchase their favorite instruments and songs on the spot. The purchase process is carried out securely through an API provided by the server. This allows users to complete the purchase process with just one click.

[1208] Examples

[1209] User Interaction

[1210] 1. A user types "guitar" into the app's search bar.

[1211] 2. The server queries the instrument information database and displays a list of matching guitars on a screen using React.

[1212] 3. The user selects a particular guitar and begins playing it virtually on the test play screen.

[1213] 4. At the same time, the virtual character will start dancing to the selected song via Unity.

[1214] 5. If the user likes it, they can tap the "Buy" button to purchase the guitar online.

[1215] Example prompts for generative AI models

[1216] "I would like to think of an application that uses a musical instrument information database that includes the types, characteristics, and playing methods of musical instruments, and a music information database that includes song titles, artists, genres, etc. In particular, please give a concrete example of a virtual music store app that integrates an interactive UI that allows users to virtually try out musical instruments on their smartphones, implements dancing characters that dance along with the music, and provides purchasing options."

[1217] This allows users to have an interactive experience with musical instruments and music online, and purchase the products they like on the spot.

[1218] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] The server receives a request from a user. The input includes information about the user's preferences and interests (e.g., favorite genres, artist names, types of instruments, etc.). Based on this information, the server retrieves a list of songs and instruments corresponding to the user from a music information database and an instrument information database. The retrieved data is output as a music list and an instrument list.

[1221] Step 2:

[1222] On the device (such as a smartphone or tablet), the acquired song and instrument lists are displayed in a user interface using React. The user browses the list and selects the song or instrument they like. This selection information is sent as input to the next processing step. The displayed list is interactive, allowing the user to make selections easily.

[1223] Step 3:

[1224] Information on the song and instrument selected by the user is sent to the server. The server receives the information and generates a virtual trial screen for the selected instrument. In addition, it uses Unity to generate a dance animation of a virtual character that matches the selected song. The generated trial screen and animation of the dancing character are output.

[1225] Step 4:

[1226] The terminal displays the generated virtual trial screen and the virtual character's dance animation. The user can interactively play the virtual instrument on the trial screen, and at the same time enjoy watching the virtual character dance to the selected music. The terminal outputs sound in real time according to the user's input (musical instrument playing motion).

[1227] Step 5:

[1228] After the user has tried out the instruments and danced to them, a purchase option is displayed on the device to allow the user to immediately purchase the instrument or song they like. When the user taps the "Purchase" button, the device sends a purchase request to the server. The server receives this request and completes the purchase procedure for the selected instrument or song. When the purchase procedure is complete, a confirmation message is displayed on the device.

[1229] Example 2

[1230] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1231] Conventional music and instrument recommendation systems do not provide appropriate recommendations based on the user's individual preferences and interests, and lack a smooth operation that allows users to immediately purchase music after enjoying playing or listening to it. Therefore, in order to improve the user experience, a new system is needed that seamlessly links music and instrument recommendations based on the user's preferences, virtual play-by-play functionality, and purchase options.

[1232] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database, a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments, a playback means for providing an interface for the user to try out a virtual instrument and playing sounds accompanying the trial of the instrument, a display means for displaying an animated character that moves in accordance with the song selected by the user, and a presentation means for presenting purchase options to the user after the user has enjoyed the trial play or animation display. This enables recommendations of songs and instruments based on the individual preferences and interests of the user, a virtual trial experience, and a smooth purchase process.

[1233] A "user" is an individual who uses the system.

[1234] "Information" is data about the user, including favorite music, favorite instruments, interests, age, gender, and so on.

[1235] A "music database" is a database in which information about songs is stored.

[1236] The "instrument database" is a database in which information about musical instruments is stored.

[1237] The "acquisition means" is a means for receiving user information and acquiring a list of songs and instruments from the song database and the instrument database based on the information.

[1238] The "selection means" is a means for displaying the obtained list and allowing the user to select at least one of a piece of music and an instrument.

[1239] A "virtual instrument" is an instrument that does not have a physical existence, but can be played virtually on an interface.

[1240] An "interface" is a means of exchanging information between a user and a system.

[1241] The "playback means" refers to a means for playing back sounds associated with the test playing of a virtual musical instrument.

[1242] An "animated character" is a virtual character that moves in sync with the music.

[1243] "Display means" refers to a means for displaying an animated character on a screen.

[1244] The "presentation means" is a means for presenting a purchase option to a user after the user has enjoyed the content.

[1245] A "list" is a list of songs and instruments corresponding to user information.

[1246] A "purchase option" is a choice that allows a user to purchase at least one of a musical instrument and a piece of music that the user likes.

[1247] In the embodiment of the present invention, the main entities are a user, a terminal, and a server. The program of this system provides recommendations of songs and instruments, trial experiences, and purchase options based on the user's information. The process flow of this system and a specific embodiment are shown below.

[1248] Entering user information

[1249] When a user starts the system, the terminal first displays a screen for inputting user information. The user enters information such as age, gender, favorite musical instrument and music into the form. The terminal receives the input information and sends it to the server.

[1250] Data processing on the server

[1251] The server analyzes the user information received from the device and accesses a music database and an instrument database (e.g., AWS's RDS or Google Cloud's Firestore). It then searches for the appropriate data based on the user's preferences and generates a list of songs and instruments. The generated list is returned to the device in JSON format.

[1252] Data display and user selection on the terminal

[1253] The device parses the JSON data received from the server and displays a list of songs and instruments to the user (e.g., displaying the list using HTML, CSS, and JavaScript). The user can select the song or instrument they like from the displayed list. For example, "Drum Set A" and "Drum Song B" are displayed.

[1254] Trial Play Function

[1255] Once the user selects an instrument, the device switches to a test-play screen, which displays a virtual instrument (e.g., a touch-activated drum set). When the user plays the instrument on the screen, real sounds are played using the Web Audio API. Each touch input corresponds to a specific sound, providing an interactive test-play experience.

[1256] Music playback and dance display function

[1257] When a user selects a song, the device streams the selected song (e.g., HLS protocol) and displays a screen in which an animated character dances along with the song (e.g., animation control using HTML5 and JavaScript). The character's movements are synchronized with the rhythm of the song.

[1258] Present purchasing options

[1259] After the user has finished trying out the instrument or listening to the music, the device presents them with a purchase option. If the user selects the option in a pop-up, they will be redirected to the purchase page of the EC platform (e.g., Shopify). This allows the user to smoothly purchase the instrument or music.

[1260] As a concrete example, if the user is a 5-year-old boy whose favorite instrument is the drum, the server receives the user information and searches for songs and instruments related to drums from the song database and instrument database. The terminal displays the list, and when the user selects the drum, a trial screen is displayed. When the user plays, the sound is played back, and when a song is selected, a dancing character is displayed. In addition, a purchase option is presented, and the user can purchase an item.

[1261] Here is an example of a prompt to be input to the generative AI model:

[1262] 1. "My 5-year-old boy's favorite instrument is the drums. Can you give me some information about songs and instruments related to drums?"

[1263] 2. "I want an interactive screen to try out drums. Can you tell me how to set that up?"

[1264] 3. "How can I play a song that involves drums and have a character dance along to it?"

[1265] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1266] Step 1:

[1267] The user enters information such as age, gender, favorite musical instrument and music into the input form displayed on the terminal screen. This input form is created using HTML, CSS, and JavaScript. The information entered by the user is temporarily stored on the terminal.

[1268] Input: Basic information entered by the user (age, gender, favorite instruments and songs)

[1269] Output: Get user information

[1270] Specific behavior:

[1271] The user enters information using a tablet or smartphone and taps the "Submit" button.

[1272] Step 2:

[1273] The device sends the information collected from the user to the server using the HTTPS protocol, and the data is encrypted for security reasons.

[1274] Input: User information (age, gender, favorite instruments and songs)

[1275] Output: User information sent to the server

[1276] Specific behavior:

[1277] The device monitors when the "Send" button is tapped and sends the data to the server as a POST request.

[1278] Step 3:

[1279] The server analyzes the user information received from the terminal. Based on the analysis result, it executes queries to the music database and the instrument database. This database access is performed using SQL queries, etc.

[1280] Input: User information submitted

[1281] Output: List of music and instrument data

[1282] Specific behavior:

[1283] The server executes a query and extracts, for example, for a user whose "favorite instrument = drums" data on songs and instruments related to drums.

[1284] Step 4:

[1285] The server returns the list of songs and instruments in JSON format to the terminal. This response is quick, providing data to the user without making them wait.

[1286] Input: List of music and instrument data

[1287] Output: Sending data to the terminal

[1288] Specific behavior:

[1289] The server generates JSON format data and sends it to the terminal as an HTTP response. For example, data is sent in the format {"music": ["drum song A", "drum song B"], "instrument": ["drum set A", "drum set B"]}.

[1290] Step 5:

[1291] The device parses the received JSON data and displays a list of songs and instruments to the user. To do this, it parses the received data with JavaScript and dynamically generates a screen using HTML and CSS.

[1292] Input: JSON data received from the server

[1293] Output: A list of songs and instruments displayed to the user

[1294] Specific behavior:

[1295] The device uses DOM manipulation on the browser to display a list of songs and instruments in a table format or button format, which the user can scroll through to select.

[1296] Step 6:

[1297] The user selects an instrument or a song from a list by tapping the touch screen.

[1298] Input: User selection (e.g. drums)

[1299] Output: ID of selected instrument or piece

[1300] Specific behavior:

[1301] A touch event is fired and visual feedback is provided, such as changing the background color of the selected item.

[1302] Step 7:

[1303] The device displays a trial screen based on the instrument selected by the user. A virtual version of the selected instrument is displayed on the trial screen, and the user can play it by touch operation.

[1304] Input: User's selection information (instrument ID)

[1305] Output: Display of the trial screen

[1306] Specific behavior:

[1307] The demo screen displays a virtual drum set, and sounds are played using the Web Audio API in response to touch controls.

[1308] Step 8:

[1309] When a user tries out a virtual instrument, the device outputs sounds in sync with the performance, such as specific drum sounds that correspond to the touched areas, using the Web Audio API.

[1310] Input: User touch actions

[1311] Output: Playback of the corresponding instrument sound

[1312] Specific behavior:

[1313] When the user taps on different parts of the drum set, sounds such as snare and hi-hat are produced.

[1314] Step 9:

[1315] When a user selects a song from the list, the device plays the song and displays a dancing animated character on the screen. The song data is played in streaming format, and the animation is controlled by JavaScript.

[1316] Input: ID of selected song

[1317] Output: Playing music and displaying animations

[1318] Specific behavior:

[1319] As the music plays, the characters dance to the rhythm.

[1320] Step 10:

[1321] After enjoying the trial and music playback, the device will display a pop-up with purchase options, and if the user wishes to purchase, a link to the purchase page will be generated.

[1322] Input: User action completed

[1323] Output: Purchase option popup

[1324] Specific behavior:

[1325] After the song plays, a pop-up message appears asking, "Do you want to purchase this instrument?", and when the user taps the purchase button, they are redirected to the e-commerce site.

[1326] (Application example 2)

[1327] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1328] Conventional instrument and music selection and purchase systems lacked content that was easy for users to use and enjoy. In addition, because suggestions were not made that reflected the individual preferences and attributes of the user, the user experience was uniform and there was insufficient effort to increase purchasing motivation. Furthermore, virtual trial playing and music experience was limited, making it difficult for users to find products they liked.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1330] In this invention, the server includes: an acquisition means for receiving user information and acquiring a list of at least one of songs and instruments corresponding to the user information from at least one of a song database and an instrument database; a selection means for displaying the acquired list and allowing the user to select at least one of the songs and instruments; a provision means for allowing the user to try out a virtual instrument, displaying a virtual character dancing to music, and presenting an option to purchase the selected instrument or song; a generation means for generating a recommendation list based on the user information using a generative AI model; and a prompt generation means for generating a prompt sentence when inputting user information. This makes it possible to provide suggestions of instruments and songs based on the individual preferences and attributes of the user, thereby improving the quality of the user experience. Furthermore, through the virtual try-out and song experience, the probability that the user will like the product increases, and it is possible to effectively increase the desire to purchase.

[1331] "User Information" is data relating to an individual user, including age, gender, preferred instrument, interests, and other demographic information.

[1332] A "music database" is an information source that stores data on multiple songs, including detailed information about each song.

[1333] An "instrument database" is an information source that stores data on multiple instruments, including detailed information about each instrument.

[1334] The "acquisition means" is a mechanism for acquiring corresponding lists from the music database and the instrument database using user information.

[1335] The "selection means" is a mechanism for displaying the acquired list and allowing the user to select at least one of a piece of music and an instrument.

[1336] A "means for providing" is a mechanism that allows a user to try out virtual instruments, displays a virtual character dancing to music, and presents purchasing options.

[1337] "Generation means" refers to a mechanism for generating a recommendation list based on user information using a generative AI model.

[1338] A "prompt generating means" is a mechanism for generating a prompt sentence to be used when inputting user information.

[1339] A "generative AI model" is a generative model that uses artificial intelligence and makes a variety of suggestions and generates information based on input data.

[1340] A "virtual character" is a digitally generated animated character that is a visual interface through which a user can enjoy a musical experience.

[1341] In order to implement this invention, it is necessary to configure a system using a smartphone, a server, a generative AI model, a database, etc.

[1342] First, the server receives user information (age, gender, preferred instruments, interests, etc.). This information is entered through a smartphone application. At this time, to make it easier for the user to enter the information, a generative AI model is used to generate an appropriate prompt sentence. For example, a prompt may be generated that reads, "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet."

[1343] The server acquires a list of songs and musical instruments corresponding to the user information from at least one of the song database and the musical instrument database by the acquisition means. The acquired list is displayed to the user through the smartphone application, and the user selects a song or musical instrument by the selection means.

[1344] As a means of providing the service, the server provides a function for the user to try out a virtual instrument on the smartphone for the selected instrument or song, and a function for displaying a virtual character dancing to the music. For example, if the user selects a trumpet, a virtual trumpet is displayed on the smartphone screen, and a sound is made when the user taps it. Also, an animation of the character dancing to the selected song is played.

[1345] The generating means causes the server to generate a recommendation list based on the user information using the generative AI model, and the recommendation list is used to suggest musical instruments and songs that match the user's preferences.

[1346] The prompt generating means generates a prompt sentence to support input of user information. For example, if the user is a 5-year-old boy who likes drums, the prompt generating means generates a prompt such as "This 5-year-old boy likes drums. Please generate a list of musical instruments and songs related to drums."

[1347] In addition, by presenting the user with a purchasing option to actually purchase the musical instruments and music they like, the system increases the user's desire to purchase. The smartphone application is designed so that after the user has enjoyed playing and listening to the instruments, they can access the purchase page with one click.

[1348] For example, if an 8-year-old girl likes the trumpet, the application will display items and songs related to the trumpet. The user can tap the trumpet on the smartphone screen to try it out and experience its sound. When a song that accompanies the trumpet is played, the application will show a virtual character dancing happily to the song.

[1349] In this way, the user experience is improved through individual proposals and rich experiences based on user information, resulting in a natural induction into product purchases.

[1350] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1351] Step 1:

[1352] Entering user information

[1353] Users input their individual information (age, gender, favorite instrument, etc.) through a smartphone application. This provides the system with basic data to suggest the most suitable instrument and music piece for the user. The input information is sent to the server. Examples of input data include age: 8 years old, gender: female, favorite instrument: trumpet, etc.

[1354] Step 2:

[1355] Generate prompt statement

[1356] The server uses the generative AI model to generate a prompt sentence based on the user information. The prompt sentence is intended to properly guide the user information and improve the accuracy of the acquired data. It generates a prompt sentence such as "This 8-year-old girl likes the trumpet. Please generate a list of instruments and songs related to the trumpet." This generated prompt sentence is used to query the database.

[1357] Step 3:

[1358] Obtaining information

[1359] The server queries the music database and instrument database based on the generated prompt text to obtain a list of music and instruments corresponding to the user's information. For example, a list of instruments and music related to "trumpet" is obtained. Through this process, a list suited to the user is generated on the server, and the result is sent to the terminal.

[1360] Step 4:

[1361] Viewing and selecting from the list

[1362] The terminal displays the list of songs and instruments received from the server. The user selects the song or instrument he is interested in from the list. For example, the user selects trumpet and "Trumpet Song A". This selection information is sent back to the server.

[1363] Step 5:

[1364] Try out virtual instruments and display virtual characters

[1365] The server sends the trial data for the selected instrument to the device, and the device displays the virtual instrument trial screen. When the user taps the trumpet on the screen, the sound of an actual trumpet is played. Also, an animation of a virtual character dancing to the selected music piece is displayed. For example, tapping the key of a trumpet produces a "C" sound.

[1366] Step 6:

[1367] Present purchasing options

[1368] After the user has finished playing or listening to the instrument, the server sends the purchase options for the selected instrument or piece to the terminal, and the terminal displays a link that allows the user to purchase the instrument or piece they like with one click. For example, the options "Purchase Trumpet" or "Purchase Piece A" may be presented.

[1369] In this way, a series of processes are carried out, from inputting user information, generating prompts using the generative AI model, retrieving information from the database, playing and displaying the products, and presenting purchasing options. This system allows users to easily select and purchase products while having fun.

[1370] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[1371] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[1372] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1373] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1374] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1375] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1376] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1377] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1378] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1379] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[1380] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[1381] The following is further disclosed regarding the above embodiment.

[1382] (Appendix 1) a prompt generating unit for generating a prompt sentence to be used when inputting user information; a generation unit that generates a list of at least one of music and musical instruments corresponding to the user information from at least one of a music database and a musical instrument database using a generation AI model; a selection unit that displays the generated list and allows the user to select at least one of the music piece and the instrument; a providing unit that provides a means for the user to try out virtual musical instruments and interact with characters that dance to music; A system including: (Appendix 2) The generating unit generates the list based on at least one of the user's favorite music pieces, favorite musical instruments, interests, age, and gender; The providing unit, an output unit that displays a trial playing screen corresponding to an instrument selected by the user, and when the user plays the instrument on the trial playing screen, outputs a sound corresponding to the performance; a display control unit that controls to display a character dancing to the music selected by the user; Including, 2. The system described in appendix 1. (Appendix 3) a presentation unit that presents a purchase option for purchasing at least one of the musical instrument and the piece of music that the user likes, 10. The system of claim 1 or 2. (Appendix 4) an emotion engine that recognizes the emotion of the user; a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument in accordance with the emotion of the user; 4. The system according to claim 1, further comprising: (Appendix 5) an implementation unit that implements an algorithm or data analysis technique for recognizing the emotion of the user using the emotion engine; The providing unit dynamically updates the list in response to the user's emotions to provide a learning experience suited to the user. Attachment 4. The system described in claim 4. (Appendix 6) an adjustment unit that adjusts at least one of the musical piece and the playing method and the difficulty level of the musical piece and the musical instrument in accordance with the user's emotion; A change unit that changes at least one of a facial expression and a movement of the dancing character in response to the emotion of the user; 6. The system of claim 4 or 5, [Explanation of symbols]

[1383] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a prompt generating unit for generating a prompt sentence to be used when inputting user information; A generator that uses a generation AI model to generate a list of at least one of a song and an instrument from at least one of a song database and an instrument database corresponding to the user information; a selection unit that displays the generated list and allows the user to select at least one of the music piece and the instrument; a providing unit that provides a means for the user to try out virtual musical instruments and interact with characters that dance to music; A system including:

2. The generating unit generates the list based on at least one of the user's favorite music pieces, favorite musical instruments, interests, age, and gender; The providing unit, an output unit that displays a trial playing screen corresponding to an instrument selected by the user, and when the user plays the instrument on the trial playing screen, outputs a sound corresponding to the performance; a display control unit that controls to display a character dancing to the music selected by the user; Including, The system of claim 1 .

3. a presentation unit that presents a purchase option for purchasing at least one of the musical instrument and the piece of music that the user likes, The system according to claim 1 or 2.

4. an emotion engine that recognizes the emotion of the user; a suggestion unit that receives information from the emotion engine and suggests at least one of a piece of music and an instrument in accordance with the emotion of the user; The system of claim 1 , comprising:

5. an implementation unit that implements an algorithm or data analysis technique for recognizing the emotion of the user using the emotion engine; The providing unit dynamically updates the list in response to the user's emotions to provide a learning experience suited to the user. The system of claim 4.

6. an adjustment unit that adjusts at least one of the musical piece and the playing method and the difficulty level of the musical piece and the musical instrument in accordance with the user's emotion; A change unit that changes at least one of a facial expression and a movement of the dancing character in response to the emotion of the user; 6. The system according to claim 4 or claim 5, comprising:

Images