system

Abstract

The system according to this embodiment aims to analyze the user's singing in real time and provide individualized instruction. [Solution] The system according to the embodiment comprises a singing input device, an analysis unit, and a teaching unit. The singing input device inputs the user's singing. The analysis unit analyzes the singing input by the singing input device in real time. The teaching unit provides individualized instruction based on the results of the analysis by the analysis unit.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, real-time feedback and individual guidance for karaoke singing are not sufficiently performed, and there is room for improvement.

[0005] The system according to the embodiment aims to analyze the singing of a user in real time and provide individual guidance.

Means for Solving the Problems

[0006] The system according to the embodiment includes a singing input device, an analysis unit, and a guidance unit. The singing input device inputs the singing of a user. The analysis unit analyzes the singing input by the singing input device in real time. The guidance unit provides individual guidance based on the results analyzed by the analysis unit.

Effects of the Invention

[0007] The system according to this embodiment can analyze the user's singing in real time and provide individualized instruction. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F . The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the reception device 38, the output device 40, and the camera 42 are connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The Karaoke AI Maestro System, according to an embodiment of the present invention, is a system that revolutionizes the karaoke experience by utilizing generative AI technology. This system begins with the user inputting their singing into a singing input device within a karaoke establishment. This device is equipped with an analysis unit that comprehensively evaluates voice, facial expressions, and movements. Next, the analysis unit analyzes the singing in real time and provides immediate feedback. This feedback provides detailed evaluations of each element, such as pitch, rhythm, and expressiveness. Furthermore, the instruction unit provides individualized guidance, automatically learning to match the user's goals and formulating an optimal learning plan. For example, it provides specific practice methods to compensate for deficiencies in singing ability and advice to enhance expressiveness. It also adjusts the instruction method according to the user's progress, ensuring continuous learning and adaptation. This system functions as a 24-hour AI vocal coach, overcoming time and financial constraints. The cost-effective AI learning system allows users to receive high-quality instruction at a low cost. Additionally, multiple AIs collaborate to provide comprehensive instruction, supporting the user in achieving their goals. In this way, the Karaoke AI Maestro System revolutionizes the karaoke experience through real-time AI analysis and immediate feedback, personalized instruction by autonomous AI agents, multimodal analysis, and continuous learning and adaptation. This allows the Karaoke AI Maestro System to analyze the user's singing in real time and provide personalized instruction, thereby revolutionizing the karaoke experience.

[0029] The karaoke AI maestro system according to this embodiment comprises a singing input device, an analysis unit, and a teaching unit. The singing input device inputs the user's singing. The singing input device inputs the user's singing as audio data, for example, using a microphone. The singing input device can also input the user's facial expressions and movements as video data using a camera. For example, the singing input device records the user's singing with a high-precision microphone and saves it as audio data. Furthermore, the singing input device can also capture the user's facial expressions with a high-resolution camera and save them as video data. Furthermore, the singing input device can detect the user's movements with a motion sensor and save them as movement data. The analysis unit analyzes the singing input by the singing input device in real time. For example, the analysis unit analyzes the audio data to evaluate pitch and rhythm. Furthermore, the analysis unit can also analyze the video data to evaluate facial expressions and movements. For example, the analysis unit analyzes the audio data to evaluate the accuracy of the pitch. Furthermore, the analysis unit can also evaluate the degree of rhythmic consistency. Furthermore, the analysis unit can also analyze video data to evaluate changes in facial expressions. For example, the analysis unit can analyze audio data to evaluate pitch accuracy. The analysis unit can also evaluate rhythmic accuracy. Furthermore, the analysis unit can analyze video data to evaluate changes in facial expressions. The instruction unit provides individual instruction based on the results analyzed by the analysis unit. For example, the instruction unit can advise on how to correct pitch. Furthermore, the instruction unit can also advise on how to improve rhythm. Furthermore, the instruction unit can also provide advice on how to enhance facial expression. For example, the instruction unit can advise on how to correct pitch. Furthermore, the instruction unit can also advise on how to improve rhythm. Furthermore, the instruction unit can also provide advice on how to enhance facial expression. As a result, the karaoke AI maestro system according to this embodiment can revolutionize the karaoke experience by analyzing the user's singing in real time and providing individual instruction.

[0030] The singing input device inputs the user's singing. For example, it can input the user's singing as audio data using a microphone. It can also input the user's facial expressions and movements as video data using a camera. Specifically, the singing input device uses a high-precision microphone to record the user's singing and save it as audio data. This microphone has a noise-canceling function to reduce ambient noise and obtain clear sound. The camera captures the user's facial expressions and movements in high resolution and saves them as video data. This camera can capture subtle changes in the user's facial expressions using facial recognition technology. Furthermore, it is possible to detect the user's movements using a motion sensor and save them as motion data. This motion sensor detects the user's body movements and gestures with high precision, recording the performance during singing in detail. As a result, the singing input device comprehensively collects audio, video, and motion data, allowing for a multifaceted understanding of the user's singing performance. Furthermore, this data is transmitted in real time to a central database, making it immediately accessible to the analysis and instruction departments. This makes it possible to evaluate the user's singing performance in real time and provide immediate feedback.

[0031] The analysis unit analyzes the singing input from the singing input device in real time. For example, the analysis unit analyzes audio data to evaluate pitch and rhythm. It can also analyze video data to evaluate facial expressions and movements. Specifically, the analysis unit uses advanced AI algorithms to analyze audio data and evaluate the accuracy of pitch and the degree of rhythmic consistency. This AI utilizes speech recognition technology to meticulously analyze the user's singing and detect pitch deviations and rhythmic irregularities. It also uses facial recognition technology and motion analysis technology to analyze video data. This allows for a detailed evaluation of the user's facial expressions and movement patterns. For example, the analysis unit reads emotional changes from the user's facial expressions and evaluates the richness of emotional expression during singing. It can also use motion analysis technology to evaluate how the user's body movements and gestures affect singing. Furthermore, by comparing the user's singing performance with past data and data from other users, the analysis unit can identify the degree of improvement in the user's singing performance and specific areas for improvement. This allows the analysis unit to evaluate the user's singing performance from multiple angles and provide detailed feedback.

[0032] The instruction department provides individualized guidance based on the results analyzed by the analysis department. For example, the instruction department may advise on how to correct pitch. They can also advise on how to improve rhythm. Furthermore, they can provide advice on how to enhance facial expression. Specifically, the instruction department uses AI to analyze the user's singing performance and identify individual areas for improvement. This AI suggests specific practice methods to correct pitch deviations and practice methods using a metronome to improve rhythm. In addition, to enhance facial expression, it advises the user on specific facial expression practice methods and techniques for singing with emotion. For example, the instruction department may suggest repeatedly practicing a specific scale as a method for correcting pitch. They may also advise practicing with a metronome and incorporating body movements in time with the rhythm as a method for improving rhythm. Furthermore, to enhance facial expression, they can provide methods for practicing while checking one's facial expressions in a mirror and video tutorials to learn techniques for singing with emotion. In this way, the instruction department can provide specific advice to comprehensively improve the user's singing performance, enabling the user to effectively improve their singing skills.

[0033] The analysis unit can comprehensively evaluate voice, facial expressions, and movements. For example, the analysis unit can analyze audio data to evaluate pitch and rhythm. It can also analyze video data to evaluate facial expressions and movements. For example, the analysis unit can analyze audio data to evaluate the accuracy of pitch. It can also evaluate the degree of rhythmic consistency. Furthermore, it can analyze video data to evaluate changes in facial expressions. By comprehensively evaluating voice, facial expressions, and movements, a more accurate analysis becomes possible. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input audio data, video data, and movement data into a generative AI, which can then evaluate pitch, rhythm, facial expressions, and movements.

[0034] The instruction unit can automatically learn to match the user's goals and formulate an optimal learning plan. For example, the instruction unit can suggest practice methods to improve the user's singing ability. It can also provide advice to enhance the user's expressiveness. Furthermore, the instruction unit can adjust the learning plan according to the user's progress. This enables effective instruction by formulating a learning plan tailored to the user's goals. Some or all of the above processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's goals and progress data into a generative AI, which can then formulate an optimal learning plan.

[0035] The instruction unit can adjust its teaching methods according to the user's progress. For example, the instruction unit can change practice methods as the user's singing ability improves. It can also change the content of advice as the user's expressiveness improves. Furthermore, the instruction unit can optimize its teaching methods based on the user's progress data. For example, the instruction unit can change practice methods as the user's singing ability improves. It can also change the content of advice as the user's expressiveness improves. Furthermore, the instruction unit can optimize its teaching methods based on the user's progress data. This enables continuous learning and adaptation by providing teaching methods tailored to the user's progress. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's progress data into a generative AI, which can then propose the optimal teaching method.

[0036] The instruction unit can function as a continuously operating AI vocal coach. For example, the instruction unit can operate 24 hours a day, 365 days a year, allowing users to receive instruction at any time. The instruction unit can also adjust the timing of instruction to match the user's schedule. Furthermore, the instruction unit can optimize the content of instruction based on the user's usage. This 24 / 7 operation allows users to receive instruction without time constraints. Some or all of the above-described processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input user usage data into a generative AI, which can then suggest optimal instruction content.

[0037] The instruction unit can achieve comprehensive instruction through the collaboration of multiple AIs. For example, the instruction unit can have a voice instruction AI and a facial expression instruction AI working together to provide instruction. Furthermore, the instruction unit can have a movement instruction AI and a voice instruction AI working together to provide instruction. In addition, the instruction unit can have multiple AIs work together to provide comprehensive instruction to the user. For example, the instruction unit can have a voice instruction AI and a facial expression instruction AI working together to provide instruction. Furthermore, the instruction unit can have a movement instruction AI and a voice instruction AI working together to provide instruction. Furthermore, the instruction unit can have multiple AIs work together to provide comprehensive instruction to the user. This allows for more comprehensive instruction through the collaboration of multiple AIs. Some or all of the above-described processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input data from the voice instruction AI, facial expression instruction AI, and movement instruction AI into a generative AI, which can then perform comprehensive instruction.

[0038] A singing input device can analyze a user's past singing history and select the optimal input method. For example, the singing input device can prioritize suggesting singing styles that the user has previously received high marks for. It can also suggest input methods to help the user improve areas they have struggled with in the past. Furthermore, the singing input device can suggest the most effective practice methods based on the user's past singing history. For example, the singing input device can prioritize suggesting singing styles that the user has previously received high marks for. It can also suggest input methods to help the user improve areas they have struggled with in the past. Furthermore, the singing input device can suggest the most effective practice methods based on the user's past singing history. In this way, by analyzing the user's past singing history, the device can provide the optimal input method. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or without a generative AI. For example, the singing input device can input the user's past singing history data into a generative AI, which can then select the optimal input method.

[0039] The singing input device can filter the input based on the user's current physical condition and voice state. For example, if the user has a cold, the singing input device will filter to reduce strain on the voice. It can also encourage singing within a comfortable range, taking into account the user's voice condition, if the user is tired. Furthermore, if the user's voice is in good condition, the singing input device can filter to maximize its potential. This allows for comfortable singing by filtering according to the user's physical condition and voice state. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or without one. For example, the singing input device can input user physical condition data and voice state data into a generative AI, which can then perform the filtering.

[0040] A singing input device can prioritize inputting songs that are highly relevant to the user's geographical location when they sing. For example, if the user is in a specific region, the singing input device will prioritize inputting songs that are popular in that region. Furthermore, if the user is traveling, the singing input device can prioritize inputting songs related to the culture and music of their travel destination. Additionally, if the user is at home, the singing input device can prioritize inputting songs that they have frequently sung at home in the past. This allows the device to provide highly relevant songs by considering the user's geographical location. Some or all of the above processing in the singing input device may be performed using, for example, generative AI, or without generative AI. For example, a singing input device can input the user's geographical location data into a generating AI, which can then select songs that are highly relevant.

[0041] A singing input device can analyze a user's social media activity and input relevant songs when a song is being input. For example, the singing input device can prioritize inputting songs that the user has shared on social media. It can also prioritize inputting songs by artists that the user follows. Furthermore, the singing input device can suggest songs that the user might be interested in based on their social media activity. In this way, by analyzing the user's social media activity, it can provide relevant songs. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or not using a generative AI. For example, the singing input device can input the user's social media activity data into a generative AI, which can then select relevant songs.

[0042] The analysis unit can adjust the level of detail in its analysis based on the importance of each vocal performance. For example, for important vocal performances, the analysis unit can perform a detailed analysis and provide specific feedback. For general vocal performances, the analysis unit can perform a basic analysis and provide concise feedback. Furthermore, for vocal performances that the user is particularly concerned with, the analysis unit can focus its analysis and provide detailed feedback. This allows for more effective feedback by performing analysis according to the importance of each vocal performance. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input vocal importance data into a generative AI, which can then adjust the level of detail in its analysis.

[0043] The analysis unit can apply different analysis algorithms depending on the category of singing during analysis. For example, the analysis unit can apply an analysis algorithm that emphasizes the consistency of rhythm and melody to pop singing. It can also apply an analysis algorithm that emphasizes pitch and expressiveness to classical singing. Furthermore, it can apply an analysis algorithm that emphasizes energy and performance to rock singing. This allows for more accurate analysis by applying an analysis algorithm appropriate to the category of singing. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input singing category data into a generative AI, which can then apply an appropriate analysis algorithm.

[0044] The analysis unit can determine the priority of analysis based on the submission date of the songs during the analysis process. For example, the analysis unit can prioritize the analysis of the most recent songs and provide immediate feedback. The analysis unit can also determine the priority based on a specific deadline set by the user. Furthermore, the analysis unit can also determine the priority by referring to songs previously submitted by the user. This enables faster feedback by providing priority based on the submission date of the songs. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the analysis unit can input the song submission date data into a generative AI, and the generative AI can determine the priority of analysis.

[0045] The analysis unit can adjust the order of analysis based on the relevance of the songs during the analysis. For example, the analysis unit can prioritize analyzing songs related to a specific theme the user has submitted. It can also prioritize analyzing songs that are highly relevant to songs the user has previously submitted. Furthermore, the analysis unit can prioritize analyzing songs in genres that the user is interested in. This allows for more effective feedback by providing an analysis order based on the relevance of the songs. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the analysis unit can input song relevance data into a generative AI, which can then adjust the order of analysis.

[0046] The instruction unit can analyze the user's past singing history during instruction to select the optimal instruction method. For example, the instruction unit can provide instruction based on the singing style in which the user has received high marks in the past. The instruction unit can also provide instruction to address areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past singing history. For example, the instruction unit can provide instruction based on the singing style in which the user has received high marks in the past. The instruction unit can also provide instruction to address areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past singing history. In this way, the instruction unit can provide the optimal instruction method by analyzing the user's past singing history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction unit can input the user's past singing history data into a generative AI, which can then select the optimal instruction method.

[0047] The instruction unit can customize the instruction methods based on the user's current living situation during instruction. For example, if the user is busy, the instruction unit can provide effective instruction in a short amount of time. Alternatively, if the user is relaxed, the instruction unit can provide detailed instruction. Furthermore, if the user has specific goals, the instruction unit can provide instruction tailored to those goals. This allows for more effective instruction by providing instruction methods that are appropriate to the user's living situation. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input user living situation data into a generative AI, which can then customize the optimal instruction methods.

[0048] The instruction unit can select the optimal instruction method during instruction, taking into account the user's geographical location. For example, if the user is in a specific region, the instruction unit can suggest popular instruction methods in that region. Furthermore, if the user is traveling, the instruction unit can suggest instruction methods related to the culture and music of their travel destination. Additionally, if the user is at home, the instruction unit can suggest instruction methods previously used at home. This allows the instruction unit to provide the optimal instruction method by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without one. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal instruction method.

[0049] The instruction department can analyze the user's social media activity during instruction and propose instructional methods. For example, the instruction department can provide instruction based on the songs the user has shared on social media. It can also provide instruction based on the singing styles of artists the user follows. Furthermore, the instruction department can suggest instructional methods that might interest the user based on their social media activity. This allows the instruction department to provide relevant instructional methods by analyzing the user's social media activity. Some or all of the above processing in the instruction department may be performed using, for example, a generative AI, or not. For example, the instruction department can input the user's social media activity data into a generative AI, which can then propose the most suitable instructional method.

[0050] The analysis unit comprehensively evaluates voice, facial expressions, and movements, and can improve the accuracy of the evaluation by referring to the user's past performance data during the evaluation of voice, facial expressions, and movements. For example, the analysis unit can evaluate the current voice based on the user's past voice data. The analysis unit can also evaluate the current facial expressions based on the user's past facial expression data. Furthermore, the analysis unit can also evaluate the current movements based on the user's past movement data. For example, the analysis unit can evaluate the current voice based on the user's past voice data. The analysis unit can also evaluate the current facial expressions based on the user's past facial expression data. Furthermore, the analysis unit can also evaluate the current movements based on the user's past movement data. This improves the accuracy of the evaluation by referring to the user's past performance data. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or it may be performed without a generative AI. For example, the analysis unit can input the user's past performance data into a generative AI, which can then improve the accuracy of the evaluation.

[0051] The analysis unit comprehensively evaluates voice, facial expressions, and movements, and can consider the user's geographical location information when evaluating voice, facial expressions, and movements. For example, if the user is in a specific region, the analysis unit will consider the culture and musical style of that region when evaluating. Furthermore, if the user is traveling, the analysis unit can consider the culture and musical style of the travel destination when evaluating. Additionally, if the user is at home, the analysis unit can evaluate based on past performances at home. This allows for a more appropriate evaluation by considering the user's geographical location information. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input the user's geographical location data into a generative AI, which can then perform the evaluation.

[0052] The instruction unit can automatically learn according to the user's goals, formulate an optimal learning plan, and, when formulating the learning plan, can refer to the user's past learning history to create the optimal plan. For example, the instruction unit can formulate an effective learning plan based on the user's past learning history. It can also formulate a learning plan to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past learning history. For example, the instruction unit can formulate an effective learning plan based on the user's past learning history. It can also formulate a learning plan to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past learning history. In this way, by referring to the user's past learning history, the instruction unit can provide an optimal learning plan. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction unit can input the user's past learning history data into a generative AI, and the generative AI can formulate an optimal learning plan.

[0053] The instructional unit can automatically learn to match the user's goals and formulate an optimal learning plan, taking into account the user's geographical location when formulating the plan. For example, if the user is in a specific region, the instructional unit can formulate a learning plan that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instructional unit can formulate a learning plan that takes into account the culture and musical style of the travel destination. Additionally, if the user is at home, the instructional unit can formulate a plan based on a learning plan previously made at home. This allows the instructional unit to provide an optimal learning plan by considering the user's geographical location. Some or all of the above processing in the instructional unit may be performed using, for example, generative AI, or without generative AI. For example, the instructional team can input the user's geographical location data into a generating AI, which can then create an optimal learning plan.

[0054] The instruction department can adjust the instruction method according to the user's progress, and when adjusting the instruction method, it can refer to the user's past progress data to select the optimal method. For example, the instruction department can select an effective instruction method based on the user's past progress data. It can also select an instruction method to reinforce areas where the user has struggled in the past. Furthermore, the instruction department can suggest the most effective practice method based on the user's past progress data. For example, the instruction department can select an effective instruction method based on the user's past progress data. It can also select an instruction method to reinforce areas where the user has struggled in the past. Furthermore, the instruction department can suggest the most effective practice method based on the user's past progress data. In this way, the instruction department can provide the optimal instruction method by referring to the user's past progress data. Some or all of the above processing in the instruction department may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction department can input the user's past progress data into a generative AI, and the generative AI can select the optimal instruction method.

[0055] The instruction unit can adjust its teaching methods according to the user's progress and select the optimal method by considering the user's geographical location. For example, if the user is in a specific region, the instruction unit can select a teaching method that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can select a teaching method that takes into account the culture and musical style of the travel destination. Additionally, if the user is at home, the instruction unit can select a method based on teaching methods previously used at home. This allows the instruction unit to provide the optimal teaching method by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without one. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal teaching method.

[0056] The instruction unit functions as a continuously operating AI vocal coach, and while continuously operating, it can refer to the user's past usage history to provide optimal instruction. For example, the instruction unit provides effective instruction based on the user's past usage history. It can also provide instruction to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past usage history. For example, the instruction unit provides effective instruction based on the user's past usage history. It can also provide instruction to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past usage history. This allows the instruction unit to provide optimal instruction by referring to the user's past usage history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's past usage history data into a generative AI, which can then provide optimal instruction.

[0057] The instruction unit functions as a continuously operating AI vocal coach, and while operating continuously, it can provide optimal instruction content by considering the user's geographical location. For example, if the user is in a specific region, the instruction unit will provide instruction content that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can provide instruction content that takes into account the culture and musical style of the destination. Additionally, if the user is at home, the instruction unit can provide instruction based on past instruction sessions conducted at home. This allows for the provision of optimal instruction content by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then provide optimal instruction content.

[0058] The instruction unit enables comprehensive instruction through the collaboration of multiple AIs, and when multiple AIs collaborate, it can select the optimal collaboration method by referring to the user's past instruction history. For example, the instruction unit can select an effective collaboration method based on the user's past instruction history. It can also select a collaboration method to complement areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past instruction history. For example, the instruction unit can select an effective collaboration method based on the user's past instruction history. It can also select a collaboration method to complement areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past instruction history. This allows the instruction unit to provide the optimal collaboration method by referring to the user's past instruction history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's past instruction history data into a generative AI, which can then select the optimal collaboration method.

[0059] The instruction unit enables multiple AIs to collaborate to provide comprehensive instruction, and when multiple AIs collaborate, it can select the optimal collaboration method by considering the user's geographical location information. For example, if the user is in a specific region, the instruction unit will select a collaboration method considering the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can select a collaboration method considering the culture and musical style of the travel destination. Additionally, if the user is at home, the instruction unit can select a collaboration method based on past instruction methods used at home. This allows the instruction unit to provide the optimal collaboration method by considering the user's geographical location information. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal collaboration method.

[0060] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0061] The Karaoke AI Maestro system allows users to save their singing data to the cloud and share it with other users. For example, users can make their singing public to other users and receive feedback. Users can also watch other users' singing to learn from them. Furthermore, the system can analyze users' singing data and provide comparison results with other users. This allows users to objectively evaluate their singing ability and enjoy competing with other users. The cloud storage function securely stores users' singing data and makes it accessible at any time. For example, users can access the same data even if they access it from different devices. The cloud storage function also automatically backs up data to prevent data loss. This allows users to use the system with peace of mind.

[0062] The Karaoke AI Maestro system can host virtual concerts based on users' singing data. For example, the system can stream users' singing data in real time for other users to watch. It can also record users' singing data for later viewing. Furthermore, the system can create virtual avatars based on users' singing data and perform on virtual stages. This allows users to share their singing with others and enjoy virtual concerts. The virtual concert function utilizes users' singing data to provide real-time and recorded performances, allowing users to enjoy concerts anytime, anywhere. Additionally, the use of virtual avatars allows users to visually enjoy their own performances.

[0063] The Karaoke AI Maestro System can support music production based on the user's singing data. For example, the system analyzes the user's singing data and assists in creating original songs. It can also suggest accompaniments and arrangements based on the user's singing data. Furthermore, the system can assist in creating lyrics based on the user's singing data. This allows users to create original songs using their own singing data. The music production support function analyzes the user's singing data and suggests optimal accompaniments and arrangements, allowing users to efficiently proceed with their song creation. Additionally, by assisting with lyric creation, users can reflect their emotions and feelings in the lyrics. This allows users to create and enjoy their own unique original songs.

[0064] The Karaoke AI Maestro System can support music education based on the user's singing data. For example, the system analyzes the user's singing data and teaches music theory and how to read sheet music. It can also teach how to play musical instruments based on the user's singing data. Furthermore, it can teach methods of composition and arrangement based on the user's singing data. This allows users to receive music education by utilizing their own singing data. The music education support function analyzes the user's singing data and teaches music theory and how to read sheet music, allowing users to learn the fundamentals of music. Additionally, by teaching how to play musical instruments and methods of composition and arrangement, users can acquire a broad range of musical knowledge. This allows users to enjoy karaoke while receiving music education.

[0065] The following briefly describes the processing flow for example form 1.

[0066] Step 1: The singing input device inputs the user's singing. For example, it can input the user's singing as audio data using a microphone, and input the user's facial expressions and movements as video data using a camera. Furthermore, it can input the user's movements as motion data using a motion sensor. Step 2: The analysis unit analyzes the singing input from the singing input device in real time. For example, it can analyze audio data to evaluate pitch and rhythm, and analyze video data to evaluate facial expressions and movements. Step 3: The instruction department provides individualized instruction based on the results analyzed by the analysis department. For example, they can offer advice on how to correct pitch, improve rhythm, and enhance facial expression.

[0067] (Example of form 2) The Karaoke AI Maestro System, according to an embodiment of the present invention, is a system that revolutionizes the karaoke experience by utilizing generative AI technology. This system begins with the user inputting their singing into a singing input device within a karaoke establishment. This device is equipped with an analysis unit that comprehensively evaluates voice, facial expressions, and movements. Next, the analysis unit analyzes the singing in real time and provides immediate feedback. This feedback provides detailed evaluations of each element, such as pitch, rhythm, and expressiveness. Furthermore, the instruction unit provides individualized guidance, automatically learning to match the user's goals and formulating an optimal learning plan. For example, it provides specific practice methods to compensate for deficiencies in singing ability and advice to enhance expressiveness. It also adjusts the instruction method according to the user's progress, ensuring continuous learning and adaptation. This system functions as a 24-hour AI vocal coach, overcoming time and financial constraints. The cost-effective AI learning system allows users to receive high-quality instruction at a low cost. Additionally, multiple AIs collaborate to provide comprehensive instruction, supporting the user in achieving their goals. In this way, the Karaoke AI Maestro System revolutionizes the karaoke experience through real-time AI analysis and immediate feedback, personalized instruction by autonomous AI agents, multimodal analysis, and continuous learning and adaptation. This allows the Karaoke AI Maestro System to analyze the user's singing in real time and provide personalized instruction, thereby revolutionizing the karaoke experience.

[0068] The karaoke AI maestro system according to this embodiment comprises a singing input device, an analysis unit, and a teaching unit. The singing input device inputs the user's singing. The singing input device inputs the user's singing as audio data, for example, using a microphone. The singing input device can also input the user's facial expressions and movements as video data using a camera. For example, the singing input device records the user's singing with a high-precision microphone and saves it as audio data. Furthermore, the singing input device can also capture the user's facial expressions with a high-resolution camera and save them as video data. Furthermore, the singing input device can detect the user's movements with a motion sensor and save them as movement data. The analysis unit analyzes the singing input by the singing input device in real time. For example, the analysis unit analyzes the audio data to evaluate pitch and rhythm. Furthermore, the analysis unit can also analyze the video data to evaluate facial expressions and movements. For example, the analysis unit analyzes the audio data to evaluate the accuracy of the pitch. Furthermore, the analysis unit can also evaluate the degree of rhythmic consistency. Furthermore, the analysis unit can also analyze video data to evaluate changes in facial expressions. For example, the analysis unit can analyze audio data to evaluate pitch accuracy. The analysis unit can also evaluate rhythmic accuracy. Furthermore, the analysis unit can analyze video data to evaluate changes in facial expressions. The instruction unit provides individual instruction based on the results analyzed by the analysis unit. For example, the instruction unit can advise on how to correct pitch. Furthermore, the instruction unit can also advise on how to improve rhythm. Furthermore, the instruction unit can also provide advice on how to enhance facial expression. For example, the instruction unit can advise on how to correct pitch. Furthermore, the instruction unit can also advise on how to improve rhythm. Furthermore, the instruction unit can also provide advice on how to enhance facial expression. As a result, the karaoke AI maestro system according to this embodiment can revolutionize the karaoke experience by analyzing the user's singing in real time and providing individual instruction.

[0069] The singing input device inputs the user's singing. For example, it can input the user's singing as audio data using a microphone. It can also input the user's facial expressions and movements as video data using a camera. Specifically, the singing input device uses a high-precision microphone to record the user's singing and save it as audio data. This microphone has a noise-canceling function to reduce ambient noise and obtain clear sound. The camera captures the user's facial expressions and movements in high resolution and saves them as video data. This camera can capture subtle changes in the user's facial expressions using facial recognition technology. Furthermore, it is possible to detect the user's movements using a motion sensor and save them as motion data. This motion sensor detects the user's body movements and gestures with high precision, recording the performance during singing in detail. As a result, the singing input device comprehensively collects audio, video, and motion data, allowing for a multifaceted understanding of the user's singing performance. Furthermore, this data is transmitted in real time to a central database, making it immediately accessible to the analysis and instruction departments. This makes it possible to evaluate the user's singing performance in real time and provide immediate feedback.

[0070] The analysis unit analyzes the singing input from the singing input device in real time. For example, the analysis unit analyzes audio data to evaluate pitch and rhythm. It can also analyze video data to evaluate facial expressions and movements. Specifically, the analysis unit uses advanced AI algorithms to analyze audio data and evaluate the accuracy of pitch and the degree of rhythmic consistency. This AI utilizes speech recognition technology to meticulously analyze the user's singing and detect pitch deviations and rhythmic irregularities. It also uses facial recognition technology and motion analysis technology to analyze video data. This allows for a detailed evaluation of the user's facial expressions and movement patterns. For example, the analysis unit reads emotional changes from the user's facial expressions and evaluates the richness of emotional expression during singing. It can also use motion analysis technology to evaluate how the user's body movements and gestures affect singing. Furthermore, by comparing the user's singing performance with past data and data from other users, the analysis unit can identify the degree of improvement in the user's singing performance and specific areas for improvement. This allows the analysis unit to evaluate the user's singing performance from multiple angles and provide detailed feedback.

[0071] The instruction department provides individualized guidance based on the results analyzed by the analysis department. For example, the instruction department may advise on how to correct pitch. They can also advise on how to improve rhythm. Furthermore, they can provide advice on how to enhance facial expression. Specifically, the instruction department uses AI to analyze the user's singing performance and identify individual areas for improvement. This AI suggests specific practice methods to correct pitch deviations and practice methods using a metronome to improve rhythm. In addition, to enhance facial expression, it advises the user on specific facial expression practice methods and techniques for singing with emotion. For example, the instruction department may suggest repeatedly practicing a specific scale as a method for correcting pitch. They may also advise practicing with a metronome and incorporating body movements in time with the rhythm as a method for improving rhythm. Furthermore, to enhance facial expression, they can provide methods for practicing while checking one's facial expressions in a mirror and video tutorials to learn techniques for singing with emotion. In this way, the instruction department can provide specific advice to comprehensively improve the user's singing performance, enabling the user to effectively improve their singing skills.

[0072] The analysis unit can comprehensively evaluate voice, facial expressions, and movements. For example, the analysis unit can analyze audio data to evaluate pitch and rhythm. It can also analyze video data to evaluate facial expressions and movements. For example, the analysis unit can analyze audio data to evaluate the accuracy of pitch. It can also evaluate the degree of rhythmic consistency. Furthermore, it can analyze video data to evaluate changes in facial expressions. By comprehensively evaluating voice, facial expressions, and movements, a more accurate analysis becomes possible. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input audio data, video data, and movement data into a generative AI, which can then evaluate pitch, rhythm, facial expressions, and movements.

[0073] The instruction unit can automatically learn to match the user's goals and formulate an optimal learning plan. For example, the instruction unit can suggest practice methods to improve the user's singing ability. It can also provide advice to enhance the user's expressiveness. Furthermore, the instruction unit can adjust the learning plan according to the user's progress. This enables effective instruction by formulating a learning plan tailored to the user's goals. Some or all of the above processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's goals and progress data into a generative AI, which can then formulate an optimal learning plan.

[0074] The instruction unit can adjust its teaching methods according to the user's progress. For example, the instruction unit can change practice methods as the user's singing ability improves. It can also change the content of advice as the user's expressiveness improves. Furthermore, the instruction unit can optimize its teaching methods based on the user's progress data. For example, the instruction unit can change practice methods as the user's singing ability improves. It can also change the content of advice as the user's expressiveness improves. Furthermore, the instruction unit can optimize its teaching methods based on the user's progress data. This enables continuous learning and adaptation by providing teaching methods tailored to the user's progress. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's progress data into a generative AI, which can then propose the optimal teaching method.

[0075] The instruction unit can function as a continuously operating AI vocal coach. For example, the instruction unit can operate 24 hours a day, 365 days a year, allowing users to receive instruction at any time. The instruction unit can also adjust the timing of instruction to match the user's schedule. Furthermore, the instruction unit can optimize the content of instruction based on the user's usage. This 24 / 7 operation allows users to receive instruction without time constraints. Some or all of the above-described processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input user usage data into a generative AI, which can then suggest optimal instruction content.

[0076] The instruction unit can achieve comprehensive instruction through the collaboration of multiple AIs. For example, the instruction unit can have a voice instruction AI and a facial expression instruction AI working together to provide instruction. Furthermore, the instruction unit can have a movement instruction AI and a voice instruction AI working together to provide instruction. In addition, the instruction unit can have multiple AIs work together to provide comprehensive instruction to the user. For example, the instruction unit can have a voice instruction AI and a facial expression instruction AI working together to provide instruction. Furthermore, the instruction unit can have a movement instruction AI and a voice instruction AI working together to provide instruction. Furthermore, the instruction unit can have multiple AIs work together to provide comprehensive instruction to the user. This allows for more comprehensive instruction through the collaboration of multiple AIs. Some or all of the above-described processes in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input data from the voice instruction AI, facial expression instruction AI, and movement instruction AI into a generative AI, which can then perform comprehensive instruction.

[0077] A singing input device can estimate the user's emotions and adjust the timing of singing input based on those emotions. For example, if the user is nervous, the singing input device will allow time for relaxation before starting singing input. Conversely, if the user is excited, the singing input device can start singing input immediately to harness their energy. Furthermore, if the user is tired, the singing input device can suggest a break and prompt singing input at an appropriate time. This allows for starting singing at a more appropriate time by adjusting the timing of singing input according to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the singing input device may be performed using, for example, a generative AI, or without a generative AI. For example, the singing input device can input user facial expression data into a generative AI, which can then estimate emotions and adjust the timing of the singing input.

[0078] A singing input device can analyze a user's past singing history and select the optimal input method. For example, the singing input device can prioritize suggesting singing styles that the user has previously received high marks for. It can also suggest input methods to help the user improve areas they have struggled with in the past. Furthermore, the singing input device can suggest the most effective practice methods based on the user's past singing history. For example, the singing input device can prioritize suggesting singing styles that the user has previously received high marks for. It can also suggest input methods to help the user improve areas they have struggled with in the past. Furthermore, the singing input device can suggest the most effective practice methods based on the user's past singing history. In this way, by analyzing the user's past singing history, the device can provide the optimal input method. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or without a generative AI. For example, the singing input device can input the user's past singing history data into a generative AI, which can then select the optimal input method.

[0079] The singing input device can filter the input based on the user's current physical condition and voice state. For example, if the user has a cold, the singing input device will filter to reduce strain on the voice. It can also encourage singing within a comfortable range, taking into account the user's voice condition, if the user is tired. Furthermore, if the user's voice is in good condition, the singing input device can filter to maximize its potential. This allows for comfortable singing by filtering according to the user's physical condition and voice state. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or without one. For example, the singing input device can input user physical condition data and voice state data into a generative AI, which can then perform the filtering.

[0080] A singing input device can estimate the user's emotions and determine the priority of songs to input based on those estimated emotions. For example, if the user is relaxed, the singing input device will prioritize songs that are difficult. Conversely, if the user is nervous, the singing input device can also prioritize songs that are easy. Furthermore, if the user is enjoying themselves, the singing input device can prioritize songs that suit the user's preferences. This allows for the selection of more appropriate songs by prioritizing songs according to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the singing input device may be performed using, for example, a generative AI, or without a generative AI. For example, the singing input device can input user emotion data into a generative AI, which can then determine the priority of the singing.

[0081] A singing input device can prioritize inputting songs that are highly relevant to the user's geographical location when they sing. For example, if the user is in a specific region, the singing input device will prioritize inputting songs that are popular in that region. Furthermore, if the user is traveling, the singing input device can prioritize inputting songs related to the culture and music of their travel destination. Additionally, if the user is at home, the singing input device can prioritize inputting songs that they have frequently sung at home in the past. This allows the device to provide highly relevant songs by considering the user's geographical location. Some or all of the above processing in the singing input device may be performed using, for example, generative AI, or without generative AI. For example, a singing input device can input the user's geographical location data into a generating AI, which can then select songs that are highly relevant.

[0082] A singing input device can analyze a user's social media activity and input relevant songs when a song is being input. For example, the singing input device can prioritize inputting songs that the user has shared on social media. It can also prioritize inputting songs by artists that the user follows. Furthermore, the singing input device can suggest songs that the user might be interested in based on their social media activity. In this way, by analyzing the user's social media activity, it can provide relevant songs. Some or all of the above processing in the singing input device may be performed using, for example, a generative AI, or not using a generative AI. For example, the singing input device can input the user's social media activity data into a generative AI, which can then select relevant songs.

[0083] The analysis unit can estimate the user's emotions and adjust the presentation of the analysis based on the estimated emotions. For example, if the user is nervous, the analysis unit provides a simple and easy-to-understand presentation. The analysis unit can also provide detailed analysis results if the user is relaxed. Furthermore, if the user is excited, the analysis unit can provide visually stimulating presentations. This allows for more appropriate analysis results by providing presentations tailored to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input user emotion data into a generating AI, which can then adjust how the analysis is presented.

[0084] The analysis unit can adjust the level of detail in its analysis based on the importance of each vocal performance. For example, for important vocal performances, the analysis unit can perform a detailed analysis and provide specific feedback. For general vocal performances, the analysis unit can perform a basic analysis and provide concise feedback. Furthermore, for vocal performances that the user is particularly concerned with, the analysis unit can focus its analysis and provide detailed feedback. This allows for more effective feedback by performing analysis according to the importance of each vocal performance. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input vocal importance data into a generative AI, which can then adjust the level of detail in its analysis.

[0085] The analysis unit can apply different analysis algorithms depending on the category of singing during analysis. For example, the analysis unit can apply an analysis algorithm that emphasizes the consistency of rhythm and melody to pop singing. It can also apply an analysis algorithm that emphasizes pitch and expressiveness to classical singing. Furthermore, it can apply an analysis algorithm that emphasizes energy and performance to rock singing. This allows for more accurate analysis by applying an analysis algorithm appropriate to the category of singing. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input singing category data into a generative AI, which can then apply an appropriate analysis algorithm.

[0086] The analysis unit can estimate the user's emotions and adjust the length of the analysis based on the estimated emotions. For example, if the user is in a hurry, the analysis unit can provide a short, to-the-point analysis. If the user is relaxed, the analysis unit can also provide a detailed analysis. Furthermore, if the user is excited, the analysis unit can also provide a visually stimulating analysis. This allows for more appropriate analysis results by providing analysis lengths that match the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processing in the analysis unit may be performed using, for example, generative AI, or without generative AI. For example, the analysis unit can input user emotion data into a generating AI, which can then adjust the length of the analysis.

[0087] The analysis unit can determine the priority of analysis based on the submission date of the songs during the analysis process. For example, the analysis unit can prioritize the analysis of the most recent songs and provide immediate feedback. The analysis unit can also determine the priority based on a specific deadline set by the user. Furthermore, the analysis unit can also determine the priority by referring to songs previously submitted by the user. This enables faster feedback by providing priority based on the submission date of the songs. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the analysis unit can input the song submission date data into a generative AI, and the generative AI can determine the priority of analysis.

[0088] The analysis unit can adjust the order of analysis based on the relevance of the songs during the analysis. For example, the analysis unit can prioritize analyzing songs related to a specific theme the user has submitted. It can also prioritize analyzing songs that are highly relevant to songs the user has previously submitted. Furthermore, the analysis unit can prioritize analyzing songs in genres that the user is interested in. This allows for more effective feedback by providing an analysis order based on the relevance of the songs. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the analysis unit can input song relevance data into a generative AI, which can then adjust the order of analysis.

[0089] The instruction unit can estimate the user's emotions and adjust the instruction method based on the estimated emotions. For example, if the user is tense, the instruction unit can provide advice on how to relax. If the user is relaxed, the instruction unit can also provide detailed instruction. Furthermore, if the user is excited, the instruction unit can also provide instruction on how to utilize their energy. This allows for more effective instruction by providing instruction methods tailored to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generative AI. The generative AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the instruction unit may be performed using a generative AI, for example, or without a generative AI. For example, the instruction unit can input the user's emotion data into a generative AI, which can then adjust the instruction method.

[0090] The instruction unit can analyze the user's past singing history during instruction to select the optimal instruction method. For example, the instruction unit can provide instruction based on the singing style in which the user has received high marks in the past. The instruction unit can also provide instruction to address areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past singing history. For example, the instruction unit can provide instruction based on the singing style in which the user has received high marks in the past. The instruction unit can also provide instruction to address areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past singing history. In this way, the instruction unit can provide the optimal instruction method by analyzing the user's past singing history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction unit can input the user's past singing history data into a generative AI, which can then select the optimal instruction method.

[0091] The instruction unit can customize the instruction methods based on the user's current living situation during instruction. For example, if the user is busy, the instruction unit can provide effective instruction in a short amount of time. Alternatively, if the user is relaxed, the instruction unit can provide detailed instruction. Furthermore, if the user has specific goals, the instruction unit can provide instruction tailored to those goals. This allows for more effective instruction by providing instruction methods that are appropriate to the user's living situation. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input user living situation data into a generative AI, which can then customize the optimal instruction methods.

[0092] The instruction unit can estimate the user's emotions and determine the priority of instruction based on the estimated emotions. For example, if the user is relaxed, the instruction unit may prioritize more difficult instruction. If the user is tense, the instruction unit may also prioritize easier instruction. Furthermore, if the user is enjoying themselves, the instruction unit may also prioritize instruction tailored to the user's preferences. This allows for more appropriate instruction by providing instruction priorities that correspond to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the instruction unit may be performed using, for example, generative AI, or not using generative AI. For example, the leadership team can input user emotion data into a generating AI, which can then determine the priority of the guidance.

[0093] The instruction unit can select the optimal instruction method during instruction, taking into account the user's geographical location. For example, if the user is in a specific region, the instruction unit can suggest popular instruction methods in that region. Furthermore, if the user is traveling, the instruction unit can suggest instruction methods related to the culture and music of their travel destination. Additionally, if the user is at home, the instruction unit can suggest instruction methods previously used at home. This allows the instruction unit to provide the optimal instruction method by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without one. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal instruction method.

[0094] The instruction department can analyze the user's social media activity during instruction and propose instructional methods. For example, the instruction department can provide instruction based on the songs the user has shared on social media. It can also provide instruction based on the singing styles of artists the user follows. Furthermore, the instruction department can suggest instructional methods that might interest the user based on their social media activity. This allows the instruction department to provide relevant instructional methods by analyzing the user's social media activity. Some or all of the above processing in the instruction department may be performed using, for example, a generative AI, or not. For example, the instruction department can input the user's social media activity data into a generative AI, which can then propose the most suitable instructional method.

[0095] The analysis unit comprehensively evaluates voice, facial expressions, and movements to estimate the user's emotions and can adjust the evaluation criteria for voice, facial expressions, and movements based on the estimated user emotions. For example, if the user is tense, the analysis unit applies evaluation criteria for a relaxed state. The analysis unit can also apply detailed evaluation criteria if the user is relaxed. Furthermore, if the user is excited, the analysis unit can apply evaluation criteria that emphasize energy. This allows for more appropriate evaluation by providing evaluation criteria that correspond to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input user emotion data into a generating AI, which can then adjust the evaluation criteria for voice, facial expressions, and movements.

[0096] The analysis unit comprehensively evaluates voice, facial expressions, and movements, and can improve the accuracy of the evaluation by referring to the user's past performance data during the evaluation of voice, facial expressions, and movements. For example, the analysis unit can evaluate the current voice based on the user's past voice data. The analysis unit can also evaluate the current facial expressions based on the user's past facial expression data. Furthermore, the analysis unit can also evaluate the current movements based on the user's past movement data. For example, the analysis unit can evaluate the current voice based on the user's past voice data. The analysis unit can also evaluate the current facial expressions based on the user's past facial expression data. Furthermore, the analysis unit can also evaluate the current movements based on the user's past movement data. This improves the accuracy of the evaluation by referring to the user's past performance data. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or it may be performed without a generative AI. For example, the analysis unit can input the user's past performance data into a generative AI, which can then improve the accuracy of the evaluation.

[0097] The analysis unit comprehensively evaluates voice, facial expressions, and movements to estimate the user's emotions and can adjust the display method of the evaluation results based on the estimated user emotions. For example, if the user is tense, the analysis unit provides a simple and highly visible display method. The analysis unit can also provide detailed evaluation results if the user is relaxed. Furthermore, if the user is excited, the analysis unit can provide a visually stimulating display method. For example, if the user is tense, the analysis unit provides a simple and highly visible display method. The analysis unit can also provide detailed evaluation results if the user is relaxed. Furthermore, if the user is excited, the analysis unit can also provide a visually stimulating display method. By providing a display method that matches the user's emotions, more appropriate evaluation results can be provided. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or a generative AI. The generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the analysis unit may be performed using a generative AI, for example, or without a generative AI. For example, the analysis unit can input user emotion data into a generating AI, which can then adjust how the evaluation results are displayed.

[0098] The analysis unit comprehensively evaluates voice, facial expressions, and movements, and can consider the user's geographical location information when evaluating voice, facial expressions, and movements. For example, if the user is in a specific region, the analysis unit will consider the culture and musical style of that region when evaluating. Furthermore, if the user is traveling, the analysis unit can consider the culture and musical style of the travel destination when evaluating. Additionally, if the user is at home, the analysis unit can evaluate based on past performances at home. This allows for a more appropriate evaluation by considering the user's geographical location information. Some or all of the above processing in the analysis unit may be performed using, for example, a generative AI, or without a generative AI. For example, the analysis unit can input the user's geographical location data into a generative AI, which can then perform the evaluation.

[0099] The instruction system can automatically learn to match the user's goals, develop an optimal learning plan, estimate the user's emotions, and adjust the content of the learning plan based on the estimated emotions. For example, if the user is nervous, the instruction system can develop a learning plan that includes relaxation exercises. If the user is relaxed, the instruction system can also develop a more detailed learning plan. Furthermore, if the user is excited, the instruction system can develop a learning plan that helps them utilize their energy. This allows for more effective learning by providing a learning plan tailored to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the instructional unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instructional unit can input user emotion data into a generative AI, which can then adjust the content of the learning plan.

[0100] The instruction unit can automatically learn according to the user's goals, formulate an optimal learning plan, and, when formulating the learning plan, can refer to the user's past learning history to create the optimal plan. For example, the instruction unit can formulate an effective learning plan based on the user's past learning history. It can also formulate a learning plan to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past learning history. For example, the instruction unit can formulate an effective learning plan based on the user's past learning history. It can also formulate a learning plan to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past learning history. In this way, by referring to the user's past learning history, the instruction unit can provide an optimal learning plan. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction unit can input the user's past learning history data into a generative AI, and the generative AI can formulate an optimal learning plan.

[0101] The learning system can automatically learn to match the user's goals, formulate an optimal learning plan, estimate the user's emotions, and prioritize the learning plan based on the estimated emotions. For example, if the user is relaxed, the learning system will prioritize a more difficult learning plan. If the user is stressed, the learning system can also prioritize an easier learning plan. Furthermore, if the user is enjoying themselves, the learning system can prioritize a learning plan tailored to the user's preferences. This allows for more effective learning by providing a learning plan priority based on the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the instructional department may be performed using, for example, a generative AI, or without a generative AI. For example, the instructional department can input user emotion data into a generative AI, which can then determine the priorities of the learning plan.

[0102] The instructional unit can automatically learn to match the user's goals and formulate an optimal learning plan, taking into account the user's geographical location when formulating the plan. For example, if the user is in a specific region, the instructional unit can formulate a learning plan that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instructional unit can formulate a learning plan that takes into account the culture and musical style of the travel destination. Additionally, if the user is at home, the instructional unit can formulate a plan based on a learning plan previously made at home. This allows the instructional unit to provide an optimal learning plan by considering the user's geographical location. Some or all of the above processing in the instructional unit may be performed using, for example, generative AI, or without generative AI. For example, the instructional team can input the user's geographical location data into a generating AI, which can then create an optimal learning plan.

[0103] The instruction unit can adjust its instruction methods according to the user's progress, estimate the user's emotions, and adjust the content of the instruction methods based on the estimated user emotions. For example, if the user is tense, the instruction unit can provide advice on how to relax. If the user is relaxed, the instruction unit can also provide detailed instruction. Furthermore, if the user is excited, the instruction unit can also provide instruction on how to utilize their energy. This enables more effective instruction by providing instruction methods that are tailored to the user's emotions. Emotion estimation is achieved using an emotion estimation function, for example, an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the instruction unit may be performed using, for example, generative AI, or not using generative AI. For example, the instruction department can input user emotion data into a generating AI, which can then adjust the content of the instruction methods.

[0104] The instruction department can adjust the instruction method according to the user's progress, and when adjusting the instruction method, it can refer to the user's past progress data to select the optimal method. For example, the instruction department can select an effective instruction method based on the user's past progress data. It can also select an instruction method to reinforce areas where the user has struggled in the past. Furthermore, the instruction department can suggest the most effective practice method based on the user's past progress data. For example, the instruction department can select an effective instruction method based on the user's past progress data. It can also select an instruction method to reinforce areas where the user has struggled in the past. Furthermore, the instruction department can suggest the most effective practice method based on the user's past progress data. In this way, the instruction department can provide the optimal instruction method by referring to the user's past progress data. Some or all of the above processing in the instruction department may be performed using, for example, a generative AI, or not using a generative AI. For example, the instruction department can input the user's past progress data into a generative AI, and the generative AI can select the optimal instruction method.

[0105] The instruction system can adjust its teaching methods according to the user's progress, estimate the user's emotions, and prioritize teaching methods based on the estimated emotions. For example, if the user is relaxed, the instruction system may prioritize more difficult instruction. If the user is tense, it may prioritize easier instruction. Furthermore, if the user is enjoying themselves, the instruction system may prioritize instruction tailored to the user's preferences. This allows for more effective instruction by providing a priority of teaching methods according to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the instruction department may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction department can input user emotion data into a generative AI, which can then determine the priority of instruction methods.

[0106] The instruction unit can adjust its teaching methods according to the user's progress and select the optimal method by considering the user's geographical location. For example, if the user is in a specific region, the instruction unit can select a teaching method that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can select a teaching method that takes into account the culture and musical style of the travel destination. Additionally, if the user is at home, the instruction unit can select a method based on teaching methods previously used at home. This allows the instruction unit to provide the optimal teaching method by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without one. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal teaching method.

[0107] The instruction unit functions as an always-on AI vocal coach, estimating the user's emotions and adjusting the instruction based on those emotions. For example, if the user is tense, the instruction unit can provide advice on how to relax. If the user is relaxed, the instruction unit can also provide more detailed instruction. Furthermore, if the user is excited, the instruction unit can provide instruction on how to harness that energy. This allows for more effective instruction by providing instruction tailored to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the instruction unit may be performed using, for example, generative AI, or not. For example, the training department can input user emotion data into a generating AI, which can then adjust the training content that is constantly running.

[0108] The instruction unit functions as a continuously operating AI vocal coach, and while continuously operating, it can refer to the user's past usage history to provide optimal instruction. For example, the instruction unit provides effective instruction based on the user's past usage history. It can also provide instruction to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past usage history. For example, the instruction unit provides effective instruction based on the user's past usage history. It can also provide instruction to reinforce areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice methods based on the user's past usage history. This allows the instruction unit to provide optimal instruction by referring to the user's past usage history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's past usage history data into a generative AI, which can then provide optimal instruction.

[0109] The instruction unit functions as an always-on AI vocal coach, estimating the user's emotions and determining instruction priorities based on those emotions. For example, if the user is relaxed, the instruction unit will prioritize more difficult instruction. If the user is nervous, it can prioritize easier instruction. Furthermore, if the user is enjoying themselves, the instruction unit can prioritize instruction tailored to the user's preferences. This allows for more effective instruction by providing instruction priorities that align with the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AI includes, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the guidance department may be performed using, for example, a generative AI, or without a generative AI. For example, the guidance department can input user emotion data into a generative AI, which can then determine the guidance priority for continuous operation.

[0110] The instruction unit functions as a continuously operating AI vocal coach, and while operating continuously, it can provide optimal instruction content by considering the user's geographical location. For example, if the user is in a specific region, the instruction unit will provide instruction content that takes into account the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can provide instruction content that takes into account the culture and musical style of the destination. Additionally, if the user is at home, the instruction unit can provide instruction based on past instruction sessions conducted at home. This allows for the provision of optimal instruction content by considering the user's geographical location. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then provide optimal instruction content.

[0111] The guidance system enables comprehensive guidance through the collaboration of multiple AIs, estimates the user's emotions, and adjusts the collaborative content of the multiple AIs based on the estimated user emotions. For example, if the user is tense, the guidance system can collaborate with an AI that provides advice on how to relax. It can also collaborate with an AI that provides detailed guidance if the user is relaxed. Furthermore, if the user is excited, the guidance system can collaborate with an AI that provides guidance on how to harness that energy. This allows for more effective guidance by providing collaborative content tailored to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AIs include, but are not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the leadership section may be performed using, for example, a generative AI, or without a generative AI. For example, the leadership section may input user emotion data into a generative AI, which can then coordinate the coordinated actions of multiple AIs.

[0112] The instruction unit enables comprehensive instruction through the collaboration of multiple AIs, and when multiple AIs collaborate, it can select the optimal collaboration method by referring to the user's past instruction history. For example, the instruction unit can select an effective collaboration method based on the user's past instruction history. It can also select a collaboration method to complement areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past instruction history. For example, the instruction unit can select an effective collaboration method based on the user's past instruction history. It can also select a collaboration method to complement areas where the user has struggled in the past. Furthermore, the instruction unit can suggest the most effective practice method based on the user's past instruction history. This allows the instruction unit to provide the optimal collaboration method by referring to the user's past instruction history. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's past instruction history data into a generative AI, which can then select the optimal collaboration method.

[0113] The instruction system enables comprehensive instruction through the collaboration of multiple AIs, estimates the user's emotions, and determines the collaboration priority of the multiple AIs based on the estimated user emotions. For example, if the user is relaxed, the instruction system will collaborate with an AI that prioritizes more difficult instruction. If the user is tense, the instruction system can also collaborate with an AI that prioritizes easier instruction. Furthermore, if the user is enjoying themselves, the instruction system can collaborate with an AI that prioritizes instruction tailored to the user's preferences. This allows for more effective instruction by providing collaboration priorities that correspond to the user's emotions. Emotion estimation is achieved using emotion estimation functions, such as an emotion engine or generative AI. Generative AIs include, but are not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above-described processes in the leadership section may be performed using, for example, a generative AI, or without a generative AI. For example, the leadership section may input user emotion data into a generative AI, which can then determine the priority order for collaboration among multiple AIs.

[0114] The instruction unit enables multiple AIs to collaborate to provide comprehensive instruction, and when multiple AIs collaborate, it can select the optimal collaboration method by considering the user's geographical location information. For example, if the user is in a specific region, the instruction unit will select a collaboration method considering the culture and musical style of that region. Furthermore, if the user is traveling, the instruction unit can select a collaboration method considering the culture and musical style of the travel destination. Additionally, if the user is at home, the instruction unit can select a collaboration method based on past instruction methods used at home. This allows the instruction unit to provide the optimal collaboration method by considering the user's geographical location information. Some or all of the above processing in the instruction unit may be performed using, for example, a generative AI, or without a generative AI. For example, the instruction unit can input the user's geographical location data into a generative AI, which can then select the optimal collaboration method.

[0115] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0116] The Karaoke AI Maestro system allows users to save their singing data to the cloud and share it with other users. For example, users can make their singing public to other users and receive feedback. Users can also watch other users' singing to learn from them. Furthermore, the system can analyze users' singing data and provide comparison results with other users. This allows users to objectively evaluate their singing ability and enjoy competing with other users. The cloud storage function securely stores users' singing data and makes it accessible at any time. For example, users can access the same data even if they access it from different devices. The cloud storage function also automatically backs up data to prevent data loss. This allows users to use the system with peace of mind.

[0117] The Karaoke AI Maestro system can create personalized playlists based on the user's singing data. For example, the system analyzes the songs the user frequently sings and their favorite genres, and suggests playlists based on that. The system can also select songs to improve the user's singing ability and expressiveness, and create practice playlists. Furthermore, the system can suggest playlists tailored to the user's emotions and mood. For example, if the user wants to relax, it will suggest relaxing songs; if they are feeling energetic, it will suggest upbeat songs. This allows users to enjoy playlists that match their mood and purpose. The playlist creation function analyzes the user's singing data and selects the most suitable songs, thus providing playlists that match the user's preferences and needs.

[0118] The Karaoke AI Maestro system can host virtual concerts based on users' singing data. For example, the system can stream users' singing data in real time for other users to watch. It can also record users' singing data for later viewing. Furthermore, the system can create virtual avatars based on users' singing data and perform on virtual stages. This allows users to share their singing with others and enjoy virtual concerts. The virtual concert function utilizes users' singing data to provide real-time and recorded performances, allowing users to enjoy concerts anytime, anywhere. Additionally, the use of virtual avatars allows users to visually enjoy their own performances.

[0119] The Karaoke AI Maestro System can support music production based on the user's singing data. For example, the system analyzes the user's singing data and assists in creating original songs. It can also suggest accompaniments and arrangements based on the user's singing data. Furthermore, the system can assist in creating lyrics based on the user's singing data. This allows users to create original songs using their own singing data. The music production support function analyzes the user's singing data and suggests optimal accompaniments and arrangements, allowing users to efficiently proceed with their song creation. Additionally, by assisting with lyric creation, users can reflect their emotions and feelings in the lyrics. This allows users to create and enjoy their own unique original songs.

[0120] The Karaoke AI Maestro System can support health management based on the user's singing data. For example, the system analyzes the user's singing data and evaluates the health of the vocal cords. It can also provide advice on vocal exercises and breathing techniques based on the user's singing data. Furthermore, the system can assess the user's stress level based on the singing data and provide advice for relaxation. This allows users to manage their health using their own singing data. The health management support function analyzes the user's singing data to evaluate the health of their vocal cords and stress levels, allowing users to understand their own health status. Additionally, by providing advice on vocal exercises and breathing techniques, users can maintain healthy singing. This allows users to enjoy karaoke while managing their health.

[0121] The Karaoke AI Maestro System can support music education based on the user's singing data. For example, the system analyzes the user's singing data and teaches music theory and how to read sheet music. It can also teach how to play musical instruments based on the user's singing data. Furthermore, it can teach methods of composition and arrangement based on the user's singing data. This allows users to receive music education by utilizing their own singing data. The music education support function analyzes the user's singing data and teaches music theory and how to read sheet music, allowing users to learn the fundamentals of music. Additionally, by teaching how to play musical instruments and methods of composition and arrangement, users can acquire a broad range of musical knowledge. This allows users to enjoy karaoke while receiving music education.

[0122] The Karaoke AI Maestro system can track emotional changes based on the user's singing data and provide feedback tailored to the user's emotions. For example, the system can detect the joy or sadness the user feels while singing and provide corresponding feedback. It can also offer advice on how to relax if the user is feeling nervous. Furthermore, if the user is excited, it can provide feedback on how to channel that energy. This allows users to receive feedback that matches their emotions. The emotion tracking function analyzes the user's singing data and detects emotional changes, allowing users to receive feedback that aligns with their feelings. By providing advice on relaxation and how to channel energy, users can sing more effectively. This allows users to enjoy karaoke while receiving feedback that matches their emotions.

[0123] The Karaoke AI Maestro System can predict emotional changes based on the user's singing data and provide singing guidance tailored to the user's emotions. For example, the system can predict the joy or sadness the user feels while singing and provide singing guidance accordingly. It can also provide singing guidance to help the user relax if they are feeling nervous. Furthermore, if the user is excited, it can provide singing guidance to help them harness that energy. This allows users to receive singing guidance that matches their emotions. The emotion prediction function analyzes the user's singing data and predicts emotional changes, allowing users to receive singing guidance tailored to their feelings. By providing singing guidance for relaxation and energy utilization, users can sing more effectively. This allows users to enjoy karaoke while receiving singing guidance that matches their emotions.

[0124] The Karaoke AI Maestro System analyzes the user's emotional changes based on their singing data and provides a singing practice plan tailored to their emotions. For example, the system analyzes the joy or sadness the user feels while singing and provides a corresponding practice plan. It can also provide a relaxation plan if the user is feeling nervous, or a plan to harness their energy if they are excited. This allows users to receive singing practice plans that match their emotions. The emotion analysis function analyzes the user's singing data and detects emotional changes, enabling users to receive a singing practice plan that suits their feelings. Furthermore, by providing singing practice plans for relaxation and energy harnessing, users can sing more effectively. This allows users to enjoy karaoke while receiving singing practice plans tailored to their emotions.

[0125] The Karaoke AI Maestro System can evaluate emotional changes based on the user's singing data and provide an evaluation of the singing performance that matches the user's emotions. For example, the system can evaluate the joy or sadness the user feels while singing and provide an evaluation of the singing performance accordingly. The system can also offer advice on how to relax if the user is feeling nervous. Furthermore, if the user is feeling excited, the system can offer advice on how to make the most of that energy. This allows users to receive an evaluation of their singing performance that reflects their emotions. The emotion evaluation function analyzes the user's singing data and evaluates emotional changes, allowing users to receive an evaluation of their singing performance that matches their emotions. By providing advice on relaxation and how to utilize energy, users can sing more effectively. This allows users to enjoy karaoke while receiving an evaluation of their singing performance that reflects their emotions.

[0126] The following briefly describes the processing flow for example form 2.

[0127] Step 1: The singing input device inputs the user's singing. For example, it can input the user's singing as audio data using a microphone, and input the user's facial expressions and movements as video data using a camera. Furthermore, it can input the user's movements as motion data using a motion sensor. Step 2: The analysis unit analyzes the singing input from the singing input device in real time. For example, it can analyze audio data to evaluate pitch and rhythm, and analyze video data to evaluate facial expressions and movements. Step 3: The instruction department provides individualized instruction based on the results analyzed by the analysis department. For example, they can offer advice on how to correct pitch, improve rhythm, and enhance facial expression.

[0128] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0129] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0130] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0131] Each of the multiple elements described above, including the singing input device, analysis unit, and instruction unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the singing input device inputs the user's singing, facial expressions, and movements using the microphone 38B and camera 42 of the smart device 14. The analysis unit is implemented in the specific processing unit 290 of the data processing unit 12, and analyzes audio and video data to evaluate pitch, rhythm, and facial expressions. The instruction unit is implemented in the specific processing unit 290 of the data processing unit 12, and provides individual instruction based on the analysis results. The correspondence between each unit and the device or control unit is not limited to the example described above, and various modifications are possible.

[0132] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0133] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0134] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0135] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0136] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0137] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0138] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0139] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0140] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0141] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0142] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0143] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0144] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0145] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0146] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0147] Each of the multiple elements described above, including the singing input device, analysis unit, and instruction unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the singing input device inputs the user's singing, facial expressions, and movements using the microphone 238 and camera 42 of the smart glasses 214. The analysis unit is implemented in the specific processing unit 290 of the data processing unit 12, and analyzes audio and video data to evaluate pitch, rhythm, and facial expressions. The instruction unit is implemented in the specific processing unit 290 of the data processing unit 12, and provides individual instruction based on the analysis results. The correspondence between each unit and the device or control unit is not limited to the example described above, and various modifications are possible.

[0148] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0149] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0150] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0151] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0152] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0153] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0154] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0155] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0156] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0157] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0158] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0159] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0160] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0161] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0162] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0163] Each of the multiple elements described above, including the singing input device, analysis unit, and instruction unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the singing input device inputs the user's singing, facial expressions, and movements using the microphone 238 and camera 42 of the headset terminal 314. The analysis unit is implemented in the specific processing unit 290 of the data processing unit 12, and analyzes audio and video data to evaluate pitch, rhythm, and facial expressions. The instruction unit is implemented in the specific processing unit 290 of the data processing unit 12, and provides individual instruction based on the analysis results. The correspondence between each unit and the device or control unit is not limited to the example described above, and various modifications are possible.

[0164] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0165] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0166] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0167] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0168] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0169] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0170] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0171] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0172] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0173] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0174] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0175] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0176] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0177] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0178] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0179] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0180] Each of the multiple elements described above, including the singing input device, analysis unit, and instruction unit, is implemented in, for example, at least one of the robot 414 and the data processing unit 12. For example, the singing input device uses the microphone 238 and camera 42 of the robot 414 to input the user's singing, facial expressions, and movements. The analysis unit is implemented in, for example, the specific processing unit 290 of the data processing unit 12, and analyzes audio and video data to evaluate pitch, rhythm, and facial expressions. The instruction unit is implemented in, for example, the specific processing unit 290 of the data processing unit 12, and provides individual instruction based on the analysis results. The correspondence between each unit and the device or control unit is not limited to the example described above, and various modifications are possible.

[0181] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0182] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0183] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0184] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0185] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0186] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0187] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0188] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0189] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0190] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0191] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0192] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0193] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0194] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0195] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0196] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0197] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0198] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0199] (Note 1) A singing input device that inputs the user's singing, An analysis unit that analyzes the singing input by the aforementioned singing input device in real time, The system comprises an instruction unit that provides individualized instruction based on the results of the analysis performed by the aforementioned analysis unit. A system characterized by the following features. (Note 2) The aforementioned analysis unit is The system described in Appendix 1, characterized by comprehensively evaluating voice, facial expressions, and movements. (Note 3) The aforementioned leadership, The system described in Appendix 1 is characterized by automatically learning according to the user's goals and formulating an optimal learning plan. (Note 4) The aforementioned leadership, The system described in Appendix 1, characterized by adjusting the instruction method according to the user's progress. (Note 5) The aforementioned leadership, The system described in Appendix 1 is characterized by functioning as a continuously operating AI vocal coach. (Note 6) The aforementioned leadership, The system described in Appendix 1 is characterized by the collaborative action of multiple AIs to provide comprehensive guidance. (Note 7) The aforementioned singing input device is The system described in Appendix 1, characterized by estimating the user's emotions and adjusting the timing of singing input based on the estimated user emotions. (Note 8) The aforementioned singing input device is The system described in Appendix 1, characterized by analyzing the user's past singing history and selecting the optimal input method. (Note 9) The aforementioned singing input device is The system described in Appendix 1, characterized in that it filters the user's current physical condition and voice state when they input singing. (Note 10) The aforementioned singing input device is The system described in Appendix 1, characterized by estimating the user's emotions and determining the priority of songs to be input based on the estimated user emotions. (Note 11) The aforementioned singing input device is The system described in Appendix 1, characterized in that, when inputting a song, it prioritizes inputting songs that are highly relevant, taking into account the user's geographical location information. (Note 12) The aforementioned singing input device is The system described in Appendix 1, characterized in that, when inputting a song, it analyzes the user's social media activity and inputs related songs. (Note 13) The aforementioned analysis unit is The system described in Appendix 1, characterized by estimating the user's emotions and adjusting the method of expressing the analysis based on the estimated user emotions. (Note 14) The aforementioned analysis unit is The system described in Appendix 1, characterized in that the level of detail of the analysis is adjusted based on the importance of the singing during the analysis. (Note 15) The aforementioned analysis unit is The system described in Appendix 1, characterized in that different analysis algorithms are applied during analysis depending on the category of singing. (Note 16) The aforementioned analysis unit is The system described in Appendix 1, characterized by estimating the user's emotions and adjusting the length of the analysis based on the estimated user emotions. (Note 17) The aforementioned analysis unit is The system described in Appendix 1, characterized in that, during analysis, the priority of analysis is determined based on the timing of the submission of the singing performance. (Note 18) The aforementioned analysis unit is The system described in Appendix 1, characterized in that the order of analysis is adjusted based on the relevance of the singing during the analysis. (Note 19) The aforementioned leadership, The system described in Appendix 1, characterized by estimating the user's emotions and adjusting the instruction method based on the estimated user emotions. (Note 20) The aforementioned leadership, The system described in Appendix 1 is characterized by analyzing the user's past singing history during instruction to select the optimal teaching method. (Note 21) The aforementioned leadership, The system described in Appendix 1, characterized in that, during instruction, the means of instruction are customized based on the user's current living situation. (Note 22) The aforementioned leadership, The system described in Appendix 1, characterized by estimating the user's emotions and determining the priority of instruction based on the estimated user emotions. (Note 23) The aforementioned leadership, The system described in Appendix 1, characterized in that it selects the optimal teaching method while taking into account the user's geographical location information during instruction. (Note 24) The aforementioned leadership, The system described in Appendix 1, characterized in that it analyzes the user's social media activity during instruction and proposes methods of instruction. (Note 25) The aforementioned analysis unit is The system described in Appendix 1 is characterized by comprehensively evaluating voice, facial expressions, and movements, estimating the user's emotions, and adjusting the evaluation criteria for voice, facial expressions, and movements based on the estimated user emotions. (Note 26) The aforementioned analysis unit is The system described in Appendix 1 is characterized by comprehensively evaluating voice, facial expressions, and movements, and by improving the accuracy of the evaluation by referring to the user's past performance data during the evaluation of voice, facial expressions, and movements. (Note 27) The aforementioned analysis unit is The system described in Appendix 1 is characterized by comprehensively evaluating voice, facial expressions, and movements, estimating the user's emotions, and adjusting the display method of the evaluation results based on the estimated user emotions. (Note 28) The aforementioned analysis unit is The system described in Appendix 1, characterized in that it comprehensively evaluates voice, facial expressions, and movements, and takes into account the user's geographical location information when evaluating voice, facial expressions, and movements. (Note 29) The aforementioned leadership, The system described in Appendix 1 is characterized by automatically learning according to the user's goals, formulating an optimal learning plan, estimating the user's emotions, and adjusting the content of the learning plan based on the estimated user emotions. (Note 30) The aforementioned leadership, The system described in Appendix 1 is characterized by automatically learning according to the user's goals, formulating an optimal learning plan, and referring to the user's past learning history when formulating the learning plan. (Note 31) The aforementioned leadership, The system described in Appendix 1 is characterized by automatically learning according to the user's goals, formulating an optimal learning plan, estimating the user's emotions, and determining the priority of the learning plan based on the estimated user emotions. (Note 32) The aforementioned leadership, The system described in Appendix 1 is characterized by automatically learning according to the user's goals, formulating an optimal learning plan, and considering the user's geographical location information when formulating the learning plan. (Note 33) The aforementioned leadership, The system described in Appendix 1, characterized by adjusting the instruction method according to the user's progress, estimating the user's emotions, and adjusting the content of the instruction method based on the estimated user emotions. (Note 34) The aforementioned leadership, The system described in Appendix 1, characterized in that it adjusts the instruction method according to the user's progress and selects the optimal method by referring to the user's past progress data when adjusting the instruction method. (Note 35) The aforementioned leadership, The system described in Appendix 1, characterized by adjusting the instruction method according to the user's progress, estimating the user's emotions, and determining the priority of the instruction method based on the estimated user emotions. (Note 36) The aforementioned leadership, The system described in Appendix 1, characterized in that it adjusts the instruction method according to the user's progress and selects the optimal method when adjusting the instruction method, taking into account the user's geographical location information. (Note 37) The aforementioned leadership, The system described in Appendix 1 is characterized by functioning as a continuously operating AI vocal coach, estimating the user's emotions, and adjusting the continuously operating instruction content based on the estimated user emotions. (Note 38) The aforementioned leadership, The system described in Appendix 1 functions as a continuously operating AI vocal coach, and is characterized by providing optimal instruction content by referring to the user's past usage history while it is running continuously. (Note 39) The aforementioned leadership, The system described in Appendix 1 is characterized by functioning as a continuously operating AI vocal coach, estimating the user's emotions, and determining the priority of instruction based on the estimated user emotions. (Note 40) The aforementioned leadership, The system described in Appendix 1 functions as a continuously operating AI vocal coach, and is characterized by providing optimal instruction content while taking into account the user's geographical location information during continuous operation. (Note 41) The aforementioned leadership, The system described in Appendix 1, characterized in that multiple AIs cooperate to provide comprehensive guidance, estimate the user's emotions, and adjust the cooperative actions of the multiple AIs based on the estimated user emotions. (Note 42) The aforementioned leadership, The system described in Appendix 1, characterized in that multiple AIs cooperate to provide comprehensive guidance, and when the multiple AIs cooperate, they refer to the user's past guidance history to select the optimal method of cooperation. (Note 43) The aforementioned leadership, The system described in Appendix 1, characterized in that multiple AIs cooperate to provide comprehensive guidance, estimate the user's emotions, and determine the priority order for cooperation among the multiple AIs based on the estimated user emotions. (Note 44) The aforementioned leadership, The system described in Appendix 1, characterized in that multiple AIs cooperate to provide comprehensive guidance, and when the multiple AIs cooperate, they select the optimal cooperation method while taking into account the user's geographical location information. [Explanation of symbols]

[0200] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A singing input device that inputs the user's singing, An analysis unit that analyzes the singing input by the aforementioned singing input device in real time, The system comprises an instruction unit that provides individualized instruction based on the results of the analysis performed by the aforementioned analysis unit. A system characterized by the following features.

2. The aforementioned analysis unit is The system according to claim 1, characterized by comprehensively evaluating voice, facial expressions, and movements.

3. The aforementioned leadership, The system according to claim 1, characterized in that it automatically learns according to the user's goals and formulates an optimal learning plan.

4. The aforementioned leadership, The system according to claim 1, characterized in that the instruction method is adjusted according to the user's progress.

5. The aforementioned leadership, The system according to claim 1, characterized in that it functions as a continuously operating AI vocal coach.

6. The aforementioned leadership, The system according to claim 1, characterized in that multiple AIs cooperate to provide comprehensive guidance.

7. The aforementioned singing input device is The system according to claim 1, characterized in that it estimates the user's emotions and adjusts the timing of the singing input based on the estimated user emotions.

8. The aforementioned singing input device is The system according to claim 1, characterized by analyzing the user's past singing history and selecting the optimal input method.

Images