System

Abstract

An object of a system according to an embodiment is to improve English pronunciation ability of a user.SOLUTION: A system includes a pronunciation analysis part, an evaluation part, an instruction part, a recommended music setting part, a pronunciation symbol display part, and an accent setting part. A pronunciation analysis part analyzes English lyrics sung by a user. The evaluation section evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis section. The instruction part instructs pronunciation based on the result evaluated by the evaluation part. The recommended music piece setting section recommends an appropriate music piece based on the English pronunciation capability of the user evaluated by the evaluation section. The phonetic symbol display unit displays phonetic symbols in the lyrics. The accent setting unit gives guidance with emphasis on pronunciation of a specific word or phrase.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology lacks a system that effectively teaches the pronunciation of English lyrics, and this has the drawback of not leading to improvement in English pronunciation skills.

[0005] The system according to the embodiment aims to improve the user's English pronunciation ability. [Means for solving the problem]

[0006] The system according to the embodiment includes a pronunciation analysis unit, an evaluation unit, an instruction unit, a recommended song setting unit, a phonetic symbol display unit, and an accent setting unit. The pronunciation analysis unit analyzes English lyrics sung by a user. The evaluation unit evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit. The instruction unit provides pronunciation instruction based on the results of the evaluation by the evaluation unit. The recommended song setting unit recommends appropriate songs based on the user's English pronunciation ability evaluated by the evaluation unit. The phonetic symbol display unit displays phonetic symbols for the lyrics. The accent setting unit provides instruction with an emphasis on the pronunciation of specific words or phrases. [Effects of the Invention]

[0007] The system according to the embodiment can improve a user's English pronunciation ability. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE INVENTION

[0009] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (for example, a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., audio and / or text). Display 40A displays visible information such as text and images in accordance with instructions from processor 46. Speaker 40B outputs audio in accordance with instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device owned by a user (e.g., a mobile phone, a robot, a home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The English karaoke instruction generation AI system according to the embodiment of the present invention analyzes the English lyrics sung by a user, evaluates the accuracy of the pronunciation, and provides instruction, thereby helping the user improve their English pronunciation.

[0029] An AI system for generating English karaoke instruction includes a pronunciation analysis unit, an evaluation unit, a teaching unit, a recommended song setting unit, a phonetic symbol display unit, and an accent setting unit. The pronunciation analysis unit analyzes English lyrics sung by a user. For example, the pronunciation analysis unit analyzes the user's pronunciation using a speech waveform analysis method. The pronunciation analysis unit can also analyze the user's pronunciation using a phoneme identification method. The pronunciation analysis unit can also analyze peaks and valleys of the speech waveform and identify subtle differences by comparing them with correct pronunciation. The evaluation unit evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit. For example, the evaluation unit evaluates the accuracy of the pronunciation based on the degree of phoneme agreement. The evaluation unit can also evaluate the accuracy of the pronunciation based on the timing of pronunciation. The evaluation unit can also evaluate the accuracy of the pronunciation by comprehensively evaluating the degree of phoneme agreement and the timing of pronunciation. The teaching unit provides pronunciation instruction based on the results of the evaluation by the evaluation unit. For example, the teaching unit provides pronunciation instruction using a feedback format. The teaching unit can also provide pronunciation instruction based on the specific content of the instruction. The teaching unit can also provide pronunciation instruction by comprehensively using the form of feedback and the specific content of the instruction. The recommended song setting unit recommends appropriate songs based on the user's English pronunciation ability evaluated by the evaluation unit. For example, the recommended song setting unit sets recommended songs based on the user's pronunciation ability. The recommended song setting unit can also set recommended songs based on past singing history. The recommended song setting unit can also set recommended songs by comprehensively taking the user's pronunciation ability and past singing history into consideration. The phonetic symbol display unit displays phonetic symbols in lyrics. For example, the phonetic symbol display unit adjusts the timing at which phonetic symbols are displayed. The phonetic symbol display unit can also use a method of highlighting phonetic symbols. The phonetic symbol display unit can also comprehensively adjust the timing at which phonetic symbols are displayed and the method of highlighting them. The accent setting unit provides instruction by focusing on the pronunciation of specific words or phrases. For example, the accent setting unit provides instruction by using a method of highlighting specific words or phrases. The accent setting unit can also provide instruction based on specific content of the instruction.The accent setting unit can also provide instruction by comprehensively using the emphasis method for specific words and phrases and the specific content of instruction. This allows the English karaoke instruction generation AI system according to the embodiment to improve the user's English pronunciation. For example, the system analyzes the English lyrics sung by the user, evaluates the accuracy of the pronunciation, and provides instruction to improve the user's pronunciation. Furthermore, the recommended song setting unit recommends appropriate songs, thereby enhancing the user's learning effectiveness. Furthermore, the phonetic symbol display unit displays phonetic symbols, making it easier for the user to learn correct pronunciation.

[0030] The pronunciation analysis unit can analyze audio waveform data in detail to detect subtle differences in pronunciation. For example, when the generation AI analyzes a user's pronunciation, the pronunciation analysis unit analyzes the audio waveform data in detail to detect subtle differences in pronunciation. For example, when a user pronounces "Hello," the pronunciation analysis unit analyzes the peaks and valleys of the audio waveform and compares it with the correct pronunciation to identify subtle differences. The pronunciation analysis unit also analyzes the stress and rhythm of the user's pronunciation based on the audio waveform data and compares it with the correct pronunciation. For example, in the pronunciation of "how are you," it detects differences in stress and rhythm of each word and points out areas for improvement. The pronunciation analysis unit also analyzes the audio waveform data in detail to extract features of the user's pronunciation. For example, if the pronunciation of a specific phoneme is inaccurate, it points out areas for improvement based on the waveform data of that phoneme. This enables the detection of subtle differences in pronunciation and more accurate pronunciation instruction.

[0031] The pronunciation analysis unit can capture the user's mouth movements with a camera and provide instruction by linking those movements with the voice. For example, the pronunciation analysis unit uses a generation AI to capture the user's mouth movements with a camera and link those movements with the voice to provide instruction. For example, when a user says "Hello," it analyzes the opening of the mouth and the position of the tongue and compares it with the correct pronunciation to point out areas for improvement. The pronunciation analysis unit also links the mouth movements captured by the camera with the voice data to evaluate the accuracy of the user's pronunciation. For example, when pronouncing "how are you," it analyzes the timing of the mouth movements and voice and compares it with the correct pronunciation to point out areas for improvement. The pronunciation analysis unit also links the mouth movements with the voice data to extract the user's pronunciation characteristics. For example, if the pronunciation of a particular phoneme is incorrect, it points out areas for improvement based on the mouth movements and voice data of that phoneme. This linking of mouth movements and voice enables more effective pronunciation instruction.

[0032] The recommended song setting unit takes into account the user's past singing history and learning history when evaluating the user's English proficiency, allowing it to recommend songs with greater accuracy. For example, when the generation AI evaluates the user's English proficiency, the recommended song setting unit takes into account the user's past singing history and learning history to recommend songs with greater accuracy. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. The recommended song setting unit also analyzes the user's past singing history and learning history to evaluate the progress of the user's English proficiency. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. The recommended song setting unit also evaluates the user's English proficiency and suggests recommended songs based on the generation AI's past singing history and learning history. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. In this way, by taking past history into consideration, it is possible to recommend songs with greater accuracy.

[0033] When evaluating a user's English proficiency, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the generation AI evaluates a user's English proficiency, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "Hello" correctly, the next song is suggested. In addition, the recommended song setting unit can analyze the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "how are you" correctly, the next song is suggested. In addition, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "Hello" correctly, the next song is suggested. In this way, progress can be tracked in real time and the recommended songs can be updated at the appropriate time.

[0034] The phonetic symbol display unit can highlight specific phonetic symbols according to the user's pronunciation weaknesses. For example, when the generation AI displays phonetic symbols, the phonetic symbol display unit highlights specific phonetic symbols according to the user's pronunciation weaknesses. For example, if the user's pronunciation of the "h" in "Hello" is weak, the phonetic symbol is highlighted. The phonetic symbol display unit also analyzes the user's pronunciation weaknesses and highlights phonetic symbols according to the weaknesses. For example, if the pronunciation of the "h" in "how are you" is weak, the phonetic symbol is highlighted. The phonetic symbol display unit also highlights specific phonetic symbols based on the user's pronunciation weaknesses. For example, if the pronunciation of a specific phoneme is inaccurate, the phonetic symbol is highlighted. In this way, highlighting phonetic symbols according to the user's pronunciation weaknesses promotes improvement of the user's pronunciation.

[0035] The phonetic symbol display unit can add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, when the generation AI displays phonetic symbols, the phonetic symbol display unit adds an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for "Hello" as "h??lo?". The phonetic symbol display unit can also add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for "how are you." The generation AI can also add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for a specific phoneme. In this way, by adding the audio playback function, the user can listen to accurate pronunciation and learn.

[0036] The accent setting unit can analyze the user's pronunciation habits and provide individually customized instruction when providing instruction with emphasis on the pronunciation of specific words or phrases. For example, when the generation AI provides instruction with emphasis on the pronunciation of specific words or phrases, the accent setting unit analyzes the user's pronunciation habits and provides individually customized instruction. For example, if the user's pronunciation of the "h" in "Hello" is weak, the accent setting unit provides instruction to strengthen that pronunciation. The accent setting unit also analyzes the user's pronunciation habits and provides individually customized instruction based on those habits. For example, if the pronunciation of the "h" in "how are you" is weak, the accent setting unit provides instruction to strengthen that pronunciation. The generation AI also provides individually customized instruction based on the user's pronunciation habits. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit provides instruction to strengthen the pronunciation of that phoneme. In this way, customized instruction based on the user's pronunciation habits is provided, promoting improvement of pronunciation.

[0037] The accent setting unit can provide a function to record a user's pronunciation and play it back for comparison when providing instruction with a focus on the pronunciation of a specific word or phrase. For example, when the generation AI provides instruction with a focus on the pronunciation of a specific word or phrase, the accent setting unit provides a function to record a user's pronunciation and play it back for comparison. For example, when a user pronounces "Hello," the recording is played back and compared with the correct pronunciation. The accent setting unit also provides a function to record a user's pronunciation and play back the recording for comparison. For example, the pronunciation of "how are you" is recorded and compared with the correct pronunciation. The accent setting unit also provides a function to allow the generation AI to record a user's pronunciation and play back the recording for comparison. For example, the pronunciation of a specific phoneme is recorded and compared with the correct pronunciation. This allows the user to check their pronunciation and identify areas for improvement through recording and playback.

[0038] The accent setting unit can compare pronunciations of different accents and suggest the optimal pronunciation to the user when providing instruction focusing on the pronunciation of a specific word or phrase. For example, when the generation AI provides instruction focusing on the pronunciation of a specific word or phrase, the accent setting unit compares American English and British English pronunciations and suggests the optimal pronunciation to the user. For example, when the generation AI provides instruction focusing on the pronunciation of a specific word or phrase, the accent setting unit analyzes the differences between American English and British English in the pronunciation of "Hello" and suggests the optimal pronunciation to the user. The accent setting unit also analyzes pronunciations of different accents and compares them with the user's pronunciation. For example, when the generation AI analyzes the differences between American English and British English in the pronunciation of "how are you," it suggests the optimal pronunciation to the user. The accent setting unit also analyzes pronunciations of different accents and extracts the user's pronunciation characteristics. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit compares American English and British English pronunciations and suggests the optimal pronunciation to the user. In this way, by comparing pronunciations of different accents, the optimal pronunciation can be suggested to the user.

[0039] The accent setting unit can also provide accent instruction for other languages ​​when providing instruction focusing on the pronunciation of specific words or phrases. For example, when the generation AI provides instruction focusing on the pronunciation of specific words or phrases, the accent setting unit can also provide accent instruction for other languages ​​(e.g., French or Spanish). For example, when a user pronounces "Bonjour," the accent setting unit analyzes the pronunciation, compares it with the correct pronunciation, and points out areas for improvement. The accent setting unit also analyzes accents in other languages ​​and compares it with the user's pronunciation. For example, when pronouncing "Hola," the accent setting unit analyzes the differences in accents between French and Spanish and suggests the optimal pronunciation for the user. The accent setting unit also allows the generation AI to analyze accents in other languages ​​and extract the user's pronunciation characteristics. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit compares it with French and Spanish accents and suggests the optimal pronunciation for the user. This allows multilingual learning to be supported by providing accent instruction for other languages.

[0040] When linked with a major karaoke system, the system can store a user's pronunciation data in the cloud and make it accessible from other devices. For example, when a generation AI links with a major karaoke system, the system stores a user's pronunciation data in the cloud and makes it accessible from other devices. For example, pronunciation data sung by a user on a karaoke machine at home can be stored in the cloud and made accessible from a smartphone. The system also builds a system that stores a user's pronunciation data in the cloud and makes it accessible from other devices. For example, pronunciation data sung by a user at a karaoke booth can be stored in the cloud and made accessible from a home computer. The system also allows a generation AI to store a user's pronunciation data in the cloud and make it accessible from other devices. For example, pronunciation data sung by a user on a smartphone can be stored in the cloud and made accessible from a karaoke machine. By storing pronunciation data in the cloud, it can be accessed from other devices, improving convenience.

[0041] When linking with a major karaoke system, the system can analyze a user's pronunciation data and reflect it in the karaoke system's rankings and evaluations. For example, when the generation AI links with a major karaoke system, the system analyzes a user's pronunciation data and reflects it in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "Hello" and reflects the results in the rankings. The system also builds a system that analyzes a user's pronunciation data and reflects the results in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "how are you" and reflects the results in the evaluations. The system also analyzes a user's pronunciation data using the generation AI and reflects the results in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "Hello" and reflects the results in the rankings. In this way, analyzing pronunciation data and reflecting the results in the rankings and evaluations improves user motivation.

[0042] When the system links with a major karaoke system, it can also link with other entertainment systems to provide a comprehensive entertainment experience. For example, when the generative AI links with a major karaoke system, the system can also link with other entertainment systems (e.g., games and movies) to provide a comprehensive entertainment experience. For example, a new entertainment experience that combines karaoke and games can be provided. The system can also link with other entertainment systems to build a system that provides a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and movies can be provided. The system can also link with other entertainment systems to build a system that provides a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and games can be provided. The generative AI can also link with other entertainment systems to provide a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and games can be provided. This makes it possible to provide a comprehensive entertainment experience by linking with other entertainment systems.

[0043] When linked with a major karaoke system, the system can hold karaoke competitions and contests based on users' pronunciation data. For example, when the generation AI links with a major karaoke system, the system holds karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a karaoke competition based on the pronunciation data of a user when he or she sings "Hello." The system also constructs a system for holding karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a contest based on the pronunciation data of a user when he or she sings "how are you." The system also allows the generation AI to hold karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a karaoke competition based on the pronunciation data of a user when he or she sings "Hello." In this way, holding karaoke competitions and contests based on pronunciation data increases users' motivation.

[0044] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0045] The system not only points out areas for improvement in pronunciation based on the user's pronunciation data, but also analyzes the user's pronunciation characteristics and provides a pronunciation practice menu that is individually customized based on those characteristics. For example, if a user has difficulty pronouncing a particular phoneme, the system provides a menu that focuses on practicing that phoneme. The system also analyzes the user's pronunciation characteristics and customizes the pronunciation practice menu based on those characteristics. For example, if a user tends to confuse the pronunciation of "r" and "l," the system provides a practice menu that distinguishes between those phonemes. The system also provides an individually customized pronunciation practice menu based on the user's pronunciation characteristics using a generation AI. For example, if the pronunciation of a particular phoneme is inaccurate, the system provides a practice menu that strengthens that phoneme. This promotes pronunciation improvement by providing a practice menu customized according to the user's pronunciation characteristics.

[0046] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the rhythm and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the rhythm and intonation of the user's pronunciation when pronouncing "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the rhythm and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the rhythm and intonation of each word and points out areas for improvement. The system also provides instruction based on the generation AI's rhythm and intonation of the user's pronunciation. For example, if the rhythm or intonation of a particular phrase is inaccurate, the system provides instruction to strengthen the rhythm and intonation of that phrase. This allows users to learn more natural pronunciation by receiving instruction tailored to rhythm and intonation.

[0047] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the stress and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the stress and intonation of the user's pronunciation of "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the stress and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in stress and intonation between each word and points out areas for improvement. The system also provides instruction based on the generation AI's stress and intonation of the user's pronunciation. For example, if the stress and intonation of a particular phrase is inaccurate, the system provides instruction to strengthen the stress and intonation of that phrase. This allows users to learn more natural pronunciation by receiving instruction based on stress and intonation.

[0048] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the user's pronunciation speed and provides instruction based on the analysis results. For example, the system analyzes the speed at which the user pronounces "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the user's pronunciation speed and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the speed of each word and points out areas for improvement. The system also provides instruction based on the generation AI's pronunciation speed. For example, if the speed of a particular phrase is inaccurate, the system provides instruction to strengthen the speed of that phrase. This allows users to learn more natural pronunciation by receiving instruction based on speed.

[0049] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the volume of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the volume of the user's pronunciation when pronouncing "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the volume of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the volume of each word and points out areas for improvement. The system also provides instruction based on the generation AI's volume of the user's pronunciation. For example, if the volume of a particular phrase is inaccurate, the system provides instruction to strengthen the volume of that phrase. This allows users to learn more natural pronunciation by providing instruction based on volume.

[0050] The processing flow of the first embodiment will be briefly explained below.

[0051] Step 1: The pronunciation analyzer analyzes the English lyrics sung by the user. For example, it analyzes the user's pronunciation using methods such as waveform analysis and phoneme identification. It can also analyze the peaks and valleys of the waveform and compare them with the correct pronunciation to identify subtle differences. Step 2: The evaluation unit evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit. For example, the accuracy of the pronunciation can be evaluated based on the degree of phoneme agreement and the timing of pronunciation, and a comprehensive evaluation can also be performed. Step 3: The teaching department provides pronunciation instruction based on the evaluation results by the evaluation department, for example, by using feedback formats and specific instruction content. Step 4: The recommended song setting unit recommends appropriate songs based on the user's English pronunciation ability evaluated by the evaluation unit. For example, the recommended songs are set based on the user's pronunciation ability and past singing history. Step 5: The phonetic symbol display unit displays phonetic symbols in the lyrics. For example, the timing of displaying the phonetic symbols can be adjusted or a method of highlighting them can be used. Step 6: The accent setting unit provides instruction with emphasis on the pronunciation of specific words or phrases, for example, by using specific accentuation and instruction content for specific words or phrases.

[0052] (Example 2) The English karaoke instruction generation AI system according to the embodiment of the present invention analyzes the English lyrics sung by a user, evaluates the accuracy of the pronunciation, and provides instruction, thereby helping the user improve their English pronunciation.

[0053] An AI system for generating English karaoke instruction includes a pronunciation analysis unit, an evaluation unit, a teaching unit, a recommended song setting unit, a phonetic symbol display unit, and an accent setting unit. The pronunciation analysis unit analyzes English lyrics sung by a user. For example, the pronunciation analysis unit analyzes the user's pronunciation using a speech waveform analysis method. The pronunciation analysis unit can also analyze the user's pronunciation using a phoneme identification method. The pronunciation analysis unit can also analyze peaks and valleys of the speech waveform and identify subtle differences by comparing them with correct pronunciation. The evaluation unit evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit. For example, the evaluation unit evaluates the accuracy of the pronunciation based on the degree of phoneme agreement. The evaluation unit can also evaluate the accuracy of the pronunciation based on the timing of pronunciation. The evaluation unit can also evaluate the accuracy of the pronunciation by comprehensively evaluating the degree of phoneme agreement and the timing of pronunciation. The teaching unit provides pronunciation instruction based on the results of the evaluation by the evaluation unit. For example, the teaching unit provides pronunciation instruction using a feedback format. The teaching unit can also provide pronunciation instruction based on the specific content of the instruction. The teaching unit can also provide pronunciation instruction by comprehensively using the form of feedback and the specific content of the instruction. The recommended song setting unit recommends appropriate songs based on the user's English pronunciation ability evaluated by the evaluation unit. For example, the recommended song setting unit sets recommended songs based on the user's pronunciation ability. The recommended song setting unit can also set recommended songs based on past singing history. The recommended song setting unit can also set recommended songs by comprehensively taking the user's pronunciation ability and past singing history into consideration. The phonetic symbol display unit displays phonetic symbols in lyrics. For example, the phonetic symbol display unit adjusts the timing at which phonetic symbols are displayed. The phonetic symbol display unit can also use a method of highlighting phonetic symbols. The phonetic symbol display unit can also comprehensively adjust the timing at which phonetic symbols are displayed and the method of highlighting them. The accent setting unit provides instruction by focusing on the pronunciation of specific words or phrases. For example, the accent setting unit provides instruction by using a method of highlighting specific words or phrases. The accent setting unit can also provide instruction based on specific content of the instruction.The accent setting unit can also provide instruction by comprehensively using the emphasis method for specific words and phrases and the specific content of instruction. This allows the English karaoke instruction generation AI system according to the embodiment to improve the user's English pronunciation. For example, the system analyzes the English lyrics sung by the user, evaluates the accuracy of the pronunciation, and provides instruction to improve the user's pronunciation. Furthermore, the recommended song setting unit recommends appropriate songs, thereby enhancing the user's learning effectiveness. Furthermore, the phonetic symbol display unit displays phonetic symbols, making it easier for the user to learn correct pronunciation.

[0054] The pronunciation analysis unit can analyze audio waveform data in detail to detect subtle differences in pronunciation. For example, when the generation AI analyzes a user's pronunciation, the pronunciation analysis unit analyzes the audio waveform data in detail to detect subtle differences in pronunciation. For example, when a user pronounces "Hello," the pronunciation analysis unit analyzes the peaks and valleys of the audio waveform and compares it with the correct pronunciation to identify subtle differences. The pronunciation analysis unit also analyzes the stress and rhythm of the user's pronunciation based on the audio waveform data and compares it with the correct pronunciation. For example, in the pronunciation of "how are you," it detects differences in stress and rhythm of each word and points out areas for improvement. The pronunciation analysis unit also analyzes the audio waveform data in detail to extract features of the user's pronunciation. For example, if the pronunciation of a specific phoneme is inaccurate, it points out areas for improvement based on the waveform data of that phoneme. This enables the detection of subtle differences in pronunciation and more accurate pronunciation instruction.

[0055] The pronunciation analysis unit can capture the user's mouth movements with a camera and provide instruction by linking those movements with the voice. For example, the pronunciation analysis unit uses a generation AI to capture the user's mouth movements with a camera and link those movements with the voice to provide instruction. For example, when a user says "Hello," it analyzes the opening of the mouth and the position of the tongue and compares it with the correct pronunciation to point out areas for improvement. The pronunciation analysis unit also links the mouth movements captured by the camera with the voice data to evaluate the accuracy of the user's pronunciation. For example, when pronouncing "how are you," it analyzes the timing of the mouth movements and voice and compares it with the correct pronunciation to point out areas for improvement. The pronunciation analysis unit also links the mouth movements with the voice data to extract the user's pronunciation characteristics. For example, if the pronunciation of a particular phoneme is incorrect, it points out areas for improvement based on the mouth movements and voice data of that phoneme. This linking of mouth movements and voice enables more effective pronunciation instruction.

[0056] The pronunciation analysis unit can use the emotion estimation function to analyze the emotion of the user when singing and point out pronunciation improvements corresponding to the emotion. The pronunciation analysis unit, for example, uses the emotion estimation function to analyze the emotion of the user when singing and point out pronunciation improvements corresponding to the emotion. For example, when a user sings "Hello" with great emotion, the pronunciation analysis unit points out pronunciation improvements corresponding to the strength of the emotion. The pronunciation analysis unit also analyzes the user's emotional state and extracts pronunciation features corresponding to the emotion. For example, when a user sings "how are you" sadly, the pronunciation analysis unit points out pronunciation improvements corresponding to the emotion. The pronunciation analysis unit also uses the emotion estimation function to analyze the user's emotional state in real time and provide pronunciation guidance corresponding to the emotion. For example, when a user sings "Hello" with a joyful emotion, the pronunciation analysis unit points out pronunciation improvements corresponding to the emotion. In this way, by pointing out pronunciation improvements corresponding to the emotion, more emotional singing is possible.

[0057] The recommended song setting unit takes into account the user's past singing history and learning history when evaluating the user's English proficiency, allowing it to recommend songs with greater accuracy. For example, when the generation AI evaluates the user's English proficiency, the recommended song setting unit takes into account the user's past singing history and learning history to recommend songs with greater accuracy. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. The recommended song setting unit also analyzes the user's past singing history and learning history to evaluate the progress of the user's English proficiency. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. The recommended song setting unit also evaluates the user's English proficiency and suggests recommended songs based on the generation AI's past singing history and learning history. For example, it suggests the next song the user should try based on pronunciation evaluations of songs the user has sung in the past. In this way, by taking past history into consideration, it is possible to recommend songs with greater accuracy.

[0058] When evaluating a user's English proficiency, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the generation AI evaluates a user's English proficiency, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "Hello" correctly, the next song is suggested. In addition, the recommended song setting unit can analyze the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "how are you" correctly, the next song is suggested. In addition, the recommended song setting unit can track the user's pronunciation progress in real time and update the recommended songs at the appropriate time. For example, when the user becomes able to pronounce "Hello" correctly, the next song is suggested. In this way, progress can be tracked in real time and the recommended songs can be updated at the appropriate time.

[0059] The recommended song setting unit uses the emotion estimation function to suggest recommended songs according to the user's emotional state, thereby increasing motivation. The recommended song setting unit, for example, uses the emotion estimation function to suggest recommended songs according to the user's emotional state, thereby increasing motivation. For example, when a user sings "Hello" with great emotion, the recommended song setting unit suggests songs that match the emotion. The recommended song setting unit also analyzes the user's emotional state and suggests recommended songs that match the emotion. For example, when a user sings "how are you" sadly, the recommended song setting unit suggests songs that match the emotion. The recommended song setting unit also uses the emotion estimation function to analyze the user's emotional state in real time and suggests recommended songs that match the emotion. For example, when a user sings "Hello" with a happy emotion, the recommended song setting unit suggests songs that match the emotion. In this way, the recommended songs that match the emotion can be suggested, thereby increasing the user's motivation.

[0060] The phonetic symbol display unit can highlight specific phonetic symbols according to the user's pronunciation weaknesses. For example, when the generation AI displays phonetic symbols, the phonetic symbol display unit highlights specific phonetic symbols according to the user's pronunciation weaknesses. For example, if the user's pronunciation of the "h" in "Hello" is weak, the phonetic symbol is highlighted. The phonetic symbol display unit also analyzes the user's pronunciation weaknesses and highlights phonetic symbols according to the weaknesses. For example, if the pronunciation of the "h" in "how are you" is weak, the phonetic symbol is highlighted. The phonetic symbol display unit also highlights specific phonetic symbols based on the user's pronunciation weaknesses. For example, if the pronunciation of a specific phoneme is inaccurate, the phonetic symbol is highlighted. In this way, highlighting phonetic symbols according to the user's pronunciation weaknesses promotes improvement of the user's pronunciation.

[0061] The phonetic symbol display unit can add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, when the generation AI displays phonetic symbols, the phonetic symbol display unit adds an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for "Hello" as "h??lo?". The phonetic symbol display unit can also add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for "how are you." The generation AI can also add an audio playback function for phonetic symbols, allowing the user to listen to accurate pronunciation and learn. For example, it plays the phonetic symbols for a specific phoneme. In this way, by adding the audio playback function, the user can listen to accurate pronunciation and learn.

[0062] The phonetic symbol display unit uses the emotion estimation function to adjust the display method of phonetic symbols according to the emotional state of the user, thereby improving the learning effect. The phonetic symbol display unit uses, for example, the emotion estimation function to adjust the display method of phonetic symbols according to the emotional state of the user, thereby improving the learning effect. For example, when the user sings "Hello" with emotion, the phonetic symbols are highlighted. The phonetic symbol display unit also analyzes the user's emotional state and adjusts the display method of phonetic symbols according to the emotion. For example, when the user sings "how are you" sadly, the phonetic symbols are highlighted. The phonetic symbol display unit also uses the emotion estimation function to analyze the user's emotional state in real time and adjust the display method of phonetic symbols. For example, when the user sings "Hello" with joy, the phonetic symbols are highlighted. In this way, by adjusting the display method according to the emotion, the learning effect can be improved.

[0063] The accent setting unit can analyze the user's pronunciation habits and provide individually customized instruction when providing instruction with emphasis on the pronunciation of specific words or phrases. For example, when the generation AI provides instruction with emphasis on the pronunciation of specific words or phrases, the accent setting unit analyzes the user's pronunciation habits and provides individually customized instruction. For example, if the user's pronunciation of the "h" in "Hello" is weak, the accent setting unit provides instruction to strengthen that pronunciation. The accent setting unit also analyzes the user's pronunciation habits and provides individually customized instruction based on those habits. For example, if the pronunciation of the "h" in "how are you" is weak, the accent setting unit provides instruction to strengthen that pronunciation. The generation AI also provides individually customized instruction based on the user's pronunciation habits. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit provides instruction to strengthen the pronunciation of that phoneme. In this way, customized instruction based on the user's pronunciation habits is provided, promoting improvement of pronunciation.

[0064] The accent setting unit can provide a function to record a user's pronunciation and play it back for comparison when providing instruction with a focus on the pronunciation of a specific word or phrase. For example, when the generation AI provides instruction with a focus on the pronunciation of a specific word or phrase, the accent setting unit provides a function to record a user's pronunciation and play it back for comparison. For example, when a user pronounces "Hello," the recording is played back and compared with the correct pronunciation. The accent setting unit also provides a function to record a user's pronunciation and play back the recording for comparison. For example, the pronunciation of "how are you" is recorded and compared with the correct pronunciation. The accent setting unit also provides a function to allow the generation AI to record a user's pronunciation and play back the recording for comparison. For example, the pronunciation of a specific phoneme is recorded and compared with the correct pronunciation. This allows the user to check their pronunciation and identify areas for improvement through recording and playback.

[0065] The accent setting unit uses the emotion estimation function to adjust the accent instruction method according to the user's emotional state, thereby improving learning effectiveness. The accent setting unit uses, for example, the emotion estimation function to adjust the accent instruction method according to the user's emotional state, thereby improving learning effectiveness. For example, when a user sings "Hello" with emotion, accent instruction is provided according to the emotion. The accent setting unit also analyzes the user's emotional state and adjusts the accent instruction method according to the emotion. For example, when a user sings "how are you" sadly, accent instruction is provided according to the emotion. The accent setting unit also uses the emotion estimation function to analyze the user's emotional state in real time and provide accent instruction according to the emotion. For example, when a user sings "Hello" with joy, accent instruction is provided according to the emotion. In this way, by adjusting the accent instruction method according to the emotion, learning effectiveness can be improved.

[0066] The accent setting unit can compare pronunciations of different accents and suggest the optimal pronunciation to the user when providing instruction focusing on the pronunciation of a specific word or phrase. For example, when the generation AI provides instruction focusing on the pronunciation of a specific word or phrase, the accent setting unit compares American English and British English pronunciations and suggests the optimal pronunciation to the user. For example, when the generation AI provides instruction focusing on the pronunciation of a specific word or phrase, the accent setting unit analyzes the differences between American English and British English in the pronunciation of "Hello" and suggests the optimal pronunciation to the user. The accent setting unit also analyzes pronunciations of different accents and compares them with the user's pronunciation. For example, when the generation AI analyzes the differences between American English and British English in the pronunciation of "how are you," it suggests the optimal pronunciation to the user. The accent setting unit also analyzes pronunciations of different accents and extracts the user's pronunciation characteristics. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit compares American English and British English pronunciations and suggests the optimal pronunciation to the user. In this way, by comparing pronunciations of different accents, the optimal pronunciation can be suggested to the user.

[0067] The accent setting unit can also provide accent instruction for other languages ​​when providing instruction focusing on the pronunciation of specific words or phrases. For example, when the generation AI provides instruction focusing on the pronunciation of specific words or phrases, the accent setting unit can also provide accent instruction for other languages ​​(e.g., French or Spanish). For example, when a user pronounces "Bonjour," the accent setting unit analyzes the pronunciation, compares it with the correct pronunciation, and points out areas for improvement. The accent setting unit also analyzes accents in other languages ​​and compares it with the user's pronunciation. For example, when pronouncing "Hola," the accent setting unit analyzes the differences in accents between French and Spanish and suggests the optimal pronunciation for the user. The accent setting unit also allows the generation AI to analyze accents in other languages ​​and extract the user's pronunciation characteristics. For example, if the pronunciation of a specific phoneme is inaccurate, the accent setting unit compares it with French and Spanish accents and suggests the optimal pronunciation for the user. This allows multilingual learning to be supported by providing accent instruction for other languages.

[0068] The accent setting unit uses the emotion estimation function to adjust the accent instruction method according to the user's emotional state, thereby improving emotion-based learning effectiveness. The accent setting unit, for example, uses the emotion estimation function to adjust the accent instruction method according to the user's emotional state, thereby improving emotion-based learning effectiveness. For example, when a user sings "Hello" with great emotion, accent instruction is provided according to the emotion. The accent setting unit also analyzes the user's emotional state and adjusts the accent instruction method according to the emotion. For example, when a user sings "how are you" sadly, accent instruction is provided according to the emotion. The accent setting unit also uses the emotion estimation function to analyze the user's emotional state in real time and provide accent instruction according to the emotion. For example, when a user sings "Hello" with a joyful emotion, accent instruction is provided according to the emotion. In this way, by adjusting the accent instruction method according to the emotion, emotion-based learning effectiveness can be improved.

[0069] When linked with a major karaoke system, the system can store a user's pronunciation data in the cloud and make it accessible from other devices. For example, when a generation AI links with a major karaoke system, the system stores a user's pronunciation data in the cloud and makes it accessible from other devices. For example, pronunciation data sung by a user on a karaoke machine at home can be stored in the cloud and made accessible from a smartphone. The system also builds a system that stores a user's pronunciation data in the cloud and makes it accessible from other devices. For example, pronunciation data sung by a user at a karaoke booth can be stored in the cloud and made accessible from a home computer. The system also allows a generation AI to store a user's pronunciation data in the cloud and make it accessible from other devices. For example, pronunciation data sung by a user on a smartphone can be stored in the cloud and made accessible from a karaoke machine. By storing pronunciation data in the cloud, it can be accessed from other devices, improving convenience.

[0070] When linking with a major karaoke system, the system can analyze a user's pronunciation data and reflect it in the karaoke system's rankings and evaluations. For example, when the generation AI links with a major karaoke system, the system analyzes a user's pronunciation data and reflects it in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "Hello" and reflects the results in the rankings. The system also builds a system that analyzes a user's pronunciation data and reflects the results in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "how are you" and reflects the results in the evaluations. The system also analyzes a user's pronunciation data using the generation AI and reflects the results in the karaoke system's rankings and evaluations. For example, the system analyzes the pronunciation data when a user sings "Hello" and reflects the results in the rankings. In this way, analyzing pronunciation data and reflecting the results in the rankings and evaluations improves user motivation.

[0071] The system can use the emotion estimation function to customize the interface of the karaoke system according to the emotional state of the user, thereby improving the user experience. For example, the system uses the emotion estimation function to customize the interface of the karaoke system according to the emotional state of the user, thereby improving the user experience. For example, when the user sings "Hello" with emotion, the color and design of the interface are changed. The system also analyzes the emotional state of the user and customizes the interface of the karaoke system according to the emotion. For example, when the user sings "how are you" sadly, the color and design of the interface are changed. The system also uses the emotion estimation function to analyze the emotional state of the user in real time and customize the interface of the karaoke system. For example, when the user sings "Hello" with joy, the color and design of the interface are changed. In this way, the user experience can be improved by customizing the interface according to emotions.

[0072] When the system links with a major karaoke system, it can also link with other entertainment systems to provide a comprehensive entertainment experience. For example, when the generative AI links with a major karaoke system, the system can also link with other entertainment systems (e.g., games and movies) to provide a comprehensive entertainment experience. For example, a new entertainment experience that combines karaoke and games can be provided. The system can also link with other entertainment systems to build a system that provides a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and movies can be provided. The system can also link with other entertainment systems to build a system that provides a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and games can be provided. The generative AI can also link with other entertainment systems to provide a comprehensive entertainment experience to users. For example, a new entertainment experience that combines karaoke and games can be provided. This makes it possible to provide a comprehensive entertainment experience by linking with other entertainment systems.

[0073] When linked with a major karaoke system, the system can hold karaoke competitions and contests based on users' pronunciation data. For example, when the generation AI links with a major karaoke system, the system holds karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a karaoke competition based on the pronunciation data of a user when he or she sings "Hello." The system also constructs a system for holding karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a contest based on the pronunciation data of a user when he or she sings "how are you." The system also allows the generation AI to hold karaoke competitions and contests based on users' pronunciation data. For example, the system determines eligibility to participate in a karaoke competition based on the pronunciation data of a user when he or she sings "Hello." In this way, holding karaoke competitions and contests based on pronunciation data increases users' motivation.

[0074] The system can use the emotion estimation function to customize the interface of the karaoke system according to the emotional state of the user, thereby providing an emotion-based entertainment experience. For example, the system uses the emotion estimation function to customize the interface of the karaoke system according to the emotional state of the user, thereby providing an emotion-based entertainment experience. For example, when the user sings "Hello" with emotion, the color and design of the interface are changed. The system also analyzes the emotional state of the user and customizes the interface of the karaoke system according to the emotion. For example, when the user sings "how are you" sadly, the color and design of the interface are changed. The system also uses the emotion estimation function to analyze the emotional state of the user in real time and customize the interface of the karaoke system. For example, when the user sings "Hello" with joy, the color and design of the interface are changed. In this way, the entertainment experience can be improved by customizing the interface based on emotions.

[0075] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0076] The system not only points out areas for improvement in pronunciation based on the user's pronunciation data, but also analyzes the user's pronunciation characteristics and provides a pronunciation practice menu that is individually customized based on those characteristics. For example, if a user has difficulty pronouncing a particular phoneme, the system provides a menu that focuses on practicing that phoneme. The system also analyzes the user's pronunciation characteristics and customizes the pronunciation practice menu based on those characteristics. For example, if a user tends to confuse the pronunciation of "r" and "l," the system provides a practice menu that distinguishes between those phonemes. The system also provides an individually customized pronunciation practice menu based on the user's pronunciation characteristics using a generation AI. For example, if the pronunciation of a particular phoneme is inaccurate, the system provides a practice menu that strengthens that phoneme. This promotes pronunciation improvement by providing a practice menu customized according to the user's pronunciation characteristics.

[0077] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the rhythm and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the rhythm and intonation of the user's pronunciation when pronouncing "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the rhythm and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the rhythm and intonation of each word and points out areas for improvement. The system also provides instruction based on the generation AI's rhythm and intonation of the user's pronunciation. For example, if the rhythm or intonation of a particular phrase is inaccurate, the system provides instruction to strengthen the rhythm and intonation of that phrase. This allows users to learn more natural pronunciation by receiving instruction tailored to rhythm and intonation.

[0078] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the stress and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the stress and intonation of the user's pronunciation of "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the stress and intonation of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in stress and intonation between each word and points out areas for improvement. The system also provides instruction based on the generation AI's stress and intonation of the user's pronunciation. For example, if the stress and intonation of a particular phrase is inaccurate, the system provides instruction to strengthen the stress and intonation of that phrase. This allows users to learn more natural pronunciation by receiving instruction based on stress and intonation.

[0079] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the user's pronunciation speed and provides instruction based on the analysis results. For example, the system analyzes the speed at which the user pronounces "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the user's pronunciation speed and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the speed of each word and points out areas for improvement. The system also provides instruction based on the generation AI's pronunciation speed. For example, if the speed of a particular phrase is inaccurate, the system provides instruction to strengthen the speed of that phrase. This allows users to learn more natural pronunciation by receiving instruction based on speed.

[0080] The system not only evaluates the accuracy of pronunciation based on the user's pronunciation data, but also analyzes the volume of the user's pronunciation and provides instruction based on the analysis results. For example, the system analyzes the volume of the user's pronunciation when pronouncing "Hello" and compares it with the correct pronunciation to point out areas for improvement. The system also analyzes the volume of the user's pronunciation and provides instruction based on the analysis results. For example, in the pronunciation of "how are you," the system detects differences in the volume of each word and points out areas for improvement. The system also provides instruction based on the generation AI's volume of the user's pronunciation. For example, if the volume of a particular phrase is inaccurate, the system provides instruction to strengthen the volume of that phrase. This allows users to learn more natural pronunciation by providing instruction based on volume.

[0081] Using the emotion estimation function, the system can not only point out areas for pronunciation improvement based on the user's emotional state, but also provide a pronunciation practice menu based on the user's emotions. For example, when a user sings "Hello" with emotion, the system provides a pronunciation practice menu that matches that emotion. The system also analyzes the user's emotional state and customizes the pronunciation practice menu based on that emotion. For example, when a user sings "how are you" sadly, the system provides a pronunciation practice menu that matches that emotion. The system also provides a pronunciation practice menu that is individually customized based on the user's emotional state using the generation AI. For example, the system provides a pronunciation practice menu based on a specific emotion. This allows for more effective pronunciation practice by providing a pronunciation practice menu based on emotions.

[0082] Using its emotion estimation function, the system can not only point out areas for pronunciation improvement based on the user's emotional state, but also provide pronunciation feedback based on the user's emotion. For example, when a user sings "Hello" with emotion, the system provides feedback that corresponds to that emotion. The system also analyzes the user's emotional state and customizes pronunciation feedback based on that emotion. For example, when a user sings "how are you" sadly, the system provides feedback that corresponds to that emotion. The system also uses a generation AI to provide pronunciation feedback that is individually customized based on the user's emotional state. For example, the system provides pronunciation feedback based on a specific emotion. This allows for more effective pronunciation improvement by providing pronunciation feedback based on emotions.

[0083] Using its emotion estimation function, the system can not only point out areas for improvement in pronunciation based on the user's emotional state, but also evaluate the user's pronunciation based on their emotions. For example, when a user sings "Hello" with great emotion, the system evaluates the user's pronunciation based on that emotion. The system also analyzes the user's emotional state and customizes the pronunciation evaluation based on that emotion. For example, when a user sings "how are you" sadly, the system evaluates the user's pronunciation based on that emotion. The system also uses a generation AI to evaluate the user's pronunciation individually, based on the user's emotional state. For example, the system evaluates the user's pronunciation based on a specific emotion. This enables more effective pronunciation improvement by evaluating the user's pronunciation based on emotions.

[0084] Using its emotion estimation function, the system can not only point out areas for pronunciation improvement based on the user's emotional state, but also provide pronunciation instruction based on the user's emotions. For example, when a user sings "Hello" with emotion, the system can provide pronunciation instruction that matches that emotion. The system can also analyze the user's emotional state and customize pronunciation instruction based on that emotion. For example, when a user sings "how are you" sadly, the system can provide pronunciation instruction that matches that emotion. The system's generation AI can also provide individually customized pronunciation instruction based on the user's emotional state. For example, the system can provide pronunciation instruction based on a specific emotion. This allows for more effective pronunciation improvement by providing pronunciation instruction based on emotions.

[0085] Using its emotion estimation function, the system can not only point out areas for improvement in pronunciation based on the user's emotional state, but also suggest pronunciation practice methods based on the user's emotions. For example, when a user sings "Hello" with emotion, the system will suggest pronunciation practice methods that correspond to that emotion. The system can also analyze the user's emotional state and customize pronunciation practice methods based on that emotion. For example, when a user sings "how are you" sadly, the system will suggest pronunciation practice methods that correspond to that emotion. The system's generation AI can also suggest individually customized pronunciation practice methods based on the user's emotional state. For example, the system can suggest pronunciation practice methods based on a specific emotion. This allows for more effective pronunciation practice by suggesting pronunciation practice methods based on emotions.

[0086] The processing flow of the second embodiment will be briefly explained below.

[0087] Step 1: The pronunciation analyzer analyzes the English lyrics sung by the user. For example, it analyzes the user's pronunciation using methods such as waveform analysis and phoneme identification. It can also analyze the peaks and valleys of the waveform and compare them with the correct pronunciation to identify subtle differences. Step 2: The evaluation unit evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit. For example, the accuracy of the pronunciation can be evaluated based on the degree of phoneme agreement and the timing of pronunciation, and a comprehensive evaluation can also be performed. Step 3: The teaching department provides pronunciation instruction based on the evaluation results by the evaluation department, for example, by using feedback formats and specific instruction content. Step 4: The recommended song setting unit recommends appropriate songs based on the user's English pronunciation ability evaluated by the evaluation unit. For example, the recommended songs are set based on the user's pronunciation ability and past singing history. Step 5: The phonetic symbol display unit displays phonetic symbols in the lyrics. For example, the timing of displaying the phonetic symbols can be adjusted or a method of highlighting them can be used. Step 6: The accent setting unit provides instruction with emphasis on the pronunciation of specific words or phrases, for example, by using specific accentuation and instruction content for specific words or phrases.

[0088] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0089] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> Examples of generative AIs include the data generation model 58, such as a neural network model (e.g., a neural network model), and a neural network model (e.g., a neural network model). The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating speech, text data indicating text, and image data indicating an image is also input to the data generation model 58. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0090] Furthermore, the processing by the data processing system 10 described above is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0091] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0092] 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0093] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0094] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0095] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0096] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0097] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0098] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0099] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0100] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0101] In the smart glasses 214, the specific processing is performed by the processor 46. A specific processing program 60 is stored in the storage 50. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0102] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0103] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0104] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0105] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, etc., and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0106] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0107] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0108] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0109] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0110] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0111] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0112] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0113] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0114] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0115] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0116] In the headset type terminal 314, the identification process is performed by the processor 46. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. Note that the headset type terminal 314 may also have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0117] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0118] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0119] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0120] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, etc., and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0121] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0122] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0123] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0124] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0125] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to instructions from the processor 46.

[0126] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0127] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0128] The control object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0129] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0130] The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0131] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0132] In the robot 414, the processor 46 performs the identification process. A identification program 60 is stored in the storage 50. The processor 46 reads the identification program 60 from the storage 50 and executes the read identification program 60 on the RAM 48. The identification process is realized by the processor 46 operating as a control unit 46A in accordance with the identification program 60 executed on the RAM 48. The robot 414 may have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59.

[0133] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 communicates with the server device having the data generation model 58 to obtain a processing result (such as a prediction result) using the data generation model 58. Furthermore, the data processing device 12 may be a server device, or may be a terminal device (for example, a mobile phone, a robot, a home appliance, etc.) owned by a user.

[0134] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0135] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 receives a prompt containing an instruction, as well as inference data such as voice data representing speech, text data representing text, and image data representing an image. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The identification processing unit 290 performs the above-mentioned identification processing using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction. In this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. The data processing device 12 and the like include multiple types of data generation models 58, and the data generation model 58 includes AIs other than the generative AI. The AI ​​other than the generative AI may be, for example, linear regression, logistic regression, decision tree, random forest, support vector machine (SVM), k-means clustering, convolutional neural network (CNN), recurrent neural network (RNN), generative adversarial network (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. The AI ​​may also be an AI agent. When the processes of each of the above-mentioned parts are performed by AI, the processes may be performed in part or entirely by AI, but are not limited to these examples. The processes performed by AI, including the generative AI, may be replaced with rule-based processes.

[0136] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0137] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0138] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion encompasses both emotions and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0139] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0140] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0141] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. Emotions can also be created for robots, cars, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is expressed, and when they approach the ideal, a state of pleasure is expressed. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on speech emotion recognition and brain physiological signal analysis systems for emotions, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the area called "reaction," where sensation is dominant. The right half of the emotion map lists emotions belonging to the area called "situation," where situational awareness is dominant.

[0142] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0143] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[0144] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers including computer 22.

[0145] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0146] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0147] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0148] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. A processor also includes a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[0149] The hardware resource that executes the specific process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0150] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0151] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0152] In the above example, the first to fourth embodiments have been described separately, but some or all of these embodiments may be combined. The smart device 14, smart glasses 214, headset terminal 314, and robot 414 are merely examples, and they may be combined, or other devices may be used. In the above example, the first and second embodiments have been described separately, but they may be combined.

[0153] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0154] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0155] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a pronunciation analysis unit that analyzes the English lyrics sung by the user; an evaluation unit that evaluates the accuracy of the pronunciation analyzed by the pronunciation analysis unit; a teaching unit that provides pronunciation instruction based on the evaluation results by the evaluation unit; a recommended song setting unit that recommends appropriate songs based on the English pronunciation ability of the user evaluated by the evaluation unit; a phonetic symbol display unit that displays phonetic symbols for the lyrics; and an accent setting unit that provides instruction with emphasis on the pronunciation of specific words or phrases. A system characterized by:

2. The pronunciation analysis unit Analyzing voice waveform data in detail to detect subtle differences in pronunciation 2. The system of claim 1.

3. The recommended song setting section When evaluating the user's English ability, the system takes into consideration past singing history and learning history to recommend songs with higher accuracy.

2. The system of claim 1.

4. The phonetic symbol display unit highlighting specific phonetic symbols according to the user's pronunciation weaknesses; 2. The system of claim 1.

5. The accent setting unit When providing the instruction with emphasis on the pronunciation of the specific word or phrase, the pronunciation habits of the user are analyzed and the instruction is individually customized.

2. The system of claim 1.

6. The pronunciation analysis unit Analyzing the emotions of the user when singing and pointing out improvements to the pronunciation according to the emotions.

2. The system of claim 1.

Images