system

Abstract

Provide a system. A means for users to upload an audio file of the original song; A means of analyzing uploaded audio files and extracting lyric data; A means for translating the extracted lyrics data into a target language with emphasis on vowels; A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics; The system includes a means for distributing the generated multilingual singing voice data to users.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Anime songs are popular internationally, and there is a demand for providing the same enjoyment in different languages. However, the time and cost required for translation and playback is a challenge. It is also difficult to ensure that the translated lyrics blend seamlessly with the original melody line, and to play them without losing the singer's vocal quality or emotion. The goal is to solve these problems and provide natural multilingual versions of anime songs in a short period of time. [Means for solving the problem]

[0005] The present invention provides a system that solves the above-mentioned problems by including a means for users to upload audio files of original songs, a means for analyzing the uploaded audio files and extracting lyric data, a means for translating the extracted lyric data into a target language with emphasis on vowels, a means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics, and a means for distributing the generated multilingual singing voice data to users.Furthermore, by including a means for evaluating and correcting whether the vowels in the translated lyrics match those of the original song, and a means for formatting and saving the generated data, it is possible to provide more natural-sounding multilingual versions of anime songs.

[0006] "Original song audio file" refers to the original audio data of an anime song uploaded by a user.

[0007] "User" means a person or entity that uses the system to upload anime songs and generate multilingual versions.

[0008] "Analysis" is the process of extracting lyric data from the uploaded audio file.

[0009] "Lyric data" is text-format lyric information extracted from an audio file.

[0010] "Target language" is the language into which the original song is translated.

[0011] "Vowel-focused translation" is a translation process that ensures that the translated lyrics match the vowels of the original song as closely as possible.

[0012] "Singing voice generation" refers to the creation of new singing voice data based on translated lyrics, preserving the vocal quality and emotion of the original singer.

[0013] "Vocal quality" refers to a singer's unique vocal characteristics.

[0014] "Emotion" refers to the feelings and nuances that a singer expresses when singing.

[0015] "Distribution" means providing the generated multilingual singing voice data to users.

[0016] "Evaluation" involves checking whether the translated lyrics match the vowels in the original song.

[0017] "Correction" refers to adjusting parts of the translated lyrics where the vowels do not match the original song.

[0018] "Formatting" is the process of combining the generated vocal data and lyrics into one file.

[0019] "Storage" means temporarily storing the generated data on a server. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention relates to a system for efficiently realizing the multilingualization of anime songs, specifically to a program for translating lyrics and generating singing voices. Here, the processing of the program of this system is explained in natural language, and an embodiment is shown with concrete examples.

[0042] Program Overview

[0043] The system is realized by combining the following main functions:

[0044] 1. A user uploads an audio file of an anime song.

[0045] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0046] 3. The server translates the lyrics into the target language, focusing on vowels.

[0047] 4. The server generates a singing voice that preserves the vocal quality and emotion based on the translated lyrics.

[0048] 5. The server distributes the generated multilingual singing voice data to the user.

[0049] Program processing (natural language explanation)

[0050] User-uploaded anime songs

[0051] The user selects the original audio file of the anime song using the terminal and uploads it to the system's server. The terminal sends the selected audio file to the server, and the server stores the received audio file.

[0052] Server-based analysis and lyric data extraction

[0053] The server analyzes the received audio file and extracts the lyrics data, using voice recognition software and a lyrics database to obtain text-format lyrics information from the audio file.

[0054] Vowel-focused lyrics translation

[0055] The server translates the extracted lyrics into the target language, placing emphasis on vowels. For example, it uses voice generation AI to translate the lyrics while paying particular attention to matching vowels to ensure that they match the melody line of the original song. It also has a function to evaluate and correct the degree of vowel matching in the translation results.

[0056] Singing voice generation that preserves vocal quality and emotion

[0057] Based on the translated lyrics, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. Specifically, the AI ​​receives the translated lyrics and the original voice as input and reproduces a realistic singing voice. The AI ​​also takes into account vocal expressions and intonation, aiming for a natural result.

[0058] Distribution of generated data

[0059] The server formats the generated multilingual vocal data and translated lyrics, combines them into a single file, and provides a download link to distribute this file to the user's device. The user clicks the link on their device to download the generated data.

[0060] Specific examples

[0061] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." The user uploads the audio file to the server, which extracts the lyrics of "Sakuranbo Kiss" and translates them into English as "Cherry Kiss," emphasizing vowels. The server then uses a voice generation AI to generate "Cherry Kiss," preserving the vocal quality and emotion of the original singer, and delivers it to the user. The user can play "Cherry Kiss" on their device and enjoy the melody line of the original song without any sense of incongruity.

[0062] In this way, the system of the present invention provides a new way to enjoy anime songs internationally by converting anime songs uploaded by users into multiple languages ​​and playing them back in natural singing voices.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server saves the received audio file.

[0066] Step 2:

[0067] The server uses audio analysis software to analyze the uploaded audio files, extracting lyrics data and converting it into text format.

[0068] Step 3:

[0069] The server then calls a text generation AI to translate the extracted lyrics into the target language, placing emphasis on matching vowels to ensure that the melody line of the original song is not out of place.

[0070] Step 4:

[0071] The server evaluates the degree of vowel matching of the generated translated lyrics and corrects any mismatches based on the evaluation results, resulting in more natural-sounding translated lyrics.

[0072] Step 5:

[0073] The server then passes the translated lyrics and the original audio file as input to the AI ​​voice generator, which then generates a new singing voice for the translated lyrics while preserving the vocal quality and emotion of the original singer.

[0074] Step 6:

[0075] The server formats the new vocal data and the translated lyrics, and combines them into a single file. The formatted data is temporarily saved.

[0076] Step 7:

[0077] The server creates a download link for the generated multilingual singing voice data and provides it to the user's device. The user clicks the provided link on their device to download the generated data.

[0078] Step 8:

[0079] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs, which are played with natural singing voices that blend seamlessly with the original melody lines.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] In the multilingualization of anime songs, it is difficult to generate natural-sounding singing voices in different languages ​​while preserving the melody and emotion of the original song. Therefore, there is a need for a system that can generate multilingual singing voices efficiently and with high quality.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes a means for users to upload audio files, a means for receiving and saving the uploaded audio files, a means for analyzing the audio files and extracting text-format lyrics data, a means for translating the extracted lyrics data into a target language with particular emphasis on matching vowels, a means for using a generative AI model to generate singing voices that retain the vocal quality and emotion based on the translated lyrics, and a means for delivering the generated multilingual singing voice data to users. This allows for efficient and high-quality multilingual anime song generation, allowing users to enjoy songs that retain the emotion of the original.

[0085] "User" refers to a person or organization that accesses the System, uploads audio files, and receives multilingual vocal data.

[0086] "Audio file" refers to a file format (e.g., MP3, WAV, AAC, etc.) that contains the audio data of the original anime song.

[0087] "Upload" refers to the act of a user sending an audio file from their own device to a server.

[0088] "Receiving" refers to the process in which the server obtains the audio file sent by the user and saves it in a specified location.

[0089] "Saving" refers to storing the received audio file in a specified directory on the server.

[0090] "Analysis" refers to the process of running a computer program to extract the necessary information from the audio file.

[0091] "Lyrics data" refers to text-format lyrics information extracted from an audio file.

[0092] "Target language" refers to the language into which the original lyrics data is translated.

[0093] "Vowel agreement" refers to the situation where the vowels in the original lyrics and the translated lyrics are similar or identical.

[0094] "Translation" refers to the process of converting the original lyrics data into a target language.

[0095] "Voice quality" refers to the sound quality and characteristics unique to each individual voice.

[0096] "Emotion" refers to the feelings and moods that a singer expresses through lyrics and melody.

[0097] "Voice generation" refers to the process of recreating the vocal quality and emotion based on translated lyrics and synthesizing a new singing voice.

[0098] "Generative AI model" refers to the mathematical model of artificial intelligence used to generate speech.

[0099] "Distribution" refers to the act of providing generated data through a user-accessible link.

[0100] "Multilingual vocal data" refers to vocal data translated into a target language.

[0101] A "prompt sentence" refers to an input command given to a generative AI model to perform appropriate processing.

[0102] The present invention relates to a system for efficiently realizing the multilingualization of anime songs, which allows users to enjoy multilingual versions of songs translated into target languages ​​while preserving the melody and emotion of the original songs.

[0103] System Program Overview

[0104] The system is realized by combining the following main functions:

[0105] 1. The user uploads an audio file.

[0106] 2. The server receives and stores the audio file.

[0107] 3. The server analyzes the audio file and extracts the lyrics data.

[0108] 4. The server translates the lyrics data into the target language, focusing on matching vowels.

[0109] 5. The server generates a singing voice using a generative AI model based on the translated lyrics.

[0110] 6. The server distributes the generated multilingual singing voice data to the user.

[0111] User uploads audio files

[0112] Users access the system's browser interface using their devices, select and upload the anime song audio files, and then click the "Upload" button to send the audio files to the server.

[0113] Receiving and storing audio files on the server

[0114] The server receives the uploaded audio file and stores it in a specified storage directory. The " / uploads" directory is used as an example.

[0115] Server-based audio file analysis and lyric data extraction

[0116] The server sends the saved audio files to speech recognition software (e.g., Google® Cloud Speech-to-Text API) to extract the lyrics data in text format, and optionally references a lyrics database (e.g., Musixmatch API) to verify the extracted results.

[0117] Server-based vowel-focused lyrics translation

[0118] The server sends the extracted lyrics data to a translation API (e.g., Google Translate API) and translates it into the target language. During this process, an algorithm is applied to ensure that vowels match. An example of a prompt is "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds."

[0119] Server-based singing voice generation that preserves vocal quality and emotion

[0120] Based on the translated lyrics, the server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model). An example of a prompt for the generative AI model is, "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion."

[0121] Server-generated data distribution

[0122] The server combines the translated lyrics and the generated vocal data into a single file and generates a link that users can access. Users can click the provided link to download and play the multilingual vocal data.

[0123] Specific examples

[0124] For example, if a user uploads an anime song called "Sakuranbo Kiss," the server extracts the lyrics and translates them into English as "Cherry Kiss." The generative AI model then generates "Cherry Kiss" while preserving the original singer's vocal quality and emotion. Users can download the generated audio file and enjoy the seamless melody and emotion of the original song.

[0125] In this way, the system of the present invention can easily generate anime songs in multiple languages ​​and provide them to users, thereby providing a new way to enjoy anime songs internationally.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1:

[0128] A user accesses the system's web interface using a terminal and selects an audio file of an anime song. The input is an audio file (e.g., in .mp3 format). The user presses the "Upload" button to send the audio file to the server. The output is an upload request and the audio file.

[0129] Step 2:

[0130] The server receives an audio file received from a user as an HTTP request. The server saves this audio file in the " / uploads" directory. The inputs are the upload request and the audio file. The server saves the file and generates a success message. The output is the saved audio file path and a save success message.

[0131] Step 3:

[0132] The server analyzes the saved audio file using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the path to the saved audio file. The server sends the audio file to the API and obtains the lyrics data in text format. The output is the lyrics data in text format.

[0133] Step 4:

[0134] The server sends the lyric data obtained through analysis to the translation API and translates it into the target language. To emphasize matching of vowels in particular, the prompt "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds" is used. The input consists of text-format lyric data and the translation API prompt. The server calls the translation API and obtains the translated lyric data. The output is the translated lyric data.

[0135] Step 5:

[0136] The server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model) based on the translated lyrics data. An example prompt is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion." The input is the translated lyrics data and the prompt from the generative AI model. The server runs the generative AI model and obtains the generated singing voice data. The output is the generated singing voice data.

[0137] Step 6:

[0138] The server combines the generated vocal data and translated lyrics data into a single file. For example, it creates "Cherry_Kiss_generated.mp3." The inputs are the generated vocal data and translated lyrics data. The server performs file generation and creates a single output file. The output is "Cherry_Kiss_generated.mp3."

[0139] Step 7:

[0140] The server generates a download link to deliver the generated file. The input is the formatted output file (e.g. "Cherry_Kiss_generated.mp3"). The server generates a download link and provides it to the user. The output is the download link.

[0141] Step 8:

[0142] The user clicks the provided download link to download the generated multilingual vocal data. The input is the download link. The user receives the file and can play it on their device. The output is the downloaded "Cherry_Kiss_generated.mp3."

[0143] (Application example 1)

[0144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] Conventionally, localizing anime songs into multiple languages ​​and generating natural-sounding singing voices has been time-consuming and costly. It has also been difficult to seamlessly integrate the translated lyrics with the original melody, and providing the translated lyrics in a format that users can easily play on various devices has been a challenge. The present invention aims to solve these problems and provide a system that efficiently translates anime songs into multiple languages ​​with high quality.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0147] In this invention, the server includes: means for users to upload audio files of original songs; means for analyzing the uploaded audio files and extracting lyric data; means for translating the extracted lyric data into a target language with emphasis on vowels; means for generating singing voices that retain the vocal quality and emotion of the original singer based on the translated lyrics; means for distributing the generated multilingual singing voice data to users; means including a tool for enabling playback of the generated multilingual singing voice data; and means for providing two-dimensional language-converted lyrics and singing voice data. This enables multilingualization of anime songs, generation of natural singing voices, and playback on a variety of devices.

[0148] The "original song audio file" is the audio data of the original song uploaded by the user.

[0149] "User uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[0150] The "means for analyzing and extracting lyric data" is a function that analyzes the uploaded audio file and extracts the text data of the lyrics using voice recognition technology.

[0151] The "means for translating into the target language with emphasis on vowels" is a function for translating extracted lyrics data into the target language, with particular emphasis on matching vowels.

[0152] "Means for generating singing voices that retain the vocal quality and emotion of the original singer" refers to a function that generates singing voices in a new language based on translated lyrics while retaining the vocal quality and emotion of the original singer.

[0153] The "means for distributing the generated multilingual singing voice data to the user" is a function for transmitting the generated multilingual singing voice data to the user's terminal.

[0154] "Means including tools that enable playback of the generated multilingual singing voice data" refers to a function that provides the software or hardware necessary to play the generated multilingual singing voice data on a device such as a smartphone or computer.

[0155] The "means for providing lyrics and singing voice data that have been converted into two-dimensional language" is a function that provides the user with lyrics translated into a plurality of target languages ​​and the corresponding singing voice data.

[0156] This invention is a system for efficiently realizing the multilingualization of anime songs, specifically translating lyrics and generating singing voices. The system aims to allow users to upload the original song audio file, have the server analyze it, extract the lyrics data, translate it into the target language, and generate and distribute the multilingual singing voice while preserving the vocal quality and emotion of the original singer.

[0157] System Program Overview

[0158] 1. Upload your audio file

[0159] Users select the audio files of anime songs from their own devices and upload them to the server, using devices such as smartphones or PCs, which then send the files to the server via the Internet.

[0160] 2. Extracting Lyric Data

[0161] The server receives the uploaded audio file, analyzes it, and extracts the lyrics data using voice recognition software such as Google Cloud Speech-to-Text, which extracts the lyrics text data from the audio file.

[0162] 3. Vowel-focused lyrics translation

[0163] The server translates the extracted lyrics into the target language using a translation API such as Google Cloud Translation, paying particular attention to matching vowels.

[0164] 4. Singing voice generation that preserves vocal quality and emotion

[0165] Based on the translated lyrics, the server uses voice generation AI such as DeepVoice and WaveNet to generate a singing voice that retains the vocal quality and emotion of the original singer, resulting in a natural-sounding singing voice that corresponds to the translated lyrics.

[0166] 5. Distribution of generated data

[0167] The generated multilingual singing voice data is distributed to the user's terminal, and the user can play back the generated singing voice data on his / her own terminal.

[0168] Specific use cases

[0169] For example, consider the case where a user uploads the original version of an anime song called "Gurenge." The server analyzes this audio file and extracts the lyrics. The server then translates the lyrics into English, checking for vowel matches and making any necessary corrections. Based on the translated lyrics, the server then uses DeepVoice and WaveNet to generate an English version of "Gurenge," creating a singing voice that retains the vocal quality and emotion of the original singer. Finally, this generated singing data is provided to the user's device as a download link.

[0170] Prompt Sentence Examples

[0171] "I would like this audio file translated into English."

[0172] "I want the translated lyrics to be used to generate a singing voice with the original quality."

[0173] This allows users to easily enjoy multilingual anime songs, providing a new way of enjoying anime among international anime fans.

[0174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0175] Step 1:

[0176] The user selects an audio file of an anime song from their device and uploads it to the server. When uploading, the user uses a specific web page or application to select and send the audio file via the upload button. The server receives and saves this file. The input is the audio file from the user, and the output is the audio file saved on the server.

[0177] Step 2:

[0178] The server analyzes the received audio file and extracts the lyrics data. This analysis is performed using the Google Cloud Speech-to-Text API. The server sends the audio file to the API and obtains the lyrics data in text format. The input is the uploaded audio file, and the output is the lyrics text data.

[0179] Step 3:

[0180] The server translates the extracted lyrics data into the target language. The translation is performed using the Google Cloud Translation API, with particular attention paid to matching vowels. The server sends the lyrics text data to the API and obtains the translated lyrics data. The input is lyrics text data, and the output is translated lyrics data.

[0181] Step 4:

[0182] The server generates a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics data. This process uses voice generation AI models such as DeepVoice and WaveNet. The server inputs the translated lyrics and the original voice features into the voice generation model to generate a realistic singing voice. The input is the translated lyrics data and the original voice feature data, and the output is the generated singing voice data.

[0183] Step 5:

[0184] The server distributes the generated multilingual singing voice data to users. At this time, it provides users with a download link so that they can obtain the data by clicking the link. The input is the generated singing voice data, and the output is the distributed singing voice data.

[0185] Step 6:

[0186] The terminal can play the distributed multilingual vocal data. Users download the data to their smartphones or computers and play the vocals using an audio player application. The input is the distributed multilingual vocal data, and the output is the played vocals.

[0187] This allows users to easily enjoy multilingual versions of anime songs with high-quality singing voices.

[0188] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0189] The present invention is a system that efficiently translates anime songs into multiple languages, and incorporates an emotion engine that can recognize the user's emotions and adjust the emotional expression of the translated lyrics and singing voice accordingly. Below, we will explain the specific program processing of this system in natural language, and provide examples of embodiments.

[0190] Program Overview

[0191] The system includes the following main functions:

[0192] 1. A user uploads an audio file of an anime song.

[0193] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0194] 3. The server translates the lyrics into the target language, focusing on vowels.

[0195] 4. The server recognizes the user's emotions and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on those emotions.

[0196] 5. Based on the translated lyrics, the server generates a singing voice that retains the vocal quality and emotion of the original singer.

[0197] 6. The server distributes the generated multilingual singing voice data to the user.

[0198] Program processing (natural language explanation)

[0199] User-uploaded anime songs

[0200] The user selects an audio file of the original anime song using the terminal and uploads it to the system's server. The terminal transmits the selected audio file to the server, which receives and stores the audio file.

[0201] Server-based analysis and lyric data extraction

[0202] The server uses audio analysis software to analyze the uploaded audio files and extract the lyrics data, which is then converted into text format.

[0203] Vowel-focused lyrics translation

[0204] The server then uses text generation AI to translate the extracted lyrics into the target language. The translation process places particular emphasis on matching vowels to ensure a seamless melody line. The translated lyrics may then be further adjusted based on the user's emotions.

[0205] User emotion recognition and expression adjustment

[0206] The server uses an emotion engine to recognize the user's emotions. This emotion recognition can be performed, for example, by the user inputting facial expressions and voice using a camera or microphone. The emotion engine identifies the user's emotion (e.g., joy, sadness, surprise, etc.) and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on that emotion. For example, if the user is recognized as "sad," the translated lyrics are adjusted to add a nuance of sadness.

[0207] Singing voice generation that preserves vocal quality and emotion

[0208] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. The voice generation AI receives the translated lyrics and the original audio data as input and reproduces a realistic and emotional singing voice.

[0209] Distribution of generated data

[0210] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. This file is temporarily saved and a download link is provided to the user's device. The user clicks the link on their device to download the generated data.

[0211] Specific examples

[0212] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server uses a voice generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer, and delivers it to the user. The user can then play "Cherry Kiss" on their device and enjoy the multilingual anime song that reflects the positive emotion.

[0213] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0214] The processing flow will be explained below.

[0215] Step 1:

[0216] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server receives and stores the uploaded audio file.

[0217] Step 2:

[0218] The server uses audio analysis software to analyze the uploaded audio file, extracting lyrics data as a result and converting it into text format.

[0219] Step 3:

[0220] The server uses text generation AI to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching and adjusting the melody line of the original song so that it fits seamlessly with the original.

[0221] Step 4:

[0222] The user's device uses a camera and microphone to transmit the user's facial expressions and voice to an emotion recognition engine, which then identifies the user's emotions (e.g., joy, sadness, surprise, etc.) in real time.

[0223] Step 5:

[0224] The server adjusts the expression of the translated lyrics based on the user's recognized emotional data. For example, if the user feels sad, the server adds a nuance of sadness to the translated lyrics.

[0225] Step 6:

[0226] The server uses a voice generation AI to generate a new singing voice based on the emotion-adjusted translated lyrics and the original voice data. The generated singing voice retains the vocal quality and emotion of the original singer while also incorporating expressions that correspond to the user's emotions.

[0227] Step 7:

[0228] The server formats the newly generated vocal data and the adjusted translated lyrics, and combines them into a single file. This formatted data is temporarily stored on the server.

[0229] Step 8:

[0230] The server provides a download link for the generated multilingual singing voice data to the user's device, and the user clicks the link using the device to download the generated data.

[0231] Step 9:

[0232] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs. The singing voices they hear are natural and reflect the user's emotions.

[0233] Example 2

[0234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0235] Conventional multilingual speech generation systems have difficulty accurately preserving the emotion of the original song, making it difficult to generate translated lyrics and speech that reflect the user's emotions. Furthermore, there are insufficient means for increasing the degree of agreement between the vowels in the translated lyrics and the melody line, resulting in the generated multilingual singing voices often sounding unnatural.

[0236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0237] In this invention, the server includes means for users to upload audio files, means for analyzing the uploaded audio files and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics and the emotional expression of the generated singing voice, means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics, and means for delivering the generated multilingual singing voice data to users. This makes it possible to generate natural multilingual singing voices that reflect the user's emotions and reduce the sense of incongruity with the melody line of the original song.

[0238] A "user" is someone who uses this system to translate anime songs into multiple languages.

[0239] An "audio file" is a file in which the original music of an anime song is saved in digital format.

[0240] "Uploading means" refers to the method or tool that a user uses to send an audio file to the server.

[0241] "Analysis means" refers to the techniques and tools used to extract lyric data from audio files.

[0242] "Lyric data" is text-format lyric information extracted from an audio file.

[0243] "Target language" is the language into which the original lyrics are translated.

[0244] "Means of translation" refers to the methods and tools used to convert the original lyrics data into the target language.

[0245] "Placing emphasis on vowels" means increasing the degree of consistency of the vowels in the translated lyrics and maintaining compatibility with the melody line.

[0246] "Means for recognizing user emotions" refers to a method or device for analyzing emotions from the user's facial expressions, voice, etc.

[0247] The "means for adjusting emotional expression" refers to a method or tool for adjusting the translated lyrics or the generated singing voice based on the recognized emotions of the user.

[0248] "Preserving vocal quality and emotion" means reproducing the original singer's unique vocal quality and emotion.

[0249] "Generative means" refers to methods and tools for generating new singing voices based on translated lyrics.

[0250] The "distribution means" refers to a method or tool for providing the generated multilingual singing voice data to users.

[0251] "Evaluation methods" are methods or tools that confirm how well the vowels in the translated lyrics match those in the original song.

[0252] "Correction methods" are methods or tools for adjusting and correcting the mismatched vowels.

[0253] "Formatting means" refers to the method or tool used to combine the generated vocal data and translated lyrics into a single file.

[0254] The present invention is a system that allows users to upload audio files of anime songs, analyzes the audio files to extract lyric data, translates the data into target languages, and finally generates and distributes expressive singing voices in multiple languages. Specific embodiments of this system are described below.

[0255] User-uploaded anime songs

[0256] The user selects an anime song audio file using a device (e.g., a smartphone or PC) and clicks the upload button on the operation screen to send the audio file to the system's server. The device then sends the selected audio file to the server, which receives and stores the audio file.

[0257] Server-based analysis and lyric data extraction

[0258] The server analyzes the uploaded audio files using audio analysis software (e.g., Adobe Audition or the open-source Audacity) and extracts the lyrics data. The resulting lyrics data is converted into text format and used for further processing.

[0259] Vowel-focused lyrics translation

[0260] The server uses text generation AI (e.g., GPT-4 (registered trademark)) to translate the extracted lyrics data into the target language. During translation, particular emphasis is placed on matching vowels, and adjustments are made to ensure a natural melody line. The translated lyrics are then used for further processing.

[0261] User emotion recognition and expression adjustment

[0262] Users can use a camera or microphone to input their emotions. The server uses an Emotion API (e.g., Google Cloud Vision API or Microsoft® Azure® Emotion API) to analyze the user's facial expressions and voice and recognize their emotions. Based on the recognized emotions, the server adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice.

[0263] Singing voice generation that preserves vocal quality and emotion

[0264] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a speech generation AI (e.g., Amazon Polly or Google Cloud Text-to-Speech) to generate a singing voice that retains the vocal quality and emotion of the original singer. The speech generation AI receives the translated lyrics and audio data as input and reproduces a realistic, emotional singing voice.

[0265] Distribution of generated data

[0266] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. The server temporarily stores this file and generates a download link for the user's device. The user can click the link on their device to download the generated data.

[0267] Specific examples

[0268] For example, consider a case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server then uses a speech generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[0269] Prompt Sentence Examples

[0270] To translate the Japanese lyrics of the anime song "Sakuranbo Kiss" into English, the following prompt sentence is input to the generative AI model:

[0271] Original lyrics: "Cherry Kiss Lyrics"

[0272] Target language: English

[0273] Emotion: Happy

[0274] Translated lyrics generation

[0275] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0276] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0277] Step 1:

[0278] The user selects and uploads an audio file of an anime song. The user uses the device to select the audio file and clicks the upload button on the operation screen. This operation sends the audio file from the device to the server.

[0279] Input: User selected audio file

[0280] Output: Audio file sent to the server

[0281] Step 2:

[0282] The server receives and stores the audio file. The server receives the audio file uploaded from the device and stores it in temporary storage.

[0283] Input: Uploaded audio file

[0284] Output: Saved audio file

[0285] Step 3:

[0286] The server analyzes the audio file and extracts the lyrics data. The server launches audio analysis software (e.g., Adobe Audition) and analyzes the saved audio file. As a result of the audio analysis, lyrics data is extracted.

[0287] Input: Saved audio file

[0288] Output: Extracted lyrics data (text format)

[0289] Step 4:

[0290] The server translates the lyric data into the target language. The server uses a text generation AI (e.g., GPT-4) to translate the extracted lyric data into the target language. The translated lyrics are generated by inputting the prompt sentence into the generation AI model.

[0291] Input: Extracted lyrics data

[0292] Output: Translated lyrics data

[0293] Step 5:

[0294] The system recognizes the user's emotions and adjusts the lyrics and vocals accordingly. The user sends emotional information to the server using a camera or microphone. The server then analyzes the user's emotions using the Emotion API and adjusts the emotional expression of the translated lyrics data and the generated vocals.

[0295] Input: User emotion information, translated lyrics data

[0296] Output: Adjusted lyrics data, singing data reflecting emotional expressions

[0297] Step 6:

[0298] Generation of singing voices that retains vocal quality and emotion. The server launches a voice generation AI (e.g., Amazon Polly) and inputs the adjusted lyrics data to generate singing voice data. This singing voice data retains the vocal quality and emotion of the original singer.

[0299] Input: Adjusted lyrics data

[0300] Output: Generated singing voice data

[0301] Step 7:

[0302] The generated data is distributed to the user. The server combines the generated vocal data and translated lyrics data into a single file and temporarily stores it. A download link is generated for the user's device, and the user clicks the link to download the file.

[0303] Input: Generated vocal data, translated lyrics data

[0304] Output: Download link, user downloaded file

[0305] (Application example 2)

[0306] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] With conventional music streaming services, users often experience a lack of emotional expression when listening to their favorite anime songs in multiple languages. This can lead to users feeling uneasy about the translated lyrics and vocals, resulting in an unsatisfying music experience. Furthermore, typical translation systems struggle to retain vowel matches and emotional nuances, preventing them from providing a more personalized music experience.

[0308] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload an audio file of an original song, means for analyzing the uploaded audio file and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics according to the emotions, means for generating a singing voice that retains the vocal quality and emotion of the singer of the original song based on the translated lyrics, and means for delivering the generated multilingual singing voice data to the user. This makes it possible to enjoy multilingual anime songs that reflect emotional expressions in real time.

[0309] "Original song audio file" refers to the audio data of the original anime song that the user uploads to the system.

[0310] "Uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[0311] "Means for analyzing and extracting lyric data" means software technology for extracting lyrics in text form from uploaded audio files.

[0312] "Target language" refers to the language into which the translation is to be made.

[0313] "Methods of translating with emphasis on vowels" refers to a translation technique that pays particular attention to the degree of agreement of vowels when translating, in order to maintain consistency with the melody of the original song.

[0314] The "means for recognizing user emotions" is a function that uses input devices such as a camera or microphone to read emotions from the user's facial expressions and tone of voice.

[0315] "Means for adjusting the expression of translated lyrics according to emotions" refers to a technology that changes the content and nuance of translated lyrics based on the recognized emotions of the user.

[0316] "Means for generating singing voices that retain vocal quality and emotion" refers to technology that generates singing voices that reproduce the vocal quality and emotion of the original singer based on translated lyrics and their emotional expressions.

[0317] The "distribution means" refers to a technique for providing the generated multilingual singing voice data to users.

[0318] The present invention relates to a system for translating anime songs into multiple languages ​​and adjusting the emotional expression of the translated lyrics and singing voice in response to the user's emotions. This system is specifically implemented using the following series of means.

[0319] Hardware and Software

[0320] To realize this system, the following hardware and software are used.

[0321] Hardware used

[0322] 1. Device: Smartphone (e.g., Apple iPhone (registered trademark), Samsung Galaxy)

[0323] 2. Server: High-performance server (e.g., cloud-based server infrastructure)

[0324] Software used

[0325] 1. Frontend: User interface using React Native

[0326] 2. Backend: Server-side program using Python (Flask) and Node.js

[0327] 3. Data Processing API:

[0328] Google Cloud Speech-to-Text: Extracting lyrics from audio

[0329] Amazon Translate: Translation into the target language

[0330] OpenAI GPT-4: Text generation and emotional expression adjustment

[0331] AWS (registered trademark) Polly: Emotionally Reflective Singing Voice Generation

[0332] System operation

[0333] 1. User Uploads

[0334] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the application. The device then sends the selected audio file to the server.

[0335] 2. Server-side analysis

[0336] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the lyrics data in text format.

[0337] 3. Lyric translation

[0338] The extracted lyrics data is translated into the target language using Amazon Translate, with particular attention paid to vowel matching.

[0339] 4. Emotion recognition

[0340] The server uses data from the camera and microphone to recognize the user's emotions using emotion recognition libraries such as OpenCV and Librosa, which determine emotions from the user's facial expressions and tone of voice.

[0341] 5. Emotion-Based Regulation

[0342] Using OpenAI GPT-4, the translation adjusts the expression of the lyrics depending on the recognized emotion. For example, if the user is recognized as "sad," the translation will add a nuance of sadness.

[0343] 6. Singing Voice Generation

[0344] Based on the translated lyrics and their emotional expressions, AWS Polly is used to generate a singing voice that retains the vocal quality and emotion of the original singer.

[0345] 7. Data Distribution

[0346] The generated multilingual singing voice data is provided to users in real time, and can be played back at any time via their smartphones.

[0347] Specific examples

[0348] For example, consider a user uploading an anime song called "Cherry Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and uses Amazon Translate to add a more positive nuance to the translated lyrics, "Cherry Kiss." AWS Polly then generates an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[0349] Prompt Sentence Examples

[0350] "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (eg, joy, sadness)."

[0351] Using these prompts, the system generates emotion-based translated lyrics in the target language, providing the user with a highly personalized musical experience.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] Uploading an audio file

[0355] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the app. The audio file selected by the user is sent from the device to the server. The input is the audio file selected by the user, and the output is the audio file saved on the server.

[0356] Step 2:

[0357] Analyzing audio files

[0358] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the text lyrics. This analysis converts the audio signal into text data. The input is the uploaded audio file, and the output is the extracted text lyrics.

[0359] Step 3:

[0360] Lyrics translation

[0361] The server uses Amazon Translate to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching to generate natural lyrics. The input is text-format lyrics data, and the output is the lyrics translated into the target language.

[0362] Step 4:

[0363] emotion recognition

[0364] The user inputs facial expressions and voice using a camera or microphone. The server uses emotion recognition libraries such as OpenCV and Librosa to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the identified emotion data.

[0365] Step 5:

[0366] Adjustment of translated lyrics

[0367] The server uses OpenAI's GPT-4 to adjust the expression of the translated lyrics based on emotion. An example prompt is "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (e.g., joy, sadness)." The input is translated lyrics data and emotion data, and the output is the adjusted translated lyrics data.

[0368] Step 6:

[0369] Singing voice generation

[0370] The server uses AWS Polly to generate a singing voice that retains the vocal quality and emotion of the original singer based on the adjusted translated lyrics data. The input is the adjusted translated lyrics data, and the output is the generated singing voice data.

[0371] Step 7:

[0372] Data distribution

[0373] The server formats the generated multilingual vocal data and provides it to the user. The final data is compiled into a single file, which the user can play in real time via their smartphone. The input is the generated vocal data and the adjusted translated lyrics data, and the output is a file provided to the user as a download link.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0375] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0377] [Second embodiment]

[0378] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0379] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0380] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0382] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0384] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0385] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0386] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0389] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0390] This invention relates to a system for efficiently realizing the multilingualization of anime songs, specifically to a program for translating lyrics and generating singing voices. Here, the processing of the program of this system is explained in natural language, and an embodiment is shown with concrete examples.

[0391] Program Overview

[0392] The system is realized by combining the following main functions:

[0393] 1. A user uploads an audio file of an anime song.

[0394] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0395] 3. The server translates the lyrics into the target language, focusing on vowels.

[0396] 4. The server generates a singing voice that preserves the vocal quality and emotion based on the translated lyrics.

[0397] 5. The server distributes the generated multilingual singing voice data to the user.

[0398] Program processing (natural language explanation)

[0399] User-uploaded anime songs

[0400] The user selects the original audio file of the anime song using the terminal and uploads it to the system's server. The terminal sends the selected audio file to the server, and the server stores the received audio file.

[0401] Server-based analysis and lyric data extraction

[0402] The server analyzes the received audio file and extracts the lyrics data, using voice recognition software and a lyrics database to obtain text-format lyrics information from the audio file.

[0403] Vowel-focused lyrics translation

[0404] The server translates the extracted lyrics into the target language, placing emphasis on vowels. For example, it uses voice generation AI to translate the lyrics while paying particular attention to matching vowels to ensure that they match the melody line of the original song. It also has a function to evaluate and correct the degree of vowel matching in the translation results.

[0405] Singing voice generation that preserves vocal quality and emotion

[0406] Based on the translated lyrics, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. Specifically, the AI ​​receives the translated lyrics and the original voice as input and reproduces a realistic singing voice. The AI ​​also takes into account vocal expressions and intonation, aiming for a natural result.

[0407] Distribution of generated data

[0408] The server formats the generated multilingual vocal data and translated lyrics, combines them into a single file, and provides a download link to distribute this file to the user's device. The user clicks the link on their device to download the generated data.

[0409] Specific examples

[0410] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." The user uploads the audio file to the server, which extracts the lyrics of "Sakuranbo Kiss" and translates them into English as "Cherry Kiss," emphasizing vowels. The server then uses a voice generation AI to generate "Cherry Kiss," preserving the vocal quality and emotion of the original singer, and delivers it to the user. The user can play "Cherry Kiss" on their device and enjoy the melody line of the original song without any sense of incongruity.

[0411] In this way, the system of the present invention provides a new way to enjoy anime songs internationally by converting anime songs uploaded by users into multiple languages ​​and playing them back in natural singing voices.

[0412] The processing flow will be explained below.

[0413] Step 1:

[0414] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server saves the received audio file.

[0415] Step 2:

[0416] The server uses audio analysis software to analyze the uploaded audio files, extracting lyrics data and converting it into text format.

[0417] Step 3:

[0418] The server then calls a text generation AI to translate the extracted lyrics into the target language, placing emphasis on matching vowels to ensure that the melody line of the original song is not out of place.

[0419] Step 4:

[0420] The server evaluates the degree of vowel matching of the generated translated lyrics and corrects any mismatches based on the evaluation results, resulting in more natural-sounding translated lyrics.

[0421] Step 5:

[0422] The server then passes the translated lyrics and the original audio file as input to the AI ​​voice generator, which then generates a new singing voice for the translated lyrics while preserving the vocal quality and emotion of the original singer.

[0423] Step 6:

[0424] The server formats the new vocal data and the translated lyrics, and combines them into a single file. The formatted data is temporarily saved.

[0425] Step 7:

[0426] The server creates a download link for the generated multilingual singing voice data and provides it to the user's device. The user clicks the provided link on their device to download the generated data.

[0427] Step 8:

[0428] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs, which are played with natural singing voices that blend seamlessly with the original melody lines.

[0429] Example 1

[0430] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0431] In the multilingualization of anime songs, it is difficult to generate natural-sounding singing voices in different languages ​​while preserving the melody and emotion of the original song. Therefore, there is a need for a system that can generate multilingual singing voices efficiently and with high quality.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0433] In this invention, the server includes a means for users to upload audio files, a means for receiving and saving the uploaded audio files, a means for analyzing the audio files and extracting text-format lyrics data, a means for translating the extracted lyrics data into a target language with particular emphasis on matching vowels, a means for using a generative AI model to generate singing voices that retain the vocal quality and emotion based on the translated lyrics, and a means for delivering the generated multilingual singing voice data to users. This allows for efficient and high-quality multilingual anime song generation, allowing users to enjoy songs that retain the emotion of the original.

[0434] "User" refers to a person or organization that accesses the System, uploads audio files, and receives multilingual vocal data.

[0435] "Audio file" refers to a file format (e.g., MP3, WAV, AAC, etc.) that contains the audio data of the original anime song.

[0436] "Upload" refers to the act of a user sending an audio file from their own device to a server.

[0437] "Receiving" refers to the process in which the server obtains the audio file sent by the user and saves it in a specified location.

[0438] "Saving" refers to storing the received audio file in a specified directory on the server.

[0439] "Analysis" refers to the process of running a computer program to extract the necessary information from the audio file.

[0440] "Lyrics data" refers to text-format lyrics information extracted from an audio file.

[0441] "Target language" refers to the language into which the original lyrics data is translated.

[0442] "Vowel agreement" refers to the situation where the vowels in the original lyrics and the translated lyrics are similar or identical.

[0443] "Translation" refers to the process of converting the original lyrics data into a target language.

[0444] "Voice quality" refers to the sound quality and characteristics unique to each individual voice.

[0445] "Emotion" refers to the feelings and moods that a singer expresses through lyrics and melody.

[0446] "Voice generation" refers to the process of recreating the vocal quality and emotion based on translated lyrics and synthesizing a new singing voice.

[0447] "Generative AI model" refers to the mathematical model of artificial intelligence used to generate speech.

[0448] "Distribution" refers to the act of providing generated data through a user-accessible link.

[0449] "Multilingual vocal data" refers to vocal data translated into a target language.

[0450] A "prompt sentence" refers to an input command given to a generative AI model to perform appropriate processing.

[0451] The present invention relates to a system for efficiently realizing the multilingualization of anime songs, which allows users to enjoy multilingual versions of songs translated into target languages ​​while preserving the melody and emotion of the original songs.

[0452] System Program Overview

[0453] The system is realized by combining the following main functions:

[0454] 1. The user uploads an audio file.

[0455] 2. The server receives and stores the audio file.

[0456] 3. The server analyzes the audio file and extracts the lyrics data.

[0457] 4. The server translates the lyrics data into the target language, focusing on matching vowels.

[0458] 5. The server generates a singing voice using a generative AI model based on the translated lyrics.

[0459] 6. The server distributes the generated multilingual singing voice data to the user.

[0460] User uploads audio files

[0461] Users access the system's browser interface using their devices, select and upload the anime song audio files, and then click the "Upload" button to send the audio files to the server.

[0462] Receiving and storing audio files on the server

[0463] The server receives the uploaded audio file and stores it in a specified storage directory. The " / uploads" directory is used as an example.

[0464] Server-based audio file analysis and lyric data extraction

[0465] The server sends the saved audio file to speech recognition software (e.g., Google Cloud Speech-to-Text API) to extract lyrics data in text format, and, if necessary, references a lyrics database (e.g., Musixmatch API) to verify the extracted results.

[0466] Server-based vowel-focused lyrics translation

[0467] The server sends the extracted lyrics data to a translation API (e.g., Google Translate API) and translates it into the target language. During this process, an algorithm is applied to ensure that vowels match. An example of a prompt is "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds."

[0468] Server-based singing voice generation that preserves vocal quality and emotion

[0469] Based on the translated lyrics, the server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model). An example of a prompt for the generative AI model is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion."

[0470] Server-generated data distribution

[0471] The server combines the translated lyrics and the generated vocal data into a single file and generates a link that users can access. Users can click the provided link to download and play the multilingual vocal data.

[0472] Specific examples

[0473] For example, if a user uploads an anime song called "Sakuranbo Kiss," the server extracts the lyrics and translates them into English as "Cherry Kiss." The generative AI model then generates "Cherry Kiss" while preserving the original singer's vocal quality and emotion. Users can download the generated audio file and enjoy the seamless melody and emotion of the original song.

[0474] In this way, the system of the present invention can easily generate anime songs in multiple languages ​​and provide them to users, thereby providing a new way to enjoy anime songs internationally.

[0475] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0476] Step 1:

[0477] A user accesses the system's web interface using a terminal and selects an audio file of an anime song. The input is an audio file (e.g., in .mp3 format). The user presses the "Upload" button to send the audio file to the server. The output is an upload request and the audio file.

[0478] Step 2:

[0479] The server receives an audio file received from a user as an HTTP request. The server saves this audio file in the " / uploads" directory. The inputs are the upload request and the audio file. The server saves the file and generates a success message. The output is the saved audio file path and a save success message.

[0480] Step 3:

[0481] The server analyzes the saved audio file using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the path to the saved audio file. The server sends the audio file to the API and obtains the lyrics data in text format. The output is the lyrics data in text format.

[0482] Step 4:

[0483] The server sends the lyric data obtained through analysis to the translation API and translates it into the target language. To emphasize matching of vowels in particular, the prompt "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds" is used. The input consists of text-format lyric data and the translation API prompt. The server calls the translation API and obtains the translated lyric data. The output is the translated lyric data.

[0484] Step 5:

[0485] The server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model) based on the translated lyrics data. An example prompt is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion." The input is the translated lyrics data and the prompt from the generative AI model. The server runs the generative AI model and obtains the generated singing voice data. The output is the generated singing voice data.

[0486] Step 6:

[0487] The server combines the generated vocal data and translated lyrics data into a single file. For example, it creates "Cherry_Kiss_generated.mp3." The inputs are the generated vocal data and translated lyrics data. The server performs file generation and creates a single output file. The output is "Cherry_Kiss_generated.mp3."

[0488] Step 7:

[0489] The server generates a download link to deliver the generated file. The input is the formatted output file (e.g. "Cherry_Kiss_generated.mp3"). The server generates a download link and provides it to the user. The output is the download link.

[0490] Step 8:

[0491] The user clicks the provided download link to download the generated multilingual vocal data. The input is the download link. The user receives the file and can play it on their device. The output is the downloaded "Cherry_Kiss_generated.mp3."

[0492] (Application example 1)

[0493] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0494] Conventionally, localizing anime songs into multiple languages ​​and generating natural-sounding singing voices has been time-consuming and costly. It has also been difficult to seamlessly integrate the translated lyrics with the original melody, and providing the translated lyrics in a format that users can easily play on various devices has been a challenge. The present invention aims to solve these problems and provide a system that efficiently translates anime songs into multiple languages ​​with high quality.

[0495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0496] In this invention, the server includes: means for users to upload audio files of original songs; means for analyzing the uploaded audio files and extracting lyric data; means for translating the extracted lyric data into a target language with emphasis on vowels; means for generating singing voices that retain the vocal quality and emotion of the original singer based on the translated lyrics; means for distributing the generated multilingual singing voice data to users; means including a tool for enabling playback of the generated multilingual singing voice data; and means for providing two-dimensional language-converted lyrics and singing voice data. This enables multilingualization of anime songs, generation of natural singing voices, and playback on a variety of devices.

[0497] The "original song audio file" is the audio data of the original song uploaded by the user.

[0498] "User uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[0499] The "means for analyzing and extracting lyric data" is a function that analyzes the uploaded audio file and extracts the text data of the lyrics using voice recognition technology.

[0500] The "means for translating into the target language with emphasis on vowels" is a function for translating extracted lyrics data into the target language, with particular emphasis on matching vowels.

[0501] "Means for generating singing voices that retain the vocal quality and emotion of the original singer" refers to a function that generates singing voices in a new language based on translated lyrics while retaining the vocal quality and emotion of the original singer.

[0502] The "means for distributing the generated multilingual singing voice data to the user" is a function for transmitting the generated multilingual singing voice data to the user's terminal.

[0503] "Means including tools that enable playback of the generated multilingual singing voice data" refers to a function that provides the software or hardware necessary to play the generated multilingual singing voice data on a device such as a smartphone or computer.

[0504] The "means for providing lyrics and singing voice data that have been converted into two-dimensional language" is a function that provides the user with lyrics translated into a plurality of target languages ​​and the corresponding singing voice data.

[0505] This invention is a system for efficiently realizing the multilingualization of anime songs, specifically translating lyrics and generating singing voices. The system aims to allow users to upload the original song audio file, have the server analyze it, extract the lyrics data, translate it into the target language, and generate and distribute the multilingual singing voice while preserving the vocal quality and emotion of the original singer.

[0506] System Program Overview

[0507] 1. Upload your audio file

[0508] Users select the audio files of anime songs from their own devices and upload them to the server, using devices such as smartphones or PCs, which then send the files to the server via the Internet.

[0509] 2. Extracting Lyric Data

[0510] The server receives the uploaded audio file, analyzes it, and extracts the lyrics data using voice recognition software such as Google Cloud Speech-to-Text, which extracts the lyrics text data from the audio file.

[0511] 3. Vowel-focused lyrics translation

[0512] The server translates the extracted lyrics into the target language using a translation API such as Google Cloud Translation, paying particular attention to matching vowels.

[0513] 4. Singing voice generation that preserves vocal quality and emotion

[0514] Based on the translated lyrics, the server uses voice generation AI such as DeepVoice and WaveNet to generate a singing voice that retains the vocal quality and emotion of the original singer, resulting in a natural-sounding singing voice that corresponds to the translated lyrics.

[0515] 5. Distribution of generated data

[0516] The generated multilingual singing voice data is distributed to the user's terminal, and the user can play back the generated singing voice data on his / her own terminal.

[0517] Specific use cases

[0518] For example, consider the case where a user uploads the original version of an anime song called "Gurenge." The server analyzes this audio file and extracts the lyrics. The server then translates the lyrics into English, checking for vowel matches and making any necessary corrections. Based on the translated lyrics, the server then uses DeepVoice and WaveNet to generate an English version of "Gurenge," creating a singing voice that retains the vocal quality and emotion of the original singer. Finally, this generated singing data is provided to the user's device as a download link.

[0519] Prompt Sentence Examples

[0520] "I would like this audio file translated into English."

[0521] "I want the translated lyrics to be used to generate a singing voice with the original quality."

[0522] This allows users to easily enjoy multilingual anime songs, providing a new way of enjoying anime among international anime fans.

[0523] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0524] Step 1:

[0525] The user selects an audio file of an anime song from their device and uploads it to the server. When uploading, the user uses a specific web page or application to select and send the audio file via the upload button. The server receives and saves this file. The input is the audio file from the user, and the output is the audio file saved on the server.

[0526] Step 2:

[0527] The server analyzes the received audio file and extracts the lyrics data. This analysis is performed using the Google Cloud Speech-to-Text API. The server sends the audio file to the API and obtains the lyrics data in text format. The input is the uploaded audio file, and the output is the lyrics text data.

[0528] Step 3:

[0529] The server translates the extracted lyrics data into the target language. The translation is performed using the Google Cloud Translation API, with particular attention paid to matching vowels. The server sends the lyrics text data to the API and obtains the translated lyrics data. The input is lyrics text data, and the output is translated lyrics data.

[0530] Step 4:

[0531] The server generates a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics data. This process uses voice generation AI models such as DeepVoice and WaveNet. The server inputs the translated lyrics and the original voice features into the voice generation model to generate a realistic singing voice. The input is the translated lyrics data and the original voice feature data, and the output is the generated singing voice data.

[0532] Step 5:

[0533] The server distributes the generated multilingual singing voice data to users. At this time, it provides users with a download link so that they can obtain the data by clicking the link. The input is the generated singing voice data, and the output is the distributed singing voice data.

[0534] Step 6:

[0535] The terminal can play the distributed multilingual vocal data. Users download the data to their smartphones or computers and play the vocals using an audio player application. The input is the distributed multilingual vocal data, and the output is the played vocals.

[0536] This allows users to easily enjoy multilingual versions of anime songs with high-quality singing voices.

[0537] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0538] The present invention is a system that efficiently translates anime songs into multiple languages, and incorporates an emotion engine that can recognize the user's emotions and adjust the emotional expression of the translated lyrics and singing voice accordingly. Below, we will explain the specific program processing of this system in natural language, and provide examples of embodiments.

[0539] Program Overview

[0540] The system includes the following main functions:

[0541] 1. A user uploads an audio file of an anime song.

[0542] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0543] 3. The server translates the lyrics into the target language, focusing on vowels.

[0544] 4. The server recognizes the user's emotions and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on those emotions.

[0545] 5. Based on the translated lyrics, the server generates a singing voice that retains the vocal quality and emotion of the original singer.

[0546] 6. The server distributes the generated multilingual singing voice data to the user.

[0547] Program processing (natural language explanation)

[0548] User-uploaded anime songs

[0549] The user selects an audio file of the original anime song using the terminal and uploads it to the system's server. The terminal transmits the selected audio file to the server, which receives and stores the audio file.

[0550] Server-based analysis and lyric data extraction

[0551] The server uses audio analysis software to analyze the uploaded audio files and extract the lyrics data, which is then converted into text format.

[0552] Vowel-focused lyrics translation

[0553] The server then uses text generation AI to translate the extracted lyrics into the target language. The translation process places particular emphasis on matching vowels to ensure a seamless melody line. The translated lyrics may then be further adjusted based on the user's emotions.

[0554] User emotion recognition and expression adjustment

[0555] The server uses an emotion engine to recognize the user's emotions. This emotion recognition can be performed, for example, by the user inputting facial expressions and voice using a camera or microphone. The emotion engine identifies the user's emotion (e.g., joy, sadness, surprise, etc.) and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on that emotion. For example, if the user is recognized as "sad," the translated lyrics are adjusted to add a nuance of sadness.

[0556] Singing voice generation that preserves vocal quality and emotion

[0557] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. The voice generation AI receives the translated lyrics and the original audio data as input and reproduces a realistic and emotional singing voice.

[0558] Distribution of generated data

[0559] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. This file is temporarily saved and a download link is provided to the user's device. The user clicks the link on their device to download the generated data.

[0560] Specific examples

[0561] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server uses a voice generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer, and delivers it to the user. The user can then play "Cherry Kiss" on their device and enjoy the multilingual anime song that reflects the positive emotion.

[0562] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0563] The processing flow will be explained below.

[0564] Step 1:

[0565] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server receives and stores the uploaded audio file.

[0566] Step 2:

[0567] The server uses audio analysis software to analyze the uploaded audio file, extracting lyrics data as a result and converting it into text format.

[0568] Step 3:

[0569] The server uses text generation AI to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching and adjusting the melody line of the original song so that it fits seamlessly with the original.

[0570] Step 4:

[0571] The user's device uses a camera and microphone to transmit the user's facial expressions and voice to an emotion recognition engine, which then identifies the user's emotions (e.g., joy, sadness, surprise, etc.) in real time.

[0572] Step 5:

[0573] The server adjusts the expression of the translated lyrics based on the user's recognized emotional data. For example, if the user feels sad, the server adds a nuance of sadness to the translated lyrics.

[0574] Step 6:

[0575] The server uses a voice generation AI to generate a new singing voice based on the emotion-adjusted translated lyrics and the original voice data. The generated singing voice retains the vocal quality and emotion of the original singer while also incorporating expressions that correspond to the user's emotions.

[0576] Step 7:

[0577] The server formats the newly generated vocal data and the adjusted translated lyrics, and combines them into a single file. This formatted data is temporarily stored on the server.

[0578] Step 8:

[0579] The server provides a download link for the generated multilingual singing voice data to the user's device, and the user clicks the link using the device to download the generated data.

[0580] Step 9:

[0581] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs. The singing voices they hear are natural and reflect the user's emotions.

[0582] Example 2

[0583] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] Conventional multilingual speech generation systems have difficulty accurately preserving the emotion of the original song, making it difficult to generate translated lyrics and speech that reflect the user's emotions. Furthermore, there are insufficient means for increasing the degree of agreement between the vowels in the translated lyrics and the melody line, resulting in the generated multilingual singing voices often sounding unnatural.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0586] In this invention, the server includes means for users to upload audio files, means for analyzing the uploaded audio files and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics and the emotional expression of the generated singing voice, means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics, and means for delivering the generated multilingual singing voice data to users. This makes it possible to generate natural multilingual singing voices that reflect the user's emotions and reduce the sense of incongruity with the melody line of the original song.

[0587] A "user" is someone who uses this system to translate anime songs into multiple languages.

[0588] An "audio file" is a file in which the original music of an anime song is saved in digital format.

[0589] "Uploading means" refers to the method or tool that a user uses to send an audio file to the server.

[0590] "Analysis means" refers to the techniques and tools used to extract lyric data from audio files.

[0591] "Lyric data" is text-format lyric information extracted from an audio file.

[0592] "Target language" is the language into which the original lyrics are translated.

[0593] "Means of translation" refers to the methods and tools used to convert the original lyrics data into the target language.

[0594] "Placing emphasis on vowels" means increasing the degree of consistency of the vowels in the translated lyrics and maintaining compatibility with the melody line.

[0595] "Means for recognizing user emotions" refers to a method or device for analyzing emotions from the user's facial expressions, voice, etc.

[0596] The "means for adjusting emotional expression" refers to a method or tool for adjusting the translated lyrics or the generated singing voice based on the recognized emotions of the user.

[0597] "Preserving vocal quality and emotion" means reproducing the original singer's unique vocal quality and emotion.

[0598] "Generative means" refers to methods and tools for generating new singing voices based on translated lyrics.

[0599] The "distribution means" refers to a method or tool for providing the generated multilingual singing voice data to users.

[0600] "Evaluation methods" are methods or tools that confirm how well the vowels in the translated lyrics match those in the original song.

[0601] "Correction methods" are methods or tools for adjusting and correcting the mismatched vowels.

[0602] "Formatting means" refers to the method or tool used to combine the generated vocal data and translated lyrics into a single file.

[0603] The present invention is a system that allows users to upload audio files of anime songs, analyzes the audio files to extract lyric data, translates the data into target languages, and finally generates and distributes expressive singing voices in multiple languages. Specific embodiments of this system are described below.

[0604] User-uploaded anime songs

[0605] The user selects an anime song audio file using a device (e.g., a smartphone or PC) and clicks the upload button on the operation screen to send the audio file to the system's server. The device then sends the selected audio file to the server, which receives and stores the audio file.

[0606] Server-based analysis and lyric data extraction

[0607] The server analyzes the uploaded audio files using audio analysis software (e.g., Adobe Audition or the open-source Audacity) and extracts the lyrics data. The resulting lyrics data is converted into text format and used for further processing.

[0608] Vowel-focused lyrics translation

[0609] The server uses text generation AI (e.g., GPT-4) to translate the extracted lyrics into the target language. During translation, particular emphasis is placed on matching vowels, and adjustments are made to ensure a natural melody line. The translated lyrics are then used for further processing.

[0610] User emotion recognition and expression adjustment

[0611] Users can use a camera or microphone to input their emotions. The server uses an Emotion API (e.g., Google Cloud Vision API or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice and recognize their emotions. Based on the recognized emotions, the server adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice.

[0612] Singing voice generation that preserves vocal quality and emotion

[0613] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a speech generation AI (e.g., Amazon Polly or Google Cloud Text-to-Speech) to generate a singing voice that retains the vocal quality and emotion of the original singer. The speech generation AI receives the translated lyrics and audio data as input and reproduces a realistic, emotional singing voice.

[0614] Distribution of generated data

[0615] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. The server temporarily stores this file and generates a download link for the user's device. The user can click the link on their device to download the generated data.

[0616] Specific examples

[0617] For example, consider a case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server then uses a speech generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[0618] Prompt Sentence Examples

[0619] To translate the Japanese lyrics of the anime song "Sakuranbo Kiss" into English, the following prompt sentence is input to the generative AI model:

[0620] Original lyrics: "Cherry Kiss Lyrics"

[0621] Target language: English

[0622] Emotion: Happy

[0623] Translated lyrics generation

[0624] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0625] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0626] Step 1:

[0627] The user selects and uploads an audio file of an anime song. The user uses the device to select the audio file and clicks the upload button on the operation screen. This operation sends the audio file from the device to the server.

[0628] Input: User selected audio file

[0629] Output: Audio file sent to the server

[0630] Step 2:

[0631] The server receives and stores the audio file. The server receives the audio file uploaded from the device and stores it in temporary storage.

[0632] Input: Uploaded audio file

[0633] Output: Saved audio file

[0634] Step 3:

[0635] The server analyzes the audio file and extracts the lyrics data. The server launches audio analysis software (e.g., Adobe Audition) and analyzes the saved audio file. As a result of the audio analysis, lyrics data is extracted.

[0636] Input: Saved audio file

[0637] Output: Extracted lyrics data (text format)

[0638] Step 4:

[0639] The server translates the lyric data into the target language. The server uses a text generation AI (e.g., GPT-4) to translate the extracted lyric data into the target language. The translated lyrics are generated by inputting the prompt sentence into the generation AI model.

[0640] Input: Extracted lyrics data

[0641] Output: Translated lyrics data

[0642] Step 5:

[0643] The system recognizes the user's emotions and adjusts the lyrics and vocals accordingly. The user sends emotional information to the server using a camera or microphone. The server then analyzes the user's emotions using the Emotion API and adjusts the emotional expression of the translated lyrics data and the generated vocals.

[0644] Input: User emotion information, translated lyrics data

[0645] Output: Adjusted lyrics data, singing data reflecting emotional expressions

[0646] Step 6:

[0647] Generation of singing voices that retains vocal quality and emotion. The server launches a voice generation AI (e.g., Amazon Polly) and inputs the adjusted lyrics data to generate singing voice data. This singing voice data retains the vocal quality and emotion of the original singer.

[0648] Input: Adjusted lyrics data

[0649] Output: Generated singing voice data

[0650] Step 7:

[0651] The generated data is distributed to the user. The server combines the generated vocal data and translated lyrics data into a single file and temporarily stores it. A download link is generated for the user's device, and the user clicks the link to download the file.

[0652] Input: Generated vocal data, translated lyrics data

[0653] Output: Download link, user downloaded file

[0654] (Application example 2)

[0655] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] With conventional music streaming services, users often experience a lack of emotional expression when listening to their favorite anime songs in multiple languages. This can lead to users feeling uneasy about the translated lyrics and vocals, resulting in an unsatisfying music experience. Furthermore, typical translation systems struggle to retain vowel matches and emotional nuances, preventing them from providing a more personalized music experience.

[0657] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload an audio file of an original song, means for analyzing the uploaded audio file and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics according to the emotions, means for generating a singing voice that retains the vocal quality and emotion of the singer of the original song based on the translated lyrics, and means for delivering the generated multilingual singing voice data to the user. This makes it possible to enjoy multilingual anime songs that reflect emotional expressions in real time.

[0658] "Original song audio file" refers to the audio data of the original anime song that the user uploads to the system.

[0659] "Uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[0660] "Means for analyzing and extracting lyric data" means software technology for extracting lyrics in text form from uploaded audio files.

[0661] "Target language" refers to the language into which the translation is to be made.

[0662] "Methods of translating with emphasis on vowels" refers to a translation technique that pays particular attention to the degree of agreement of vowels when translating, in order to maintain consistency with the melody of the original song.

[0663] The "means for recognizing user emotions" is a function that uses input devices such as a camera or microphone to read emotions from the user's facial expressions and tone of voice.

[0664] "Means for adjusting the expression of translated lyrics according to emotions" refers to a technology that changes the content and nuance of translated lyrics based on the recognized emotions of the user.

[0665] "Means for generating singing voices that retain vocal quality and emotion" refers to technology that generates singing voices that reproduce the vocal quality and emotion of the original singer based on translated lyrics and their emotional expressions.

[0666] The "distribution means" refers to a technique for providing the generated multilingual singing voice data to users.

[0667] The present invention relates to a system for translating anime songs into multiple languages ​​and adjusting the emotional expression of the translated lyrics and singing voice in response to the user's emotions. This system is specifically implemented using the following series of means.

[0668] Hardware and Software

[0669] To realize this system, the following hardware and software are used.

[0670] Hardware used

[0671] 1. Device: Smartphone (e.g. Apple iPhone, Samsung Galaxy)

[0672] 2. Server: High-performance server (e.g., cloud-based server infrastructure)

[0673] Software used

[0674] 1. Frontend: User interface using React Native

[0675] 2. Backend: Server-side program using Python (Flask) and Node.js

[0676] 3. Data Processing API:

[0677] Google Cloud Speech-to-Text: Extracting lyrics from audio

[0678] Amazon Translate: Translation into the target language

[0679] OpenAI GPT-4: Text generation and emotional expression adjustment

[0680] AWS Polly: Emotionally Reflective Singing Voice Generation

[0681] System operation

[0682] 1. User Uploads

[0683] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the application. The device then sends the selected audio file to the server.

[0684] 2. Server-side analysis

[0685] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the lyrics data in text format.

[0686] 3. Lyric translation

[0687] The extracted lyrics data is translated into the target language using Amazon Translate, with particular attention paid to vowel matching.

[0688] 4. Emotion recognition

[0689] The server uses data from the camera and microphone to recognize the user's emotions using emotion recognition libraries such as OpenCV and Librosa, which determine emotions from the user's facial expressions and tone of voice.

[0690] 5. Emotion-Based Regulation

[0691] Using OpenAI GPT-4, the translation adjusts the expression of the lyrics depending on the recognized emotion. For example, if the user is recognized as "sad," the translation will add a nuance of sadness.

[0692] 6. Singing Voice Generation

[0693] Based on the translated lyrics and their emotional expressions, AWS Polly is used to generate a singing voice that retains the vocal quality and emotion of the original singer.

[0694] 7. Data Distribution

[0695] The generated multilingual singing voice data is provided to users in real time, and can be played back at any time via their smartphones.

[0696] Specific examples

[0697] For example, consider a user uploading an anime song called "Cherry Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and uses Amazon Translate to add a more positive nuance to the translated lyrics, "Cherry Kiss." AWS Polly then generates an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[0698] Prompt Sentence Examples

[0699] "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (eg, joy, sadness)."

[0700] Using these prompts, the system generates emotion-based translated lyrics in the target language, providing the user with a highly personalized musical experience.

[0701] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0702] Step 1:

[0703] Uploading an audio file

[0704] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the app. The audio file selected by the user is sent from the device to the server. The input is the audio file selected by the user, and the output is the audio file saved on the server.

[0705] Step 2:

[0706] Analyzing audio files

[0707] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the text lyrics. This analysis converts the audio signal into text data. The input is the uploaded audio file, and the output is the extracted text lyrics.

[0708] Step 3:

[0709] Lyrics translation

[0710] The server uses Amazon Translate to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching to generate natural lyrics. The input is text-format lyrics data, and the output is the lyrics translated into the target language.

[0711] Step 4:

[0712] emotion recognition

[0713] The user inputs facial expressions and voice using a camera or microphone. The server uses emotion recognition libraries such as OpenCV and Librosa to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the identified emotion data.

[0714] Step 5:

[0715] Adjustment of translated lyrics

[0716] The server uses OpenAI's GPT-4 to adjust the expression of the translated lyrics based on emotion. An example prompt is "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (e.g., joy, sadness)." The input is translated lyrics data and emotion data, and the output is the adjusted translated lyrics data.

[0717] Step 6:

[0718] Singing voice generation

[0719] The server uses AWS Polly to generate a singing voice that retains the vocal quality and emotion of the original singer based on the adjusted translated lyrics data. The input is the adjusted translated lyrics data, and the output is the generated singing voice data.

[0720] Step 7:

[0721] Data distribution

[0722] The server formats the generated multilingual vocal data and provides it to the user. The final data is compiled into a single file, which the user can play in real time via their smartphone. The input is the generated vocal data and the adjusted translated lyrics data, and the output is a file provided to the user as a download link.

[0723] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0724] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0725] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0726] [Third embodiment]

[0727] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0728] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0729] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0730] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0731] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0732] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0733] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0734] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0735] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0736] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0737] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0738] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0739] This invention relates to a system for efficiently realizing the multilingualization of anime songs, specifically to a program for translating lyrics and generating singing voices. Here, the processing of the program of this system is explained in natural language, and an embodiment is shown with concrete examples.

[0740] Program Overview

[0741] The system is realized by combining the following main functions:

[0742] 1. A user uploads an audio file of an anime song.

[0743] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0744] 3. The server translates the lyrics into the target language, focusing on vowels.

[0745] 4. The server generates a singing voice that preserves the vocal quality and emotion based on the translated lyrics.

[0746] 5. The server distributes the generated multilingual singing voice data to the user.

[0747] Program processing (natural language explanation)

[0748] User-uploaded anime songs

[0749] The user selects the original audio file of the anime song using the terminal and uploads it to the system's server. The terminal sends the selected audio file to the server, and the server stores the received audio file.

[0750] Server-based analysis and lyric data extraction

[0751] The server analyzes the received audio file and extracts the lyrics data, using voice recognition software and a lyrics database to obtain text-format lyrics information from the audio file.

[0752] Vowel-focused lyrics translation

[0753] The server translates the extracted lyrics into the target language, placing emphasis on vowels. For example, it uses voice generation AI to translate the lyrics while paying particular attention to matching vowels to ensure that they match the melody line of the original song. It also has a function to evaluate and correct the degree of vowel matching in the translation results.

[0754] Singing voice generation that preserves vocal quality and emotion

[0755] Based on the translated lyrics, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. Specifically, the AI ​​receives the translated lyrics and the original voice as input and reproduces a realistic singing voice. The AI ​​also takes into account vocal expressions and intonation, aiming for a natural result.

[0756] Distribution of generated data

[0757] The server formats the generated multilingual vocal data and translated lyrics, combines them into a single file, and provides a download link to distribute this file to the user's device. The user clicks the link on their device to download the generated data.

[0758] Specific examples

[0759] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." The user uploads the audio file to the server, which extracts the lyrics of "Sakuranbo Kiss" and translates them into English as "Cherry Kiss," emphasizing vowels. The server then uses a voice generation AI to generate "Cherry Kiss," preserving the vocal quality and emotion of the original singer, and delivers it to the user. The user can play "Cherry Kiss" on their device and enjoy the melody line of the original song without any sense of incongruity.

[0760] In this way, the system of the present invention provides a new way to enjoy anime songs internationally by converting anime songs uploaded by users into multiple languages ​​and playing them back in natural singing voices.

[0761] The processing flow will be explained below.

[0762] Step 1:

[0763] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server saves the received audio file.

[0764] Step 2:

[0765] The server uses audio analysis software to analyze the uploaded audio files, extracting lyrics data and converting it into text format.

[0766] Step 3:

[0767] The server then calls a text generation AI to translate the extracted lyrics into the target language, placing emphasis on matching vowels to ensure that the melody line of the original song is not out of place.

[0768] Step 4:

[0769] The server evaluates the degree of vowel matching of the generated translated lyrics and corrects any mismatches based on the evaluation results, resulting in more natural-sounding translated lyrics.

[0770] Step 5:

[0771] The server then passes the translated lyrics and the original audio file as input to the AI ​​voice generator, which then generates a new singing voice for the translated lyrics while preserving the vocal quality and emotion of the original singer.

[0772] Step 6:

[0773] The server formats the new vocal data and the translated lyrics, and combines them into a single file. The formatted data is temporarily saved.

[0774] Step 7:

[0775] The server creates a download link for the generated multilingual singing voice data and provides it to the user's device. The user clicks the provided link on their device to download the generated data.

[0776] Step 8:

[0777] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs, which are played with natural singing voices that blend seamlessly with the original melody lines.

[0778] Example 1

[0779] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0780] In the multilingualization of anime songs, it is difficult to generate natural-sounding singing voices in different languages ​​while preserving the melody and emotion of the original song. Therefore, there is a need for a system that can generate multilingual singing voices efficiently and with high quality.

[0781] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0782] In this invention, the server includes a means for users to upload audio files, a means for receiving and saving the uploaded audio files, a means for analyzing the audio files and extracting text-format lyrics data, a means for translating the extracted lyrics data into a target language with particular emphasis on matching vowels, a means for using a generative AI model to generate singing voices that retain the vocal quality and emotion based on the translated lyrics, and a means for delivering the generated multilingual singing voice data to users. This allows for efficient and high-quality multilingual anime song generation, allowing users to enjoy songs that retain the emotion of the original.

[0783] "User" refers to a person or organization that accesses the System, uploads audio files, and receives multilingual vocal data.

[0784] "Audio file" refers to a file format (e.g., MP3, WAV, AAC, etc.) that contains the audio data of the original anime song.

[0785] "Upload" refers to the act of a user sending an audio file from their own device to a server.

[0786] "Receiving" refers to the process in which the server obtains the audio file sent by the user and saves it in a specified location.

[0787] "Saving" refers to storing the received audio file in a specified directory on the server.

[0788] "Analysis" refers to the process of running a computer program to extract the necessary information from the audio file.

[0789] "Lyrics data" refers to text-format lyrics information extracted from an audio file.

[0790] "Target language" refers to the language into which the original lyrics data is translated.

[0791] "Vowel agreement" refers to the situation where the vowels in the original lyrics and the translated lyrics are similar or identical.

[0792] "Translation" refers to the process of converting the original lyrics data into a target language.

[0793] "Voice quality" refers to the sound quality and characteristics unique to each individual voice.

[0794] "Emotion" refers to the feelings and moods that a singer expresses through lyrics and melody.

[0795] "Voice generation" refers to the process of recreating the vocal quality and emotion based on translated lyrics and synthesizing a new singing voice.

[0796] "Generative AI model" refers to the mathematical model of artificial intelligence used to generate speech.

[0797] "Distribution" refers to the act of providing generated data through a user-accessible link.

[0798] "Multilingual vocal data" refers to vocal data translated into a target language.

[0799] A "prompt sentence" refers to an input command given to a generative AI model to perform appropriate processing.

[0800] The present invention relates to a system for efficiently realizing the multilingualization of anime songs, which allows users to enjoy multilingual versions of songs translated into target languages ​​while preserving the melody and emotion of the original songs.

[0801] System Program Overview

[0802] The system is realized by combining the following main functions:

[0803] 1. The user uploads an audio file.

[0804] 2. The server receives and stores the audio file.

[0805] 3. The server analyzes the audio file and extracts the lyrics data.

[0806] 4. The server translates the lyrics data into the target language, focusing on matching vowels.

[0807] 5. The server generates a singing voice using a generative AI model based on the translated lyrics.

[0808] 6. The server distributes the generated multilingual singing voice data to the user.

[0809] User uploads audio files

[0810] Users access the system's browser interface using their devices, select and upload the anime song audio files, and then click the "Upload" button to send the audio files to the server.

[0811] Receiving and storing audio files on the server

[0812] The server receives the uploaded audio file and stores it in a specified storage directory. The " / uploads" directory is used as an example.

[0813] Server-based audio file analysis and lyric data extraction

[0814] The server sends the saved audio file to speech recognition software (e.g., Google Cloud Speech-to-Text API) to extract lyrics data in text format, and, if necessary, references a lyrics database (e.g., Musixmatch API) to verify the extracted results.

[0815] Server-based vowel-focused lyrics translation

[0816] The server sends the extracted lyrics data to a translation API (e.g., Google Translate API) and translates it into the target language. During this process, an algorithm is applied to ensure that vowels match. An example of a prompt is "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds."

[0817] Server-based singing voice generation that preserves vocal quality and emotion

[0818] Based on the translated lyrics, the server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model). An example of a prompt for the generative AI model is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion."

[0819] Server-generated data distribution

[0820] The server combines the translated lyrics and the generated vocal data into a single file and generates a link that users can access. Users can click the provided link to download and play the multilingual vocal data.

[0821] Specific examples

[0822] For example, if a user uploads an anime song called "Sakuranbo Kiss," the server extracts the lyrics and translates them into English as "Cherry Kiss." The generative AI model then generates "Cherry Kiss" while preserving the original singer's vocal quality and emotion. Users can download the generated audio file and enjoy the seamless melody and emotion of the original song.

[0823] In this way, the system of the present invention can easily generate anime songs in multiple languages ​​and provide them to users, thereby providing a new way to enjoy anime songs internationally.

[0824] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0825] Step 1:

[0826] A user accesses the system's web interface using a terminal and selects an audio file of an anime song. The input is an audio file (e.g., in .mp3 format). The user presses the "Upload" button to send the audio file to the server. The output is an upload request and the audio file.

[0827] Step 2:

[0828] The server receives an audio file received from a user as an HTTP request. The server saves this audio file in the " / uploads" directory. The inputs are the upload request and the audio file. The server saves the file and generates a success message. The output is the saved audio file path and a save success message.

[0829] Step 3:

[0830] The server analyzes the saved audio file using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the path to the saved audio file. The server sends the audio file to the API and obtains the lyrics data in text format. The output is the lyrics data in text format.

[0831] Step 4:

[0832] The server sends the lyric data obtained through analysis to the translation API and translates it into the target language. To emphasize matching of vowels in particular, the prompt "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds" is used. The input consists of text-format lyric data and the translation API prompt. The server calls the translation API and obtains the translated lyric data. The output is the translated lyric data.

[0833] Step 5:

[0834] The server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model) based on the translated lyrics data. An example prompt is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion." The input is the translated lyrics data and the prompt from the generative AI model. The server runs the generative AI model and obtains the generated singing voice data. The output is the generated singing voice data.

[0835] Step 6:

[0836] The server combines the generated vocal data and translated lyrics data into a single file. For example, it creates "Cherry_Kiss_generated.mp3." The inputs are the generated vocal data and translated lyrics data. The server performs file generation and creates a single output file. The output is "Cherry_Kiss_generated.mp3."

[0837] Step 7:

[0838] The server generates a download link to deliver the generated file. The input is the formatted output file (e.g. "Cherry_Kiss_generated.mp3"). The server generates a download link and provides it to the user. The output is the download link.

[0839] Step 8:

[0840] The user clicks the provided download link to download the generated multilingual vocal data. The input is the download link. The user receives the file and can play it on their device. The output is the downloaded "Cherry_Kiss_generated.mp3."

[0841] (Application example 1)

[0842] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0843] Conventionally, localizing anime songs into multiple languages ​​and generating natural-sounding singing voices has been time-consuming and costly. It has also been difficult to seamlessly integrate the translated lyrics with the original melody, and providing the translated lyrics in a format that users can easily play on various devices has been a challenge. The present invention aims to solve these problems and provide a system that efficiently translates anime songs into multiple languages ​​with high quality.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0845] In this invention, the server includes: means for users to upload audio files of original songs; means for analyzing the uploaded audio files and extracting lyric data; means for translating the extracted lyric data into a target language with emphasis on vowels; means for generating singing voices that retain the vocal quality and emotion of the original singer based on the translated lyrics; means for distributing the generated multilingual singing voice data to users; means including a tool for enabling playback of the generated multilingual singing voice data; and means for providing two-dimensional language-converted lyrics and singing voice data. This enables multilingualization of anime songs, generation of natural singing voices, and playback on a variety of devices.

[0846] The "original song audio file" is the audio data of the original song uploaded by the user.

[0847] "User uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[0848] The "means for analyzing and extracting lyric data" is a function that analyzes the uploaded audio file and extracts the text data of the lyrics using voice recognition technology.

[0849] The "means for translating into the target language with emphasis on vowels" is a function for translating extracted lyrics data into the target language, with particular emphasis on matching vowels.

[0850] "Means for generating singing voices that retain the vocal quality and emotion of the original singer" refers to a function that generates singing voices in a new language based on translated lyrics while retaining the vocal quality and emotion of the original singer.

[0851] The "means for distributing the generated multilingual singing voice data to the user" is a function for transmitting the generated multilingual singing voice data to the user's terminal.

[0852] "Means including tools that enable playback of the generated multilingual singing voice data" refers to a function that provides the software or hardware necessary to play the generated multilingual singing voice data on a device such as a smartphone or computer.

[0853] The "means for providing lyrics and singing voice data that have been converted into two-dimensional language" is a function that provides the user with lyrics translated into a plurality of target languages ​​and the corresponding singing voice data.

[0854] This invention is a system for efficiently realizing the multilingualization of anime songs, specifically translating lyrics and generating singing voices. The system aims to allow users to upload the original song audio file, have the server analyze it, extract the lyrics data, translate it into the target language, and generate and distribute the multilingual singing voice while preserving the vocal quality and emotion of the original singer.

[0855] System Program Overview

[0856] 1. Upload your audio file

[0857] Users select the audio files of anime songs from their own devices and upload them to the server, using devices such as smartphones or PCs, which then send the files to the server via the Internet.

[0858] 2. Extracting Lyric Data

[0859] The server receives the uploaded audio file, analyzes it, and extracts the lyrics data using voice recognition software such as Google Cloud Speech-to-Text, which extracts the lyrics text data from the audio file.

[0860] 3. Vowel-focused lyrics translation

[0861] The server translates the extracted lyrics into the target language using a translation API such as Google Cloud Translation, paying particular attention to matching vowels.

[0862] 4. Singing voice generation that preserves vocal quality and emotion

[0863] Based on the translated lyrics, the server uses voice generation AI such as DeepVoice and WaveNet to generate a singing voice that retains the vocal quality and emotion of the original singer, resulting in a natural-sounding singing voice that corresponds to the translated lyrics.

[0864] 5. Distribution of generated data

[0865] The generated multilingual singing voice data is distributed to the user's terminal, and the user can play back the generated singing voice data on his / her own terminal.

[0866] Specific use cases

[0867] For example, consider the case where a user uploads the original version of an anime song called "Gurenge." The server analyzes this audio file and extracts the lyrics. The server then translates the lyrics into English, checking for vowel matches and making any necessary corrections. Based on the translated lyrics, the server then uses DeepVoice and WaveNet to generate an English version of "Gurenge," creating a singing voice that retains the vocal quality and emotion of the original singer. Finally, this generated singing data is provided to the user's device as a download link.

[0868] Prompt Sentence Examples

[0869] "I would like this audio file translated into English."

[0870] "I want the translated lyrics to be used to generate a singing voice with the original quality."

[0871] This allows users to easily enjoy multilingual anime songs, providing a new way of enjoying anime among international anime fans.

[0872] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0873] Step 1:

[0874] The user selects an audio file of an anime song from their device and uploads it to the server. When uploading, the user uses a specific web page or application to select and send the audio file via the upload button. The server receives and saves this file. The input is the audio file from the user, and the output is the audio file saved on the server.

[0875] Step 2:

[0876] The server analyzes the received audio file and extracts the lyrics data. This analysis is performed using the Google Cloud Speech-to-Text API. The server sends the audio file to the API and obtains the lyrics data in text format. The input is the uploaded audio file, and the output is the lyrics text data.

[0877] Step 3:

[0878] The server translates the extracted lyrics data into the target language. The translation is performed using the Google Cloud Translation API, with particular attention paid to matching vowels. The server sends the lyrics text data to the API and obtains the translated lyrics data. The input is lyrics text data, and the output is translated lyrics data.

[0879] Step 4:

[0880] The server generates a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics data. This process uses voice generation AI models such as DeepVoice and WaveNet. The server inputs the translated lyrics and the original voice features into the voice generation model to generate a realistic singing voice. The input is the translated lyrics data and the original voice feature data, and the output is the generated singing voice data.

[0881] Step 5:

[0882] The server distributes the generated multilingual singing voice data to users. At this time, it provides users with a download link so that they can obtain the data by clicking the link. The input is the generated singing voice data, and the output is the distributed singing voice data.

[0883] Step 6:

[0884] The terminal can play the distributed multilingual vocal data. Users download the data to their smartphones or computers and play the vocals using an audio player application. The input is the distributed multilingual vocal data, and the output is the played vocals.

[0885] This allows users to easily enjoy multilingual versions of anime songs with high-quality singing voices.

[0886] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0887] The present invention is a system that efficiently translates anime songs into multiple languages, and incorporates an emotion engine that can recognize the user's emotions and adjust the emotional expression of the translated lyrics and singing voice accordingly. Below, we will explain the specific program processing of this system in natural language, and provide examples of embodiments.

[0888] Program Overview

[0889] The system includes the following main functions:

[0890] 1. A user uploads an audio file of an anime song.

[0891] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[0892] 3. The server translates the lyrics into the target language, focusing on vowels.

[0893] 4. The server recognizes the user's emotions and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on those emotions.

[0894] 5. Based on the translated lyrics, the server generates a singing voice that retains the vocal quality and emotion of the original singer.

[0895] 6. The server distributes the generated multilingual singing voice data to the user.

[0896] Program processing (natural language explanation)

[0897] User-uploaded anime songs

[0898] The user selects an audio file of the original anime song using the terminal and uploads it to the system's server. The terminal transmits the selected audio file to the server, which receives and stores the audio file.

[0899] Server-based analysis and lyric data extraction

[0900] The server uses audio analysis software to analyze the uploaded audio files and extract the lyrics data, which is then converted into text format.

[0901] Vowel-focused lyrics translation

[0902] The server then uses text generation AI to translate the extracted lyrics into the target language. The translation process places particular emphasis on matching vowels to ensure a seamless melody line. The translated lyrics may then be further adjusted based on the user's emotions.

[0903] User emotion recognition and expression adjustment

[0904] The server uses an emotion engine to recognize the user's emotions. This emotion recognition can be performed, for example, by the user inputting facial expressions and voice using a camera or microphone. The emotion engine identifies the user's emotion (e.g., joy, sadness, surprise, etc.) and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on that emotion. For example, if the user is recognized as "sad," the translated lyrics are adjusted to add a nuance of sadness.

[0905] Singing voice generation that preserves vocal quality and emotion

[0906] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. The voice generation AI receives the translated lyrics and the original audio data as input and reproduces a realistic and emotional singing voice.

[0907] Distribution of generated data

[0908] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. This file is temporarily saved and a download link is provided to the user's device. The user clicks the link on their device to download the generated data.

[0909] Specific examples

[0910] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server uses a voice generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer, and delivers it to the user. The user can then play "Cherry Kiss" on their device and enjoy the multilingual anime song that reflects the positive emotion.

[0911] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0912] The processing flow will be explained below.

[0913] Step 1:

[0914] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server receives and stores the uploaded audio file.

[0915] Step 2:

[0916] The server uses audio analysis software to analyze the uploaded audio file, extracting lyrics data as a result and converting it into text format.

[0917] Step 3:

[0918] The server uses text generation AI to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching and adjusting the melody line of the original song so that it fits seamlessly with the original.

[0919] Step 4:

[0920] The user's device uses a camera and microphone to transmit the user's facial expressions and voice to an emotion recognition engine, which then identifies the user's emotions (e.g., joy, sadness, surprise, etc.) in real time.

[0921] Step 5:

[0922] The server adjusts the expression of the translated lyrics based on the user's recognized emotional data. For example, if the user feels sad, the server adds a nuance of sadness to the translated lyrics.

[0923] Step 6:

[0924] The server uses a voice generation AI to generate a new singing voice based on the emotion-adjusted translated lyrics and the original voice data. The generated singing voice retains the vocal quality and emotion of the original singer while also incorporating expressions that correspond to the user's emotions.

[0925] Step 7:

[0926] The server formats the newly generated vocal data and the adjusted translated lyrics, and combines them into a single file. This formatted data is temporarily stored on the server.

[0927] Step 8:

[0928] The server provides a download link for the generated multilingual singing voice data to the user's device, and the user clicks the link using the device to download the generated data.

[0929] Step 9:

[0930] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs. The singing voices they hear are natural and reflect the user's emotions.

[0931] Example 2

[0932] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0933] Conventional multilingual speech generation systems have difficulty accurately preserving the emotion of the original song, making it difficult to generate translated lyrics and speech that reflect the user's emotions. Furthermore, there are insufficient means for increasing the degree of agreement between the vowels in the translated lyrics and the melody line, resulting in the generated multilingual singing voices often sounding unnatural.

[0934] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0935] In this invention, the server includes means for users to upload audio files, means for analyzing the uploaded audio files and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics and the emotional expression of the generated singing voice, means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics, and means for delivering the generated multilingual singing voice data to users. This makes it possible to generate natural multilingual singing voices that reflect the user's emotions and reduce the sense of incongruity with the melody line of the original song.

[0936] A "user" is someone who uses this system to translate anime songs into multiple languages.

[0937] An "audio file" is a file in which the original music of an anime song is saved in digital format.

[0938] "Uploading means" refers to the method or tool that a user uses to send an audio file to the server.

[0939] "Analysis means" refers to the techniques and tools used to extract lyric data from audio files.

[0940] "Lyric data" is text-format lyric information extracted from an audio file.

[0941] "Target language" is the language into which the original lyrics are translated.

[0942] "Means of translation" refers to the methods and tools used to convert the original lyrics data into the target language.

[0943] "Placing emphasis on vowels" means increasing the degree of consistency of the vowels in the translated lyrics and maintaining compatibility with the melody line.

[0944] "Means for recognizing user emotions" refers to a method or device for analyzing emotions from the user's facial expressions, voice, etc.

[0945] The "means for adjusting emotional expression" refers to a method or tool for adjusting the translated lyrics or the generated singing voice based on the recognized emotions of the user.

[0946] "Preserving vocal quality and emotion" means reproducing the original singer's unique vocal quality and emotion.

[0947] "Generative means" refers to methods and tools for generating new singing voices based on translated lyrics.

[0948] The "distribution means" refers to a method or tool for providing the generated multilingual singing voice data to users.

[0949] "Evaluation methods" are methods or tools that confirm how well the vowels in the translated lyrics match those in the original song.

[0950] "Correction methods" are methods or tools for adjusting and correcting the mismatched vowels.

[0951] "Formatting means" refers to the method or tool used to combine the generated vocal data and translated lyrics into a single file.

[0952] The present invention is a system that allows users to upload audio files of anime songs, analyzes the audio files to extract lyric data, translates the data into target languages, and finally generates and distributes expressive singing voices in multiple languages. Specific embodiments of this system are described below.

[0953] User-uploaded anime songs

[0954] The user selects an anime song audio file using a device (e.g., a smartphone or PC) and clicks the upload button on the operation screen to send the audio file to the system's server. The device then sends the selected audio file to the server, which receives and stores the audio file.

[0955] Server-based analysis and lyric data extraction

[0956] The server analyzes the uploaded audio files using audio analysis software (e.g., Adobe Audition or the open-source Audacity) and extracts the lyrics data. The resulting lyrics data is converted into text format and used for further processing.

[0957] Vowel-focused lyrics translation

[0958] The server uses text generation AI (e.g., GPT-4) to translate the extracted lyrics into the target language. During translation, particular emphasis is placed on matching vowels, and adjustments are made to ensure a natural melody line. The translated lyrics are then used for further processing.

[0959] User emotion recognition and expression adjustment

[0960] Users can use a camera or microphone to input their emotions. The server uses an Emotion API (e.g., Google Cloud Vision API or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice and recognize their emotions. Based on the recognized emotions, the server adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice.

[0961] Singing voice generation that preserves vocal quality and emotion

[0962] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a speech generation AI (e.g., Amazon Polly or Google Cloud Text-to-Speech) to generate a singing voice that retains the vocal quality and emotion of the original singer. The speech generation AI receives the translated lyrics and audio data as input and reproduces a realistic, emotional singing voice.

[0963] Distribution of generated data

[0964] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. The server temporarily stores this file and generates a download link for the user's device. The user can click the link on their device to download the generated data.

[0965] Specific examples

[0966] For example, consider a case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server then uses a speech generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[0967] Prompt Sentence Examples

[0968] To translate the Japanese lyrics of the anime song "Sakuranbo Kiss" into English, the following prompt sentence is input to the generative AI model:

[0969] Original lyrics: "Cherry Kiss Lyrics"

[0970] Target language: English

[0971] Emotion: Happy

[0972] Translated lyrics generation

[0973] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[0974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0975] Step 1:

[0976] The user selects and uploads an audio file of an anime song. The user uses the device to select the audio file and clicks the upload button on the operation screen. This operation sends the audio file from the device to the server.

[0977] Input: User selected audio file

[0978] Output: Audio file sent to the server

[0979] Step 2:

[0980] The server receives and stores the audio file. The server receives the audio file uploaded from the device and stores it in temporary storage.

[0981] Input: Uploaded audio file

[0982] Output: Saved audio file

[0983] Step 3:

[0984] The server analyzes the audio file and extracts the lyrics data. The server launches audio analysis software (e.g., Adobe Audition) and analyzes the saved audio file. As a result of the audio analysis, lyrics data is extracted.

[0985] Input: Saved audio file

[0986] Output: Extracted lyrics data (text format)

[0987] Step 4:

[0988] The server translates the lyric data into the target language. The server uses a text generation AI (e.g., GPT-4) to translate the extracted lyric data into the target language. The translated lyrics are generated by inputting the prompt sentence into the generation AI model.

[0989] Input: Extracted lyrics data

[0990] Output: Translated lyrics data

[0991] Step 5:

[0992] The system recognizes the user's emotions and adjusts the lyrics and vocals accordingly. The user sends emotional information to the server using a camera or microphone. The server then analyzes the user's emotions using the Emotion API and adjusts the emotional expression of the translated lyrics data and the generated vocals.

[0993] Input: User emotion information, translated lyrics data

[0994] Output: Adjusted lyrics data, singing data reflecting emotional expressions

[0995] Step 6:

[0996] Generation of singing voices that retains vocal quality and emotion. The server launches a voice generation AI (e.g., Amazon Polly) and inputs the adjusted lyrics data to generate singing voice data. This singing voice data retains the vocal quality and emotion of the original singer.

[0997] Input: Adjusted lyrics data

[0998] Output: Generated singing voice data

[0999] Step 7:

[1000] The generated data is distributed to the user. The server combines the generated vocal data and translated lyrics data into a single file and temporarily stores it. A download link is generated for the user's device, and the user clicks the link to download the file.

[1001] Input: Generated vocal data, translated lyrics data

[1002] Output: Download link, user downloaded file

[1003] (Application example 2)

[1004] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1005] With conventional music streaming services, users often experience a lack of emotional expression when listening to their favorite anime songs in multiple languages. This can lead to users feeling uneasy about the translated lyrics and vocals, resulting in an unsatisfying music experience. Furthermore, typical translation systems struggle to retain vowel matches and emotional nuances, preventing them from providing a more personalized music experience.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload an audio file of an original song, means for analyzing the uploaded audio file and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics according to the emotions, means for generating a singing voice that retains the vocal quality and emotion of the singer of the original song based on the translated lyrics, and means for delivering the generated multilingual singing voice data to the user. This makes it possible to enjoy multilingual anime songs that reflect emotional expressions in real time.

[1007] "Original song audio file" refers to the audio data of the original anime song that the user uploads to the system.

[1008] "Uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[1009] "Means for analyzing and extracting lyric data" means software technology for extracting lyrics in text form from uploaded audio files.

[1010] "Target language" refers to the language into which the translation is to be made.

[1011] "Methods of translating with emphasis on vowels" refers to a translation technique that pays particular attention to the degree of agreement of vowels when translating, in order to maintain consistency with the melody of the original song.

[1012] The "means for recognizing user emotions" is a function that uses input devices such as a camera or microphone to read emotions from the user's facial expressions and tone of voice.

[1013] "Means for adjusting the expression of translated lyrics according to emotions" refers to a technology that changes the content and nuance of translated lyrics based on the recognized emotions of the user.

[1014] "Means for generating singing voices that retain vocal quality and emotion" refers to technology that generates singing voices that reproduce the vocal quality and emotion of the original singer based on translated lyrics and their emotional expressions.

[1015] The "distribution means" refers to a technique for providing the generated multilingual singing voice data to users.

[1016] The present invention relates to a system for translating anime songs into multiple languages ​​and adjusting the emotional expression of the translated lyrics and singing voice in response to the user's emotions. This system is specifically implemented using the following series of means.

[1017] Hardware and Software

[1018] To realize this system, the following hardware and software are used.

[1019] Hardware used

[1020] 1. Device: Smartphone (e.g. Apple iPhone, Samsung Galaxy)

[1021] 2. Server: High-performance server (e.g., cloud-based server infrastructure)

[1022] Software used

[1023] 1. Frontend: User interface using React Native

[1024] 2. Backend: Server-side program using Python (Flask) and Node.js

[1025] 3. Data Processing API:

[1026] Google Cloud Speech-to-Text: Extracting lyrics from audio

[1027] Amazon Translate: Translation into the target language

[1028] OpenAI GPT-4: Text generation and emotional expression adjustment

[1029] AWS Polly: Emotionally Reflective Singing Voice Generation

[1030] System operation

[1031] 1. User Uploads

[1032] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the application. The device then sends the selected audio file to the server.

[1033] 2. Server-side analysis

[1034] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the lyrics data in text format.

[1035] 3. Lyric translation

[1036] The extracted lyrics data is translated into the target language using Amazon Translate, with particular attention paid to vowel matching.

[1037] 4. Emotion recognition

[1038] The server uses data from the camera and microphone to recognize the user's emotions using emotion recognition libraries such as OpenCV and Librosa, which determine emotions from the user's facial expressions and tone of voice.

[1039] 5. Emotion-Based Regulation

[1040] Using OpenAI GPT-4, the translation adjusts the expression of the lyrics depending on the recognized emotion. For example, if the user is recognized as "sad," the translation will add a nuance of sadness.

[1041] 6. Singing Voice Generation

[1042] Based on the translated lyrics and their emotional expressions, AWS Polly is used to generate a singing voice that retains the vocal quality and emotion of the original singer.

[1043] 7. Data Distribution

[1044] The generated multilingual singing voice data is provided to users in real time, and can be played back at any time via their smartphones.

[1045] Specific examples

[1046] For example, consider a user uploading an anime song called "Cherry Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and uses Amazon Translate to add a more positive nuance to the translated lyrics, "Cherry Kiss." AWS Polly then generates an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[1047] Prompt Sentence Examples

[1048] "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (eg, joy, sadness)."

[1049] Using these prompts, the system generates emotion-based translated lyrics in the target language, providing the user with a highly personalized musical experience.

[1050] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1051] Step 1:

[1052] Uploading an audio file

[1053] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the app. The audio file selected by the user is sent from the device to the server. The input is the audio file selected by the user, and the output is the audio file saved on the server.

[1054] Step 2:

[1055] Analyzing audio files

[1056] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the text lyrics. This analysis converts the audio signal into text data. The input is the uploaded audio file, and the output is the extracted text lyrics.

[1057] Step 3:

[1058] Lyrics translation

[1059] The server uses Amazon Translate to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching to generate natural lyrics. The input is text-format lyrics data, and the output is the lyrics translated into the target language.

[1060] Step 4:

[1061] emotion recognition

[1062] The user inputs facial expressions and voice using a camera or microphone. The server uses emotion recognition libraries such as OpenCV and Librosa to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the identified emotion data.

[1063] Step 5:

[1064] Adjustment of translated lyrics

[1065] The server uses OpenAI's GPT-4 to adjust the expression of the translated lyrics based on emotion. An example prompt is "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (e.g., joy, sadness)." The input is translated lyrics data and emotion data, and the output is the adjusted translated lyrics data.

[1066] Step 6:

[1067] Singing voice generation

[1068] The server uses AWS Polly to generate a singing voice that retains the vocal quality and emotion of the original singer based on the adjusted translated lyrics data. The input is the adjusted translated lyrics data, and the output is the generated singing voice data.

[1069] Step 7:

[1070] Data distribution

[1071] The server formats the generated multilingual vocal data and provides it to the user. The final data is compiled into a single file, which the user can play in real time via their smartphone. The input is the generated vocal data and the adjusted translated lyrics data, and the output is a file provided to the user as a download link.

[1072] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1073] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1074] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1075] [Fourth embodiment]

[1076] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1077] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1078] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1079] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1080] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1081] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1082] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1083] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1084] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1085] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1086] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1087] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1088] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1089] This invention relates to a system for efficiently realizing the multilingualization of anime songs, specifically to a program for translating lyrics and generating singing voices. Here, the processing of the program of this system is explained in natural language, and an embodiment is shown with concrete examples.

[1090] Program Overview

[1091] The system is realized by combining the following main functions:

[1092] 1. A user uploads an audio file of an anime song.

[1093] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[1094] 3. The server translates the lyrics into the target language, focusing on vowels.

[1095] 4. The server generates a singing voice that preserves the vocal quality and emotion based on the translated lyrics.

[1096] 5. The server distributes the generated multilingual singing voice data to the user.

[1097] Program processing (natural language explanation)

[1098] User-uploaded anime songs

[1099] The user selects the original audio file of the anime song using the terminal and uploads it to the system's server. The terminal sends the selected audio file to the server, and the server stores the received audio file.

[1100] Server-based analysis and lyric data extraction

[1101] The server analyzes the received audio file and extracts the lyrics data, using voice recognition software and a lyrics database to obtain text-format lyrics information from the audio file.

[1102] Vowel-focused lyrics translation

[1103] The server translates the extracted lyrics into the target language, placing emphasis on vowels. For example, it uses voice generation AI to translate the lyrics while paying particular attention to matching vowels to ensure that they match the melody line of the original song. It also has a function to evaluate and correct the degree of vowel matching in the translation results.

[1104] Singing voice generation that preserves vocal quality and emotion

[1105] Based on the translated lyrics, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. Specifically, the AI ​​receives the translated lyrics and the original voice as input and reproduces a realistic singing voice. The AI ​​also takes into account vocal expressions and intonation, aiming for a natural result.

[1106] Distribution of generated data

[1107] The server formats the generated multilingual vocal data and translated lyrics, combines them into a single file, and provides a download link to distribute this file to the user's device. The user clicks the link on their device to download the generated data.

[1108] Specific examples

[1109] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." The user uploads the audio file to the server, which extracts the lyrics of "Sakuranbo Kiss" and translates them into English as "Cherry Kiss," emphasizing vowels. The server then uses a voice generation AI to generate "Cherry Kiss," preserving the vocal quality and emotion of the original singer, and delivers it to the user. The user can play "Cherry Kiss" on their device and enjoy the melody line of the original song without any sense of incongruity.

[1110] In this way, the system of the present invention provides a new way to enjoy anime songs internationally by converting anime songs uploaded by users into multiple languages ​​and playing them back in natural singing voices.

[1111] The processing flow will be explained below.

[1112] Step 1:

[1113] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server saves the received audio file.

[1114] Step 2:

[1115] The server uses audio analysis software to analyze the uploaded audio files, extracting lyrics data and converting it into text format.

[1116] Step 3:

[1117] The server then calls a text generation AI to translate the extracted lyrics into the target language, placing emphasis on matching vowels to ensure that the melody line of the original song is not out of place.

[1118] Step 4:

[1119] The server evaluates the degree of vowel matching of the generated translated lyrics and corrects any mismatches based on the evaluation results, resulting in more natural-sounding translated lyrics.

[1120] Step 5:

[1121] The server then passes the translated lyrics and the original audio file as input to the AI ​​voice generator, which then generates a new singing voice for the translated lyrics while preserving the vocal quality and emotion of the original singer.

[1122] Step 6:

[1123] The server formats the new vocal data and the translated lyrics, and combines them into a single file. The formatted data is temporarily saved.

[1124] Step 7:

[1125] The server creates a download link for the generated multilingual singing voice data and provides it to the user's device. The user clicks the provided link on their device to download the generated data.

[1126] Step 8:

[1127] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs, which are played with natural singing voices that blend seamlessly with the original melody lines.

[1128] Example 1

[1129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1130] In the multilingualization of anime songs, it is difficult to generate natural-sounding singing voices in different languages ​​while preserving the melody and emotion of the original song. Therefore, there is a need for a system that can generate multilingual singing voices efficiently and with high quality.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1132] In this invention, the server includes a means for users to upload audio files, a means for receiving and saving the uploaded audio files, a means for analyzing the audio files and extracting text-format lyrics data, a means for translating the extracted lyrics data into a target language with particular emphasis on matching vowels, a means for using a generative AI model to generate singing voices that retain the vocal quality and emotion based on the translated lyrics, and a means for delivering the generated multilingual singing voice data to users. This allows for efficient and high-quality multilingual anime song generation, allowing users to enjoy songs that retain the emotion of the original.

[1133] "User" refers to a person or organization that accesses the System, uploads audio files, and receives multilingual vocal data.

[1134] "Audio file" refers to a file format (e.g., MP3, WAV, AAC, etc.) that contains the audio data of the original anime song.

[1135] "Upload" refers to the act of a user sending an audio file from their own device to a server.

[1136] "Receiving" refers to the process in which the server obtains the audio file sent by the user and saves it in a specified location.

[1137] "Saving" refers to storing the received audio file in a specified directory on the server.

[1138] "Analysis" refers to the process of running a computer program to extract the necessary information from the audio file.

[1139] "Lyrics data" refers to text-format lyrics information extracted from an audio file.

[1140] "Target language" refers to the language into which the original lyrics data is translated.

[1141] "Vowel agreement" refers to the situation where the vowels in the original lyrics and the translated lyrics are similar or identical.

[1142] "Translation" refers to the process of converting the original lyrics data into a target language.

[1143] "Voice quality" refers to the sound quality and characteristics unique to each individual voice.

[1144] "Emotion" refers to the feelings and moods that a singer expresses through lyrics and melody.

[1145] "Voice generation" refers to the process of recreating the vocal quality and emotion based on translated lyrics and synthesizing a new singing voice.

[1146] "Generative AI model" refers to the mathematical model of artificial intelligence used to generate speech.

[1147] "Distribution" refers to the act of providing generated data through a user-accessible link.

[1148] "Multilingual vocal data" refers to vocal data translated into a target language.

[1149] A "prompt sentence" refers to an input command given to a generative AI model to perform appropriate processing.

[1150] The present invention relates to a system for efficiently realizing the multilingualization of anime songs, which allows users to enjoy multilingual versions of songs translated into target languages ​​while preserving the melody and emotion of the original songs.

[1151] System Program Overview

[1152] The system is realized by combining the following main functions:

[1153] 1. The user uploads an audio file.

[1154] 2. The server receives and stores the audio file.

[1155] 3. The server analyzes the audio file and extracts the lyrics data.

[1156] 4. The server translates the lyrics data into the target language, focusing on matching vowels.

[1157] 5. The server generates a singing voice using a generative AI model based on the translated lyrics.

[1158] 6. The server distributes the generated multilingual singing voice data to the user.

[1159] User uploads audio files

[1160] Users access the system's browser interface using their devices, select and upload the anime song audio files, and then click the "Upload" button to send the audio files to the server.

[1161] Receiving and storing audio files on the server

[1162] The server receives the uploaded audio file and stores it in a specified storage directory. The " / uploads" directory is used as an example.

[1163] Server-based audio file analysis and lyric data extraction

[1164] The server sends the saved audio file to speech recognition software (e.g., Google Cloud Speech-to-Text API) to extract lyrics data in text format, and, if necessary, references a lyrics database (e.g., Musixmatch API) to verify the extracted results.

[1165] Server-based vowel-focused lyrics translation

[1166] The server sends the extracted lyrics data to a translation API (e.g., Google Translate API) and translates it into the target language. During this process, an algorithm is applied to ensure that vowels match. An example of a prompt is "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds."

[1167] Server-based singing voice generation that preserves vocal quality and emotion

[1168] Based on the translated lyrics, the server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model). An example of a prompt for the generative AI model is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion."

[1169] Server-generated data distribution

[1170] The server combines the translated lyrics and the generated vocal data into a single file and generates a link that users can access. Users can click the provided link to download and play the multilingual vocal data.

[1171] Specific examples

[1172] For example, if a user uploads an anime song called "Sakuranbo Kiss," the server extracts the lyrics and translates them into English as "Cherry Kiss." The generative AI model then generates "Cherry Kiss" while preserving the original singer's vocal quality and emotion. Users can download the generated audio file and enjoy the seamless melody and emotion of the original song.

[1173] In this way, the system of the present invention can easily generate anime songs in multiple languages ​​and provide them to users, thereby providing a new way to enjoy anime songs internationally.

[1174] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1175] Step 1:

[1176] A user accesses the system's web interface using a terminal and selects an audio file of an anime song. The input is an audio file (e.g., in .mp3 format). The user presses the "Upload" button to send the audio file to the server. The output is an upload request and the audio file.

[1177] Step 2:

[1178] The server receives an audio file received from a user as an HTTP request. The server saves this audio file in the " / uploads" directory. The inputs are the upload request and the audio file. The server saves the file and generates a success message. The output is the saved audio file path and a save success message.

[1179] Step 3:

[1180] The server analyzes the saved audio file using speech recognition software (e.g., Google Cloud Speech-to-Text API). The input is the path to the saved audio file. The server sends the audio file to the API and obtains the lyrics data in text format. The output is the lyrics data in text format.

[1181] Step 4:

[1182] The server sends the lyric data obtained through analysis to the translation API and translates it into the target language. To emphasize matching of vowels in particular, the prompt "Translate 'Sakuranbo Kiss' to English while preserving rhyme and vowel sounds" is used. The input consists of text-format lyric data and the translation API prompt. The server calls the translation API and obtains the translated lyric data. The output is the translated lyric data.

[1183] Step 5:

[1184] The server generates a new singing voice using a generative AI model (e.g., OpenAI's voice generation model) based on the translated lyrics data. An example prompt is "Generate a singing voice for the translated lyrics 'Cherry Kiss' while maintaining the original singer's timbre and emotion." The input is the translated lyrics data and the prompt from the generative AI model. The server runs the generative AI model and obtains the generated singing voice data. The output is the generated singing voice data.

[1185] Step 6:

[1186] The server combines the generated vocal data and translated lyrics data into a single file. For example, it creates "Cherry_Kiss_generated.mp3." The inputs are the generated vocal data and translated lyrics data. The server performs file generation and creates a single output file. The output is "Cherry_Kiss_generated.mp3."

[1187] Step 7:

[1188] The server generates a download link to deliver the generated file. The input is the formatted output file (e.g. "Cherry_Kiss_generated.mp3"). The server generates a download link and provides it to the user. The output is the download link.

[1189] Step 8:

[1190] The user clicks the provided download link to download the generated multilingual vocal data. The input is the download link. The user receives the file and can play it on their device. The output is the downloaded "Cherry_Kiss_generated.mp3."

[1191] (Application example 1)

[1192] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1193] Conventionally, localizing anime songs into multiple languages ​​and generating natural-sounding singing voices has been time-consuming and costly. It has also been difficult to seamlessly integrate the translated lyrics with the original melody, and providing the translated lyrics in a format that users can easily play on various devices has been a challenge. The present invention aims to solve these problems and provide a system that efficiently translates anime songs into multiple languages ​​with high quality.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1195] In this invention, the server includes: means for users to upload audio files of original songs; means for analyzing the uploaded audio files and extracting lyric data; means for translating the extracted lyric data into a target language with emphasis on vowels; means for generating singing voices that retain the vocal quality and emotion of the original singer based on the translated lyrics; means for distributing the generated multilingual singing voice data to users; means including a tool for enabling playback of the generated multilingual singing voice data; and means for providing two-dimensional language-converted lyrics and singing voice data. This enables multilingualization of anime songs, generation of natural singing voices, and playback on a variety of devices.

[1196] The "original song audio file" is the audio data of the original song uploaded by the user.

[1197] "User uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[1198] The "means for analyzing and extracting lyric data" is a function that analyzes the uploaded audio file and extracts the text data of the lyrics using voice recognition technology.

[1199] The "means for translating into the target language with emphasis on vowels" is a function for translating extracted lyrics data into the target language, with particular emphasis on matching vowels.

[1200] "Means for generating singing voices that retain the vocal quality and emotion of the original singer" refers to a function that generates singing voices in a new language based on translated lyrics while retaining the vocal quality and emotion of the original singer.

[1201] The "means for distributing the generated multilingual singing voice data to the user" is a function for transmitting the generated multilingual singing voice data to the user's terminal.

[1202] "Means including tools that enable playback of the generated multilingual singing voice data" refers to a function that provides the software or hardware necessary to play the generated multilingual singing voice data on a device such as a smartphone or computer.

[1203] The "means for providing lyrics and singing voice data that have been converted into two-dimensional language" is a function that provides the user with lyrics translated into a plurality of target languages ​​and the corresponding singing voice data.

[1204] This invention is a system for efficiently realizing the multilingualization of anime songs, specifically translating lyrics and generating singing voices. The system aims to allow users to upload the original song audio file, have the server analyze it, extract the lyrics data, translate it into the target language, and generate and distribute the multilingual singing voice while preserving the vocal quality and emotion of the original singer.

[1205] System Program Overview

[1206] 1. Upload your audio file

[1207] Users select the audio files of anime songs from their own devices and upload them to the server, using devices such as smartphones or PCs, which then send the files to the server via the Internet.

[1208] 2. Extracting Lyric Data

[1209] The server receives the uploaded audio file, analyzes it, and extracts the lyrics data using voice recognition software such as Google Cloud Speech-to-Text, which extracts the lyrics text data from the audio file.

[1210] 3. Vowel-focused lyrics translation

[1211] The server translates the extracted lyrics into the target language using a translation API such as Google Cloud Translation, paying particular attention to matching vowels.

[1212] 4. Singing voice generation that preserves vocal quality and emotion

[1213] Based on the translated lyrics, the server uses voice generation AI such as DeepVoice and WaveNet to generate a singing voice that retains the vocal quality and emotion of the original singer, resulting in a natural-sounding singing voice that corresponds to the translated lyrics.

[1214] 5. Distribution of generated data

[1215] The generated multilingual singing voice data is distributed to the user's terminal, and the user can play back the generated singing voice data on his / her own terminal.

[1216] Specific use cases

[1217] For example, consider the case where a user uploads the original version of an anime song called "Gurenge." The server analyzes this audio file and extracts the lyrics. The server then translates the lyrics into English, checking for vowel matches and making any necessary corrections. Based on the translated lyrics, the server then uses DeepVoice and WaveNet to generate an English version of "Gurenge," creating a singing voice that retains the vocal quality and emotion of the original singer. Finally, this generated singing data is provided to the user's device as a download link.

[1218] Prompt Sentence Examples

[1219] "I would like this audio file translated into English."

[1220] "I want the translated lyrics to be used to generate a singing voice with the original quality."

[1221] This allows users to easily enjoy multilingual anime songs, providing a new way of enjoying anime among international anime fans.

[1222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1223] Step 1:

[1224] The user selects an audio file of an anime song from their device and uploads it to the server. When uploading, the user uses a specific web page or application to select and send the audio file via the upload button. The server receives and saves this file. The input is the audio file from the user, and the output is the audio file saved on the server.

[1225] Step 2:

[1226] The server analyzes the received audio file and extracts the lyrics data. This analysis is performed using the Google Cloud Speech-to-Text API. The server sends the audio file to the API and obtains the lyrics data in text format. The input is the uploaded audio file, and the output is the lyrics text data.

[1227] Step 3:

[1228] The server translates the extracted lyrics data into the target language. The translation is performed using the Google Cloud Translation API, with particular attention paid to matching vowels. The server sends the lyrics text data to the API and obtains the translated lyrics data. The input is lyrics text data, and the output is translated lyrics data.

[1229] Step 4:

[1230] The server generates a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics data. This process uses voice generation AI models such as DeepVoice and WaveNet. The server inputs the translated lyrics and the original voice features into the voice generation model to generate a realistic singing voice. The input is the translated lyrics data and the original voice feature data, and the output is the generated singing voice data.

[1231] Step 5:

[1232] The server distributes the generated multilingual singing voice data to users. At this time, it provides users with a download link so that they can obtain the data by clicking the link. The input is the generated singing voice data, and the output is the distributed singing voice data.

[1233] Step 6:

[1234] The terminal can play the distributed multilingual vocal data. Users download the data to their smartphones or computers and play the vocals using an audio player application. The input is the distributed multilingual vocal data, and the output is the played vocals.

[1235] This allows users to easily enjoy multilingual versions of anime songs with high-quality singing voices.

[1236] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1237] The present invention is a system that efficiently translates anime songs into multiple languages, and incorporates an emotion engine that can recognize the user's emotions and adjust the emotional expression of the translated lyrics and singing voice accordingly. Below, we will explain the specific program processing of this system in natural language, and provide examples of embodiments.

[1238] Program Overview

[1239] The system includes the following main functions:

[1240] 1. A user uploads an audio file of an anime song.

[1241] 2. The server receives the audio file, analyzes it, and extracts the lyrics data.

[1242] 3. The server translates the lyrics into the target language, focusing on vowels.

[1243] 4. The server recognizes the user's emotions and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on those emotions.

[1244] 5. Based on the translated lyrics, the server generates a singing voice that retains the vocal quality and emotion of the original singer.

[1245] 6. The server distributes the generated multilingual singing voice data to the user.

[1246] Program processing (natural language explanation)

[1247] User-uploaded anime songs

[1248] The user selects an audio file of the original anime song using the terminal and uploads it to the system's server. The terminal transmits the selected audio file to the server, which receives and stores the audio file.

[1249] Server-based analysis and lyric data extraction

[1250] The server uses audio analysis software to analyze the uploaded audio files and extract the lyrics data, which is then converted into text format.

[1251] Vowel-focused lyrics translation

[1252] The server then uses text generation AI to translate the extracted lyrics into the target language. The translation process places particular emphasis on matching vowels to ensure a seamless melody line. The translated lyrics may then be further adjusted based on the user's emotions.

[1253] User emotion recognition and expression adjustment

[1254] The server uses an emotion engine to recognize the user's emotions. This emotion recognition can be performed, for example, by the user inputting facial expressions and voice using a camera or microphone. The emotion engine identifies the user's emotion (e.g., joy, sadness, surprise, etc.) and adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice based on that emotion. For example, if the user is recognized as "sad," the translated lyrics are adjusted to add a nuance of sadness.

[1255] Singing voice generation that preserves vocal quality and emotion

[1256] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a voice generation AI to generate a singing voice that retains the vocal quality and emotion of the original singer. The voice generation AI receives the translated lyrics and the original audio data as input and reproduces a realistic and emotional singing voice.

[1257] Distribution of generated data

[1258] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. This file is temporarily saved and a download link is provided to the user's device. The user clicks the link on their device to download the generated data.

[1259] Specific examples

[1260] For example, consider the case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server uses a voice generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer, and delivers it to the user. The user can then play "Cherry Kiss" on their device and enjoy the multilingual anime song that reflects the positive emotion.

[1261] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[1262] The processing flow will be explained below.

[1263] Step 1:

[1264] The user selects and uploads an audio file of an anime song. The user's device sends the selected audio file to the server. The server receives and stores the uploaded audio file.

[1265] Step 2:

[1266] The server uses audio analysis software to analyze the uploaded audio file, extracting lyrics data as a result and converting it into text format.

[1267] Step 3:

[1268] The server uses text generation AI to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching and adjusting the melody line of the original song so that it fits seamlessly with the original.

[1269] Step 4:

[1270] The user's device uses a camera and microphone to transmit the user's facial expressions and voice to an emotion recognition engine, which then identifies the user's emotions (e.g., joy, sadness, surprise, etc.) in real time.

[1271] Step 5:

[1272] The server adjusts the expression of the translated lyrics based on the user's recognized emotional data. For example, if the user feels sad, the server adds a nuance of sadness to the translated lyrics.

[1273] Step 6:

[1274] The server uses a voice generation AI to generate a new singing voice based on the emotion-adjusted translated lyrics and the original voice data. The generated singing voice retains the vocal quality and emotion of the original singer while also incorporating expressions that correspond to the user's emotions.

[1275] Step 7:

[1276] The server formats the newly generated vocal data and the adjusted translated lyrics, and combines them into a single file. This formatted data is temporarily stored on the server.

[1277] Step 8:

[1278] The server provides a download link for the generated multilingual singing voice data to the user's device, and the user clicks the link using the device to download the generated data.

[1279] Step 9:

[1280] Users can play the downloaded files on their devices and enjoy the translated multilingual anime songs. The singing voices they hear are natural and reflect the user's emotions.

[1281] Example 2

[1282] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] Conventional multilingual speech generation systems have difficulty accurately preserving the emotion of the original song, making it difficult to generate translated lyrics and speech that reflect the user's emotions. Furthermore, there are insufficient means for increasing the degree of agreement between the vowels in the translated lyrics and the melody line, resulting in the generated multilingual singing voices often sounding unnatural.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1285] In this invention, the server includes means for users to upload audio files, means for analyzing the uploaded audio files and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics and the emotional expression of the generated singing voice, means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics, and means for delivering the generated multilingual singing voice data to users. This makes it possible to generate natural multilingual singing voices that reflect the user's emotions and reduce the sense of incongruity with the melody line of the original song.

[1286] A "user" is someone who uses this system to translate anime songs into multiple languages.

[1287] An "audio file" is a file in which the original music of an anime song is saved in digital format.

[1288] "Uploading means" refers to the method or tool that a user uses to send an audio file to the server.

[1289] "Analysis means" refers to the techniques and tools used to extract lyric data from audio files.

[1290] "Lyric data" is text-format lyric information extracted from an audio file.

[1291] "Target language" is the language into which the original lyrics are translated.

[1292] "Means of translation" refers to the methods and tools used to convert the original lyrics data into the target language.

[1293] "Placing emphasis on vowels" means increasing the degree of consistency of the vowels in the translated lyrics and maintaining compatibility with the melody line.

[1294] "Means for recognizing user emotions" refers to a method or device for analyzing emotions from the user's facial expressions, voice, etc.

[1295] The "means for adjusting emotional expression" refers to a method or tool for adjusting the translated lyrics or the generated singing voice based on the recognized emotions of the user.

[1296] "Preserving vocal quality and emotion" means reproducing the original singer's unique vocal quality and emotion.

[1297] "Generative means" refers to methods and tools for generating new singing voices based on translated lyrics.

[1298] The "distribution means" refers to a method or tool for providing the generated multilingual singing voice data to users.

[1299] "Evaluation methods" are methods or tools that confirm how well the vowels in the translated lyrics match those in the original song.

[1300] "Correction methods" are methods or tools for adjusting and correcting the mismatched vowels.

[1301] "Formatting means" refers to the method or tool used to combine the generated vocal data and translated lyrics into a single file.

[1302] The present invention is a system that allows users to upload audio files of anime songs, analyzes the audio files to extract lyric data, translates the data into target languages, and finally generates and distributes expressive singing voices in multiple languages. Specific embodiments of this system are described below.

[1303] User-uploaded anime songs

[1304] The user selects an anime song audio file using a device (e.g., a smartphone or PC) and clicks the upload button on the operation screen to send the audio file to the system's server. The device then sends the selected audio file to the server, which receives and stores the audio file.

[1305] Server-based analysis and lyric data extraction

[1306] The server analyzes the uploaded audio files using audio analysis software (e.g., Adobe Audition or the open-source Audacity) and extracts the lyrics data. The resulting lyrics data is converted into text format and used for further processing.

[1307] Vowel-focused lyrics translation

[1308] The server uses text generation AI (e.g., GPT-4) to translate the extracted lyrics into the target language. During translation, particular emphasis is placed on matching vowels, and adjustments are made to ensure a natural melody line. The translated lyrics are then used for further processing.

[1309] User emotion recognition and expression adjustment

[1310] Users can use a camera or microphone to input their emotions. The server uses an Emotion API (e.g., Google Cloud Vision API or Microsoft Azure Emotion API) to analyze the user's facial expressions and voice and recognize their emotions. Based on the recognized emotions, the server adjusts the expression of the translated lyrics and the emotional expression of the generated singing voice.

[1311] Singing voice generation that preserves vocal quality and emotion

[1312] Based on the translated lyrics and lyrics adjusted based on the user's emotions, the server uses a speech generation AI (e.g., Amazon Polly or Google Cloud Text-to-Speech) to generate a singing voice that retains the vocal quality and emotion of the original singer. The speech generation AI receives the translated lyrics and audio data as input and reproduces a realistic, emotional singing voice.

[1313] Distribution of generated data

[1314] The server formats the generated multilingual vocal data and translated lyrics and combines them into a single file. The server temporarily stores this file and generates a download link for the user's device. The user can click the link on their device to download the generated data.

[1315] Specific examples

[1316] For example, consider a case where a user uploads an anime song called "Sakuranbo Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and adds a more positive nuance to the translated lyrics, "Cherry Kiss." The server then uses a speech generation AI to generate an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[1317] Prompt Sentence Examples

[1318] To translate the Japanese lyrics of the anime song "Sakuranbo Kiss" into English, the following prompt sentence is input to the generative AI model:

[1319] Original lyrics: "Cherry Kiss Lyrics"

[1320] Target language: English

[1321] Emotion: Happy

[1322] Translated lyrics generation

[1323] In this way, the system of the present invention recognizes the user's emotions and generates translated lyrics and singing voices accordingly, thereby providing more personalized multilingual anime song versions.

[1324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1325] Step 1:

[1326] The user selects and uploads an audio file of an anime song. The user uses the device to select the audio file and clicks the upload button on the operation screen. This operation sends the audio file from the device to the server.

[1327] Input: User selected audio file

[1328] Output: Audio file sent to the server

[1329] Step 2:

[1330] The server receives and stores the audio file. The server receives the audio file uploaded from the device and stores it in temporary storage.

[1331] Input: Uploaded audio file

[1332] Output: Saved audio file

[1333] Step 3:

[1334] The server analyzes the audio file and extracts the lyrics data. The server launches audio analysis software (e.g., Adobe Audition) and analyzes the saved audio file. As a result of the audio analysis, lyrics data is extracted.

[1335] Input: Saved audio file

[1336] Output: Extracted lyrics data (text format)

[1337] Step 4:

[1338] The server translates the lyric data into the target language. The server uses a text generation AI (e.g., GPT-4) to translate the extracted lyric data into the target language. The translated lyrics are generated by inputting the prompt sentence into the generation AI model.

[1339] Input: Extracted lyrics data

[1340] Output: Translated lyrics data

[1341] Step 5:

[1342] The system recognizes the user's emotions and adjusts the lyrics and vocals accordingly. The user sends emotional information to the server using a camera or microphone. The server then analyzes the user's emotions using the Emotion API and adjusts the emotional expression of the translated lyrics data and the generated vocals.

[1343] Input: User emotion information, translated lyrics data

[1344] Output: Adjusted lyrics data, singing data reflecting emotional expressions

[1345] Step 6:

[1346] Generation of singing voices that retains vocal quality and emotion. The server launches a voice generation AI (e.g., Amazon Polly) and inputs the adjusted lyrics data to generate singing voice data. This singing voice data retains the vocal quality and emotion of the original singer.

[1347] Input: Adjusted lyrics data

[1348] Output: Generated singing voice data

[1349] Step 7:

[1350] The generated data is distributed to the user. The server combines the generated vocal data and translated lyrics data into a single file and temporarily stores it. A download link is generated for the user's device, and the user clicks the link to download the file.

[1351] Input: Generated vocal data, translated lyrics data

[1352] Output: Download link, user downloaded file

[1353] (Application example 2)

[1354] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1355] With conventional music streaming services, users often experience a lack of emotional expression when listening to their favorite anime songs in multiple languages. This can lead to users feeling uneasy about the translated lyrics and vocals, resulting in an unsatisfying music experience. Furthermore, typical translation systems struggle to retain vowel matches and emotional nuances, preventing them from providing a more personalized music experience.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for a user to upload an audio file of an original song, means for analyzing the uploaded audio file and extracting lyric data, means for translating the extracted lyric data into a target language with emphasis on vowels, means for recognizing the user's emotions and adjusting the expression of the translated lyrics according to the emotions, means for generating a singing voice that retains the vocal quality and emotion of the singer of the original song based on the translated lyrics, and means for delivering the generated multilingual singing voice data to the user. This makes it possible to enjoy multilingual anime songs that reflect emotional expressions in real time.

[1357] "Original song audio file" refers to the audio data of the original anime song that the user uploads to the system.

[1358] "Uploading means" is a function that allows a user to send an audio file from their own terminal to the server.

[1359] "Means for analyzing and extracting lyric data" means software technology for extracting lyrics in text form from uploaded audio files.

[1360] "Target language" refers to the language into which the translation is to be made.

[1361] "Methods of translating with emphasis on vowels" refers to a translation technique that pays particular attention to the degree of agreement of vowels when translating, in order to maintain consistency with the melody of the original song.

[1362] The "means for recognizing user emotions" is a function that uses input devices such as a camera or microphone to read emotions from the user's facial expressions and tone of voice.

[1363] "Means for adjusting the expression of translated lyrics according to emotions" refers to a technology that changes the content and nuance of translated lyrics based on the recognized emotions of the user.

[1364] "Means for generating singing voices that retain vocal quality and emotion" refers to technology that generates singing voices that reproduce the vocal quality and emotion of the original singer based on translated lyrics and their emotional expressions.

[1365] The "distribution means" refers to a technique for providing the generated multilingual singing voice data to users.

[1366] The present invention relates to a system for translating anime songs into multiple languages ​​and adjusting the emotional expression of the translated lyrics and singing voice in response to the user's emotions. This system is specifically implemented using the following series of means.

[1367] Hardware and Software

[1368] To realize this system, the following hardware and software are used.

[1369] Hardware used

[1370] 1. Device: Smartphone (e.g. Apple iPhone, Samsung Galaxy)

[1371] 2. Server: High-performance server (e.g., cloud-based server infrastructure)

[1372] Software used

[1373] 1. Frontend: User interface using React Native

[1374] 2. Backend: Server-side program using Python (Flask) and Node.js

[1375] 3. Data Processing API:

[1376] Google Cloud Speech-to-Text: Extracting lyrics from audio

[1377] Amazon Translate: Translation into the target language

[1378] OpenAI GPT-4: Text generation and emotional expression adjustment

[1379] AWS Polly: Emotionally Reflective Singing Voice Generation

[1380] System operation

[1381] 1. User Uploads

[1382] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the application. The device then sends the selected audio file to the server.

[1383] 2. Server-side analysis

[1384] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the lyrics data in text format.

[1385] 3. Lyric translation

[1386] The extracted lyrics data is translated into the target language using Amazon Translate, with particular attention paid to vowel matching.

[1387] 4. Emotion recognition

[1388] The server uses data from the camera and microphone to recognize the user's emotions using emotion recognition libraries such as OpenCV and Librosa, which determine emotions from the user's facial expressions and tone of voice.

[1389] 5. Emotion-Based Regulation

[1390] Using OpenAI GPT-4, the translation adjusts the expression of the lyrics depending on the recognized emotion. For example, if the user is recognized as "sad," the translation will add a nuance of sadness.

[1391] 6. Singing Voice Generation

[1392] Based on the translated lyrics and their emotional expressions, AWS Polly is used to generate a singing voice that retains the vocal quality and emotion of the original singer.

[1393] 7. Data Distribution

[1394] The generated multilingual singing voice data is provided to users in real time, and can be played back at any time via their smartphones.

[1395] Specific examples

[1396] For example, consider a user uploading an anime song called "Cherry Kiss." If the user expresses the emotion "happy," the server recognizes that emotion and uses Amazon Translate to add a more positive nuance to the translated lyrics, "Cherry Kiss." AWS Polly then generates an emotive version of "Cherry Kiss" that preserves the vocal quality and emotion of the original singer and delivers it to the user.

[1397] Prompt Sentence Examples

[1398] "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (eg, joy, sadness)."

[1399] Using these prompts, the system generates emotion-based translated lyrics in the target language, providing the user with a highly personalized musical experience.

[1400] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1401] Step 1:

[1402] Uploading an audio file

[1403] The user selects the original audio file of the anime song using their smartphone and uploads it to the server through the app. The audio file selected by the user is sent from the device to the server. The input is the audio file selected by the user, and the output is the audio file saved on the server.

[1404] Step 2:

[1405] Analyzing audio files

[1406] The server uses the Google Cloud Speech-to-Text API to analyze the uploaded audio file and extract the text lyrics. This analysis converts the audio signal into text data. The input is the uploaded audio file, and the output is the extracted text lyrics.

[1407] Step 3:

[1408] Lyrics translation

[1409] The server uses Amazon Translate to translate the extracted lyrics into the target language, paying particular attention to the degree of vowel matching to generate natural lyrics. The input is text-format lyrics data, and the output is the lyrics translated into the target language.

[1410] Step 4:

[1411] emotion recognition

[1412] The user inputs facial expressions and voice using a camera or microphone. The server uses emotion recognition libraries such as OpenCV and Librosa to analyze the user's emotions. The input is the user's facial expression and voice data, and the output is the identified emotion data.

[1413] Step 5:

[1414] Adjustment of translated lyrics

[1415] The server uses OpenAI's GPT-4 to adjust the expression of the translated lyrics based on emotion. An example prompt is "Translate the given anime song lyrics into {target_language} while maintaining the emotional tone of the original. Adjust the translation to align with the mood indicated by the user's detected emotions (e.g., joy, sadness)." The input is translated lyrics data and emotion data, and the output is the adjusted translated lyrics data.

[1416] Step 6:

[1417] Singing voice generation

[1418] The server uses AWS Polly to generate a singing voice that retains the vocal quality and emotion of the original singer based on the adjusted translated lyrics data. The input is the adjusted translated lyrics data, and the output is the generated singing voice data.

[1419] Step 7:

[1420] Data distribution

[1421] The server formats the generated multilingual vocal data and provides it to the user. The final data is compiled into a single file, which the user can play in real time via their smartphone. The input is the generated vocal data and the adjusted translated lyrics data, and the output is a file provided to the user as a download link.

[1422] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1424] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1425] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1426] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1427] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1428] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1429] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1430] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1431] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1432] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1433] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1434] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1435] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1436] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1437] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1438] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1439] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1440] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1441] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1442] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1443] The following is further disclosed regarding the above embodiment.

[1444] (Claim 1)

[1445] A means for users to upload the original song audio file,

[1446] A means of analyzing uploaded audio files and extracting lyric data;

[1447] A means for translating the extracted lyrics data into a target language with emphasis on vowels;

[1448] A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics;

[1449] The system includes a means for distributing the generated multilingual singing voice data to users.

[1450] (Claim 2)

[1451] 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

[1452] (Claim 3)

[1453] 2. The system according to claim 1, further comprising means for formatting the generated multilingual vocal data and translated lyrics and combining them into one file.

[1454] "Example 1"

[1455] (Claim 1)

[1456] a means for a user to upload an audio file;

[1457] means for receiving and storing uploaded audio files;

[1458] A means for analyzing the audio file and extracting lyrics data in text format;

[1459] A means for translating the extracted lyrics data into a target language, with particular emphasis on vowel matching;

[1460] Using a generative AI model to generate a singing voice that preserves the vocal quality and emotion based on the translated lyrics;

[1461] The system includes a means for distributing the generated multilingual singing voice data to users.

[1462] (Claim 2)

[1463] 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

[1464] (Claim 3)

[1465] 2. The system according to claim 1, further comprising means for formatting the generated multilingual vocal data and translated lyrics and combining them into one file.

[1466] "Application Example 1"

[1467] (Claim 1)

[1468] A means for users to upload the original song audio file,

[1469] A means of analyzing uploaded audio files and extracting lyric data;

[1470] A means for translating the extracted lyrics data into a target language with emphasis on vowels;

[1471] A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics;

[1472] a means for distributing the generated multilingual singing voice data to users;

[1473] means for enabling playback of the generated multilingual singing voice data;

[1474] A system including means for providing two-dimensional language-converted lyrics and vocal data.

[1475] (Claim 2)

[1476] 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

[1477] (Claim 3)

[1478] 2. The system according to claim 1, further comprising means for formatting the generated multilingual vocal data and translated lyrics and combining them into one file.

[1479] "Example 2: Combining Emotion Engines"

[1480] (Claim 1)

[1481] a means for a user to upload an audio file;

[1482] A means of analyzing uploaded audio files and extracting lyric data;

[1483] A means for translating the extracted lyrics data into a target language with emphasis on vowels;

[1484] A means for recognizing a user's emotions and adjusting the expression of the translated lyrics and the emotional expression of the generated singing voice;

[1485] A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics;

[1486] The system includes a means for distributing the generated multilingual singing voice data to users.

[1487] (Claim 2)

[1488] 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

[1489] (Claim 3)

[1490] 2. The system according to claim 1, further comprising means for formatting the generated multilingual vocal data and translated lyrics and combining them into one file.

[1491] "Application example 2 when combining emotion engines"

[1492] (Claim 1)

[1493] A means for users to upload the original song audio file,

[1494] A means of analyzing uploaded audio files and extracting lyric data;

[1495] A means for translating the extracted lyrics data into a target language with emphasis on vowels;

[1496] A means for recognizing a user's emotion and adjusting the expression of the translated lyrics according to the emotion;

[1497] A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics;

[1498] The system includes a means for distributing the generated multilingual singing voice data to users.

[1499] (Claim 2)

[1500] 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

[1501] (Claim 3)

[1502] 2. The system according to claim 1, further comprising means for arranging the multilingual vocal data and the translated lyrics generated based on the emotion-adjusted translated lyrics and combining them into one file. [Explanation of symbols]

[1503] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for users to upload the original song audio file, A means of analyzing uploaded audio files and extracting lyric data; A means for translating the extracted lyrics data into a target language with emphasis on vowels; A means for generating a singing voice that retains the vocal quality and emotion of the original singer based on the translated lyrics; The system includes a means for distributing the generated multilingual singing voice data to users.

2. 10. The system of claim 1, further comprising means for evaluating whether the vowels in the translated lyrics match the vowels in the original song and correcting any mismatches.

3. 2. The system according to claim 1, further comprising means for arranging the generated multilingual vocal data and translated lyrics and combining them into one file.

Images