Method and apparatus for processing a song

Abstract

Embodiments of the present specification provide a song processing method and device, wherein the song processing method comprises: determining a to-be-translated song based on a received song translation request; identifying to-be-translated lyrics in the to-be-translated song, source tonal information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tonal information; inputting the to-be-translated lyrics, the source tonal information, and the source alignment information into a song translation model, wherein the song translation model is used to translate the to-be-translated lyrics into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information; obtaining the target lyrics output by the song translation model and the target alignment information between the target lyrics and the source tonal information; and generating a target song according to the target lyrics, the source tonal information, and the target alignment information.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a song processing method. BACKGROUND

[0002] Song translation technology is a technical effort to improve cross-cultural communication; in order to improve the efficiency of song translation, a neural network model can be used to achieve efficient translation of songs.

[0003] However, in order to make the translated lyrics have a good corresponding relationship with the melody, the number of notes and lyrics characters is usually controlled to be consistent, resulting in a lack of notes or lyrics characters in the translated song, which cannot well maintain the consistency of the playing effect of the translated song and the original song. SUMMARY

[0004] Therefore, embodiments of the present specification provide a song processing method. One or more embodiments of the present specification also relate to a song processing apparatus, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects in the prior art.

[0005] According to a first aspect of embodiments of the present specification, a song processing method is provided, comprising:

[0006] determining a song to be translated based on a received song translation request;

[0007] identifying lyrics to be translated in the song to be translated, source pitch information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source pitch information;

[0008] inputting the lyrics to be translated, the source pitch information, and the source alignment information into a song translation model, wherein the song translation model is configured to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source pitch information;

[0009] obtaining target lyrics output by the song translation model and target alignment information between the target lyrics and the source pitch information;

[0010] generating a target song according to the target lyrics, the source pitch information, and the target alignment information.

[0011] According to a second aspect of embodiments of the present specification, a song processing apparatus is provided, comprising:

[0012] a determining module configured to determine a song to be translated based on a received song translation request;

[0013] The identification module is configured to identify lyrics to be translated in the song to be translated, source tonal information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source tonal information.

[0014] The input module is configured to input the lyrics to be translated, the source tonal information, and the source alignment information into a song translation model, where the song translation model is used to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information.

[0015] The acquisition module is configured to acquire target lyrics output by the song translation model and target alignment information between the target lyrics and the source tonal information.

[0016] The generation module is configured to generate a target song according to the target lyrics, the source tonal information, and the target alignment information.

[0017] According to a third aspect of an embodiment of the present specification, a song processing system is provided, where the song processing system includes an end-side device and a cloud-side device.

[0018] The end-side device is configured to generate a song translation request according to a song to be translated and send the song translation request to the cloud-side device.

[0019] The cloud-side device is configured to determine a song to be translated based on the received song translation request, identify lyrics to be translated in the song to be translated, source tonal information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source tonal information, input the lyrics to be translated, the source tonal information, and the source alignment information into a song translation model, where the song translation model is used to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information, acquire target lyrics output by the song translation model and target alignment information between the target lyrics and the source tonal information, generate a target song according to the target lyrics, the source tonal information, and the target alignment information, and send the target song to the end-side device.

[0020] The end-side device is further configured to play the target song.

[0021] According to a fourth aspect of an embodiment of the present specification, a computing device is provided, including:

[0022] a memory and a processor;

[0023] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the song processing method.

[0024] According to a fifth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, and the instructions, when executed by a processor, implement the steps of the song processing method.

[0025] According to a sixth aspect of an embodiment of the present specification, a computer program is provided, and when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the song processing method.

[0026] One embodiment of the present specification realizes, based on a received song translation request, determining a to-be-translated song; identifying to-be-translated lyrics in the to-be-translated song, source tone information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tone information; inputting the to-be-translated lyrics, the source tone information, and the source alignment information into a song translation model, wherein the song translation model is configured to translate the to-be-translated lyrics into corresponding target lyrics, and determine target alignment information between the target lyrics and the source tone information; obtaining the target lyrics and the target alignment information output by the song translation model; and generating a target song according to the target lyrics, the source tone information, and the target alignment information.

[0027] By determining the to-be-translated song, the to-be-translated lyrics in the to-be-translated song, the source tone information, and the alignment information between the to-be-translated lyrics and the source tone information are further determined, thereby facilitating further processing of the to-be-translated song; the determined to-be-translated lyrics, source tone information, and alignment information between the to-be-translated lyrics and the source tone information are input into the song translation model, and the song translation model can translate the to-be-translated lyrics, thereby realizing translation of the lyrics.

[0028] Further, based on the source alignment information of the to-be-translated song, the target lyrics obtained by translation and the source tone information can be matched to obtain target alignment information; and based on the target lyrics, the source tone information, and the target alignment information, a target song is generated, thereby obtaining a translated song with lyrics and tone matching, ensuring accurate alignment between lyrics and tone on the basis of a better translation result, and further making the generated target song consistent with the to-be-translated song in terms of playing effect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a scene schematic diagram of a song processing method provided by one embodiment of the present specification;

[0030] Figure 2 is a flowchart of a processing method of a song according to an embodiment of the present specification;

[0031] Figure 3 is a schematic diagram of a pooling embedding layer according to an embodiment of the present specification;

[0032] Figure 4a is a processing schematic diagram of an alignment decoder according to an embodiment of the present specification;

[0033] Figure 4b is a schematic diagram of a pitch corresponding to a song character according to an embodiment of the present specification;

[0034] Figure 5 is a flowchart of a processing process of a processing method of a song according to an embodiment of the present specification;

[0035] Figure 6 is a structural schematic diagram of a processing device of a song according to an embodiment of the present specification;

[0036] Figure 7 is a structural block diagram of a computing device according to an embodiment of the present specification;

[0037] Figure 8 is a structural schematic diagram of a processing system of a song according to an embodiment of the present specification. DETAILED DESCRIPTION

[0038] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it is understood that the present specification will encompass numerous variations apart from those described in the following disclosure. It will further be understood that the present specification includes all twists and modifications of these preferred embodiments and includes equivalents or alternatives where practical and where the technical functional descriptions permit.

[0039] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0040] It should be understood that, although the terms first, second, etc. can be employed in describing various information in one or more embodiments of the present specification, the information should not be limited to such terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, without departing from the scope of one or more embodiments of the present specification, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "upon" or "in response to determining".

[0041] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0042] First, the nomenclature involved in one or more embodiments of the present specification is explained.

[0043] Data augmentation: refers to some methods of generating more effective training data from limited data to increase the number and diversity of training samples, so as to achieve the performance of the model.

[0044] Back-translation: a data augmentation method that translates the target language into the source language, thereby generating training data to expand the data set.

[0045] Lyrics-melody alignment: refers to the correspondence of lyrics and notes in the melody, indicating the pitch and time value of the lyrics singing.

[0046] Adaptive grouping algorithm: refers to a grouping prediction algorithm that adapts to variable-length objects at different periods based on a variable condition.

[0047] Autoregression: using the performance of the same variable at previous periods to predict the performance of the variable at the current period, which in this case mainly refers to the prediction of character probability.

[0048] GagaST: Guided Alignment For Automatic Song Translation, which proposes a song translation method using multiple translation corpora and lyrics corpora for pre-training and controlling the number of translated lyrics characters.

[0049] MIDI: Musical Instrument Digital Interface, MIDI is the most widely used music standard format in the field of composition, which can be called "music that computers can understand".

[0050] Transformer model: a neural network that learns context and therefore meaning by tracking relationships in sequential data, such as the words in this sentence.

[0051] Song translation technology is a technical research made by human beings for higher level cross-cultural communication. However, despite the progress of machine translation technology, especially neural machine translation, it has not been fully explored in the field of natural language processing. Currently, there are some challenges including the lack of efficient way to collect parallel lyrics and alignment data, the difficulty in modeling the complex interaction between text and melody, and the lack of intuitive evaluation of the singing way specified by the music score. Song translation, although closely related to text translation, is essentially a more complex task. Artificial translators of songs also need to have the background of the target language, understand the source language and make poetic expressions in the target language. In addition, the translated lyrics need to be reasonably aligned with the melody to maintain the beauty of the song, which is an indispensable element in song translation. In order to solve the above technical challenges, the paper proposes a lyrics-melody co-translation model with adaptive grouping, which jointly models lyrics translation and lyrics-melody alignment within a Transformer model-based encoder-decoder framework, where the Transformer model: a neural network that learns context and therefore meaning by tracking relationships in sequential data.

[0052] GagaST also achieves the effect of lyrics translation, and can also give a limited one-to-one lyrics-melody alignment by limiting the length of lyrics, but cannot give a more flexible and reasonable melody-lyrics alignment, so the obtained lyrics singing method is not natural enough and cannot well maintain the aesthetics of the source song. Compared with GagaST, the scheme of the present specification, by the design of the note-pooling embedding and the alignment decoder, integrates the prediction of the alignment into the process of autoregressive translation, and the alignment and the translation influence and restrict each other, which can obtain better lyrics translation results and, on this basis, obtain flexible and reasonable lyrics-melody alignment prediction, wherein the note-pooling embedding can perform pooling processing on the song data, and the alignment decoder is used to determine the pitch data corresponding to the subsequent lyrics according to the current song pitch data. In addition, we also transform the back-translation data enhancement technology commonly used in machine translation to use the lyrics and lyrics-melody alignment data provided by the monolingual song data for data enhancement, which well alleviates the problem of insufficient bilingual parallel data for this task.

[0053] In the present specification, a song processing method is provided, and the present specification also relates to a song processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0054] Referring to Figure 1 , Figure 1 A scene schematic diagram of a song processing method according to one embodiment of the present specification is shown, which specifically includes the following steps.

[0055] A song to be translated is determined, and the lyrics to be translated "But you play it to the beat" in the song to be translated are input to an encoder of a song translation model; the pitch information G4-E4 corresponding to the lyrics to be translated, and the time length information 1 / 16, 1 / 2 corresponding to the pitch information, and the source alignment information A src of the lyrics and the notes are input to a first note-pooling embedding layer to obtain a source pitch matrix and a source alignment matrix; the source pitch matrix and the source alignment matrix are input to the encoder and the alignment decoder; the lyrics encoding matrix, the pitch encoding matrix and the alignment encoding matrix output by the encoder are input to the decoder to obtain the current lyrics decoding matrix "but you don't do well" and the current pitch decoding matrix "G4-G4 and the corresponding 1 / 16" as shown in the dashed box in Figure 1 ; the current lyrics decoding matrix "but you don't do well" is input to the decoder to obtain the hidden vector h j corresponding to the predicted translation lyrics character "Jian" output by the decoder, and the hidden vector h jInput to the alignment decoder; the target alignment information corresponding to the current pitch decoding matrix and the current pitch decoding matrix is input to the second pooling embedding layer to obtain the target alignment matrix, where j-1 represents the j-1th lyric character; the target alignment information and the target alignment matrix e align (y j-1：0 ) is input to the alignment decoder, so that the alignment decoder outputs the number of notes, the pitch, and the duration corresponding to the pitch of the character "Zhen" based on the source alignment matrix e align(x) , the target alignment matrix e align (y j-1：0 ), the alignment information and the hidden vector h j corresponding to the character "Zhen".

[0056] The processing method of the song of the present specification determines the song to be translated, further determines the lyrics to be translated, the source pitch information, and the alignment information between the lyrics to be translated and the source pitch information in the song to be translated, thereby facilitating further processing; the determined lyrics to be translated, source pitch information, and alignment information between the lyrics to be translated and the source pitch information are input into the song translation model, thereby obtaining the target lyrics output by the song translation model and the alignment information between the target lyrics and the source pitch information, i.e. in the case of obtaining the translated lyrics, the translated lyrics are aligned with the source pitch, so that the alignment information between the lyrics and the pitch of the target song generated based on the target lyrics and the target alignment information is more accurate

[0057] Referring to Figure 2 , Figure 2 A flowchart of a processing method of a song according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0058] Step 202: Based on the received song translation request, determine the song to be translated.

[0059] Specifically, the song translation request is received, which refers to translating a song in a current language into a song in a target language, for example, translating a Chinese song into an English song; the song translation request contains a song identifier, wherein the song identifier can be a song name, a song number, etc.; based on the song identifier, the song is obtained as the song to be translated, wherein the song to be translated refers to a song that needs to be translated.

[0060] In an embodiment of the present specification, the song translation request is received; after analyzing the song translation request, the song identifier a is determined; based on the song identifier a, the song identifier a corresponding song A is determined as the song to be translated.

[0061] The song translation request is received, and a song to be translated is determined based on the song translation request, so as to make further processing on the song to be translated.

[0062] In step 204, lyrics to be translated, source tone information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source tone information are identified in the song to be translated.

[0063] Specifically, after determining the song to be translated, the content in the song to be translated needs to be parsed, that is, the lyrics to be translated, the source tone information, and the source alignment information in the song to be translated are determined by voice recognition technology. The lyrics to be translated refer to the lyrics in the song to be translated that need to be translated. For example, if the lyrics "hello" are identified in the lyrics to be translated, the lyrics "hello" are taken as the lyrics to be translated. The source tone information refers to the tone information corresponding to the lyrics to be translated in the song to be translated. The source tone information includes pitch information and duration information. The pitch information refers to the sound height corresponding to the lyrics to be translated. The duration information refers to the sound length corresponding to the pitch information. For example, it is determined that the pitch information corresponding to the lyrics "you" is C, and the duration information corresponding to the lyrics "you" is 1 / 4. The source alignment information refers to the alignment information between the lyrics to be translated and the source tone information. For example, the source alignment information is that the lyrics "good" correspond to the pitch C and the pitch D.

[0064] In an embodiment of the present specification, after determining the song to be translated A, the lyrics to be translated "today", the source tone information "tone G and tone F", and the source alignment information "today corresponds to tone F, and today corresponds to tone G" between the lyrics to be translated and the source tone information are identified in the song to be translated A.

[0065] It should be noted that, in addition to identifying the lyrics to be translated, the source tone information, and the source alignment information in the song to be translated, if the song to be translated has pre-set song data, the lyrics to be translated, the source tone information, and the source alignment information corresponding to the song to be translated can be directly obtained without identification. The present specification does not make further limitation on the source of the lyrics to be translated, the source tone information, and the source alignment information.

[0066] The lyrics to be translated, the source tone information, and the source alignment information corresponding to the song to be translated are determined, so that further analysis can be made on the song to be translated based on the lyrics to be translated, the source tone information, and the source alignment information.

[0067] In step 206, the lyrics to be translated, the source tone information, and the source alignment information are input into a song translation model. The song translation model is used to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tone information.

[0068] The song translation model is a model that is pre-trained and used for translating lyrics and aligning source tonal information of the translated lyrics to obtain target alignment information. The target lyrics are lyrics translated based on the song translation model. For example, if the song to be translated is an English song, the song translation model translates the English lyrics of the English song into Chinese lyrics based on a preset translation requirement, i.e., obtains the target lyrics. The target alignment information is alignment information between the target lyrics and the source tonal information. For example, if the Chinese lyrics of the song to be translated are "school" and the corresponding tonal information is "tonal C and tonal D", the target lyrics obtained by translating the Chinese lyrics into English lyrics are "school", and the target alignment information of the target lyrics and the source tonal information is "school corresponds to tonal C and tonal D".

[0069] Specifically, the song translation model is input with the lyrics to be translated, source tonal information, and source alignment information, and includes an encoder, a decoder, a pooling embedding layer, and an alignment decoder.

[0070] In actual application, the specific steps of inputting the lyrics to be translated, the source tonal information, and the source alignment information into the song translation model can include steps 1-6:

[0071] Step 1: input the source tonal information and the source alignment information into the pooling embedding layer to obtain a tonal matrix and an alignment matrix.

[0072] The pooling embedding layer is a data processing layer that performs a pooling operation on input data. The tonal matrix is a matrix obtained by pooling the source tonal information, and the alignment matrix is a matrix obtained by pooling the source alignment information.

[0073] Specifically, the pooling embedding layer inputs the source tonal information and the source alignment information and outputs an embedding representation corresponding to the source tonal information and an embedding representation corresponding to the source alignment information, i.e., obtains a tonal matrix corresponding to the source tonal information and an alignment matrix corresponding to the source alignment information.

[0074] In actual application, for example, Figure 3The structure diagram of the pooling embedding layer is shown. The pooling embedding layer takes the notes and lyrics and the alignment information of the notes as input, and outputs the pooled note embedding representation and the embedding representation of the alignment; the pitch embedding layer, the position embedding layer and the time embedding in the alignment decoder process the pitch, the sequence position and the duration in the pitch information respectively; the input note sequence is composed of the MIDI pitch and the sound duration of each note; the non-overlapping mean pooling operation is performed on the note embedding representation sequence according to the alignment; the embedding representation of the alignment is obtained by processing the pitch information through the alignment probability embedding layer and the one-dimensional convolution layer; specifically, the embedding representations of the continuous notes aligned with the same lyric character are calculated by averaging. The alignment information A is represented as a 01 matrix M, where M ∈ {0, 1} L×N , wherein L is the length of the note sequence, and N is the length of the lyric sequence. When the i th note is aligned to the j th character of the lyric, M ij = 1.

[0075] The source pitch information and the source alignment information are processed by the pooling embedding layer, so as to realize the compression of the input data, simplify the network calculation complexity, and extract the main features of the data.

[0076] Step 2: input the to-be-translated lyrics, the pitch matrix and the alignment matrix into the encoder to obtain a lyric encoding matrix, a pitch encoding matrix and an alignment encoding matrix.

[0077] By encoding the to-be-translated lyrics, the pitch matrix and the alignment matrix, the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix containing the position vector are obtained, so as to facilitate further processing of the song data based on the position vector.

[0078] Step 3: input the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix into the decoder to obtain a target lyric decoding matrix and a target pitch decoding matrix.

[0079] Specifically, the decoder and the encoder in the song translation model are the encoder-decoder structure of the Transformer; the Transformer-based encoder-decoder is pre-trained on large-scale ordinary bilingual translation corpus, lyric bilingual translation corpus and lyric monolingual corpus through denoising auto-encoding and translation task. When performing translation prediction, the conditional probability P (y j |y j-1∶0 , X) of the decoding character at the current step is calculated according to the translation result and the original text before each step, wherein P represents the probability of the decoding character, y j represents the current character translation result, y j-1：0 represents the character translation result of the previous step, and X represents the translation result of the original text.

[0080] wherein the lyric encoding matrix refers to a matrix obtained by encoding the lyrics to be translated by an encoder; the tone encoding matrix refers to a matrix obtained by encoding the tone matrix by the encoder; the alignment encoding matrix refers to a matrix obtained by encoding the alignment matrix by the encoder; the target lyric decoding matrix refers to a matrix obtained by decoding the lyric encoding matrix and the alignment encoding matrix by a decoder; and the target tone decoding matrix refers to a matrix obtained by decoding the tone encoding matrix and the alignment encoding matrix by the decoder.

[0081] Further, the decoder comprises m decoding layers, wherein m is a positive integer greater than or equal to 2.

[0082] The method for inputting the lyric encoding matrix, the tone encoding matrix and the alignment encoding matrix into the decoder to obtain the target lyric decoding matrix and the target tone decoding matrix can comprise:

[0083] S61, obtaining a reference tone matrix, a reference alignment matrix and a reference lyric matrix;

[0084] S62, inputting the reference tone matrix, the reference alignment matrix, the reference lyric matrix, the lyric encoding matrix, the tone encoding matrix and the alignment encoding matrix into the first decoding layer to obtain the target lyric decoding matrix and the target tone decoding matrix output by the first decoding layer;

[0085] S63, inputting the target lyric decoding matrix and the target tone decoding matrix and the target lyric decoding matrix and the target tone decoding matrix output by the e-1 decoding layer into the e decoding layer to obtain the target lyric decoding matrix and the target tone decoding matrix output by the e decoding layer, wherein 2≤e≤m;

[0086] S64, increasing e by 1, and determining whether e is greater than m, if yes, obtaining the target lyric decoding matrix and the target tone decoding matrix, and if no, executing step S63.

[0087] The reference tone matrix, the reference alignment matrix and the reference lyric matrix refer to matrices preset in the decoder and used for decoding the input lyric encoding matrix, the tone encoding matrix and the alignment encoding matrix.

[0088] By inputting the lyric encoding matrix, the tone encoding matrix and the alignment encoding matrix into the decoder to obtain the target lyric decoding matrix and the target tone decoding matrix, accurate translation of the lyrics to be translated is realized, and the target tone decoding matrix representing the tone corresponding to the current lyric character is determined, so as to determine the tone information of the lyrics that do not match based on the target tone decoding matrix.

[0089] Step 4: determining target tone information and target alignment information corresponding to the target tone information according to the target tone decoding matrix.

[0090] Specifically, the method of determining target tone information and target alignment information corresponding to the target tone information according to the target tone decoding matrix can include:

[0091] determining target tone information and target alignment information corresponding to the target tone information according to the source tone information and the target tone decoding matrix.

[0092] wherein, based on the source tone information, the total number of notes corresponding to the song to be translated can be determined, and based on the target tone decoding matrix, the current number of notes that have been aligned with the target lyrics can be determined; the number of unaligned notes can be determined by subtracting the current number of notes from the total number of notes; and the target tone information and the target alignment information corresponding thereto can be determined according to the number of unaligned notes.

[0093] The target tone decoding matrix and the source tone information are used to determine the unmatched tone information in the song to be translated, and then the alignment information between the current unmatched lyrics characters and the unmatched tone can be determined in the unmatched tone information in the subsequent process.

[0094] Step 5: inputting the target tone information and the target alignment information into the pooling embedding layer to obtain a target tone matrix and a target alignment matrix.

[0095] Specifically, two pooling embedding layers can be included in the song translation model, a first pooling embedding layer is used to convert the source tone information and the source alignment information into a tone matrix and an alignment matrix, and a second pooling embedding layer is used to convert the target tone information into a target tone matrix and the target alignment information into a target alignment matrix.

[0096] The target tone information and the target alignment information are processed by the pooling embedding layer, so as to realize the extraction of main features of data and the compression of data, and simplify the network calculation complexity.

[0097] Step 6: inputting the target lyrics decoding matrix, the target alignment information, the target tone matrix and the target alignment matrix into the alignment decoder to obtain target lyrics and target alignment information between the target lyrics and the source tone information.

[0098] Specifically, the method of inputting the target lyrics decoding matrix, the target alignment information, the target tone matrix and the target alignment matrix into the alignment decoder to obtain target lyrics and target alignment information between the target lyrics and the source tone information can include:

[0099] determining a preset lyrics hidden vector corresponding to the target lyrics decoding matrix;

[0100] Input the preset lyric hidden vector, the target alignment information, the pitch matrix, and the target alignment matrix into the alignment decoder to obtain the target lyrics and the target alignment information between the target lyrics and the source pitch information.

[0101] The preset lyric hidden vector refers to the hidden vector extracted from the target lyric decoding matrix for which the corresponding number of notes needs to be determined.

[0102] In practical applications, during autoregressive decoding, the alignment decoder dynamically predicts the number of notes to be aligned to the lyrics; calculates the alignment of the lyrics and the melody based on the adaptive note grouping module; specifically, as Figure 4b shown, Figure 4b is a schematic diagram of the pitch corresponding to the song characters provided by an embodiment of this specification. The adaptive note grouping module can predict how many consecutive notes should be assigned to the current lyric character. For example, it is determined that the pitch information corresponding to the lyric character "珍" is A4, A4, G4, F4. k represents the serial number of the pitch corresponding to the lyric character, represents the matching probability between the current k-th pitch and the character, represents the hidden vector of the previous lyric character, represents calculating the total sum of the current matching probabilities and determining whether the current total probability reaches a probability of ≥ 1 - ∈. ∈ is a probability threshold, which can be set based on requirements, such as 0.1, 0.2, etc. R(j) represents the proportion of the matching degree of the F4 pitch to the character "珍"; first, calculate the remaining number of unaligned notes at the current decoding step Then input the notes into the average pooling layer to obtain the representation vector of the notes, so that it can be added to the representation of the translated lyric characters predicted in the previous steps with variable lengths. For all input data, use a network g(·) to process its corresponding representation, as Figure 4a shown, Figure 4a is a schematic diagram of the processing of the alignment decoder provided by an embodiment of this specification. The output of the neural network processed by the sigmoid function σ is used to calculate the sum of the matches When the sum meets the judgment condition, determine the pitch corresponding to the current character; if not, continue to determine the next pitch to match the current lyric character until the sum of the matching probabilities meets the judgment condition, then use the currently determined pitch as the pitch corresponding to the character; the sum of each probability represents the possibility of aligning the current k notes with the target lyric character, thereby achieving the alignment annotation of each character of the lyrics and the alignment of the number of notes in the whole sentence to form supervision for training the alignment decoder.

[0103] Step 208: Obtain the target lyrics output by the song translation model and the target alignment information between the target lyrics and the source pitch information.

[0104] Specifically, obtain the target lyrics corresponding to the lyrics to be translated output by the song translation model and the target alignment information corresponding to the target lyrics.

[0105] In a specific embodiment of this specification, obtain the target lyrics "hello" corresponding to the lyrics to be translated "您好" output by the song translation model and the target alignment information corresponding to the pitch information of the song to be translated for the target lyrics "hello".

[0106] In practical applications, the song translation model needs to be trained. Specifically, the song translation model can be trained based on the following steps:

[0107] Obtain song sample data, sample translation labels, and sample alignment labels, where the song sample data, sample translation labels, and sample alignment labels are generated based on monolingual song data and / or bilingual song data;

[0108] Input the song sample data into the song translation model;

[0109] Receive the predicted translation lyrics output by the song translation model and the predicted alignment information between the translation lyrics and the pitch information;

[0110] Calculate the loss value according to the predicted translation lyrics, the predicted alignment information, the sample translation labels, and the sample alignment labels;

[0111] Iteratively train the song translation model based on the loss value until the training stop condition is reached.

[0112] Among them, the song sample data refers to the monolingual song data used for training the song translation model, the sample translation labels refer to the target lyrics obtained by translating the lyrics in the song sample data into the target language, and the sample alignment labels refer to the alignment information between the target lyrics and the pitch information in the song sample data; the translation lyrics refer to the translated lyrics output by the song translation model during training; the predicted alignment information refers to the alignment information output by the song translation model during training.

[0113] It should be noted that the above song sample data, sample translation labels, and sample alignment labels can be generated based on monolingual song data and / or bilingual song data; the bilingual song data refers to the song with corresponding translated lyrics and the alignment information of the translated lyrics. For example, a bilingual song dataset that is manually translated and marked with alignment information; the monolingual song refers to a song for which the corresponding translated lyrics and the alignment information of the translated lyrics are not directly obtained.

[0114] In practical applications, the song sample data, the sample translation label and the sample alignment label are generated based on single-language song data through the following steps:

[0115] Obtaining single-language song data;

[0116] Inputting the single-language song data into a preset song translation model;

[0117] Obtaining translated lyrics and alignment information output by the preset song translation model;

[0118] Taking the translated lyrics as song sample data, and taking the translated lyrics and the alignment information as the sample translation label and the sample alignment label respectively.

[0119] The preset song translation model refers to a model for realizing lyrics translation and alignment through length control, for example, a GagaST (Automatic Song Translation for Tonal Languages) model.

[0120] Specifically, only bilingual song data, i.e., a bilingual song dataset artificially translated and annotated with alignment information, can train the preset song translation model to a desired processing effect; however, the amount of bilingual song data is limited, and the annotation time is long and the cost is high. Therefore, the scheme of the present specification obtains single-language song data and constructs a lyrics translation model with length control, i.e., a preset song translation model; the preset song translation model is used to translate single-language lyrics data back into the source language, and length control can ensure that the number of translated characters is the same as the number of musical notes, thereby generating one-to-one alignment between musical notes and characters; using the above method, a sample dataset with a large number of samples can be obtained based on single-language song data, and the alignment information is more accurate.

[0121] The song processing method of the present specification can achieve good translation results using a small amount of bilingual song annotation data and a large amount of single-language song data, and has certain business value and landing potential for opening up new song businesses.

[0122] Step 210: generating a target song according to the target lyrics, the source tonal information and the target alignment information.

[0123] Specifically, after determining the target lyrics, the source tonal information and the target alignment information, a song can be sung by a person based on the target lyrics, the source tonal information and the target alignment information, thereby obtaining a target song; or a preset voice synthesis model can be used to sing a song based on the target lyrics, the source tonal information and the target alignment information to automatically generate a target song.

[0124] One embodiment of the specification realizes determining a to-be-translated song based on a received song translation request; identifying to-be-translated lyrics in the to-be-translated song, source tonal information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tonal information; inputting the to-be-translated lyrics, the source tonal information, and the source alignment information into a song translation model, wherein the song translation model is used to translate the to-be-translated lyrics into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information; obtaining the target lyrics and the target alignment information between the target lyrics and the source tonal information output by the song translation model; and generating a target song according to the target lyrics, the source tonal information, and the target alignment information.

[0125] By determining the to-be-translated song, the to-be-translated lyrics in the to-be-translated song, the source tonal information, and the alignment information between the to-be-translated lyrics and the source tonal information are further determined, thereby facilitating further processing of the to-be-translated song; the determined to-be-translated lyrics, source tonal information, and alignment information between the to-be-translated lyrics and the source tonal information are input into the song translation model, and the song translation model can translate the to-be-translated lyrics, thereby realizing translation of the lyrics.

[0126] Further, the target alignment information can be obtained by matching the target lyrics and the source tonal information translated based on the source alignment information of the to-be-translated song; and the target song can be generated based on the target lyrics, the source tonal information, and the target alignment information, thereby obtaining a translated song with lyrics and tonal matching, ensuring accurate alignment between the lyrics and the tonal on the basis of a better translation result, and further making the generated target song consistent with the to-be-translated song in terms of playing effect.

[0127] The following describes the song processing method provided by the specification in the application of the song processing method to Chinese songs. Figure 5 The song processing method provided by the specification is further described below with reference to the application of the song processing method to Chinese songs. Figure 5 FIG. 6 shows a process flow diagram of a song processing method according to one embodiment of the specification, which specifically includes the following steps.

[0128] In step 502, song sample data, sample translation labels, and sample alignment labels are obtained, wherein the song sample data, sample translation labels, and sample alignment labels are generated based on single-language song data and bilingual song data.

[0129] In step 504, the song sample data is input into a song translation model.

[0130] In step 506, predicted translation lyrics and predicted alignment information between the translation lyrics and tonal information output by the song translation model are received.

[0131] Step 508: calculating a loss value according to the predicted translation lyrics, the predicted alignment information, the sample translation label and the sample alignment label.

[0132] Step 510: iteratively training the song translation model based on the loss value until a training stop condition is reached, to obtain a trained song translation model.

[0133] Step 512: determining a Chinese song to be translated based on a received song translation request.

[0134] Step 514: identifying Chinese lyrics to be translated in the Chinese song to be translated, source tonal information corresponding to the Chinese lyrics to be translated, and source alignment information between the Chinese lyrics to be translated and the source tonal information.

[0135] Step 516: inputting the Chinese lyrics to be translated, the source tonal information and the source alignment information into a song translation model.

[0136] Step 518: obtaining English lyrics output by the song translation model and target alignment information between the English lyrics and the source tonal information.

[0137] Step 520: generating an English song consistent with the tune of the Chinese song to be translated according to the English lyrics, the source tonal information and the target alignment information.

[0138] One embodiment of the present specification realizes determining a song to be translated based on a received song translation request; identifying lyrics to be translated in the song to be translated, source tonal information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source tonal information; inputting the lyrics to be translated, the source tonal information and the source alignment information into a song translation model, wherein the song translation model is used to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information; obtaining target lyrics output by the song translation model and target alignment information between the target lyrics and the source tonal information; and generating a target song according to the target lyrics, the source tonal information and the target alignment information.

[0139] By determining the to-be-translated song, further determining the to-be-translated lyrics in the to-be-translated song, source tonal information, and alignment information between the to-be-translated lyrics and the source tonal information, further processing is facilitated; the determined to-be-translated lyrics, source tonal information, and alignment information between the to-be-translated lyrics and the source tonal information are input into a song translation model, so as to obtain target lyrics output by the song translation model and alignment information between the target lyrics and the source tonal information, that is, in the case of obtaining translated lyrics, the translated lyrics are aligned with the source tonal information, so that the alignment information between the lyrics and the tonal information of the target song generated based on the target lyrics and the target alignment information is more accurate.

[0140] Corresponding to the method embodiments described above, the present specification also provides song processing device embodiments, Figure 6 A structural schematic diagram of a song processing device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 6 The device comprises:

[0141] A determination module 602 is configured to determine a to-be-translated song based on a received song translation request;

[0142] An identification module 604 is configured to identify to-be-translated lyrics in the to-be-translated song, source tonal information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tonal information;

[0143] An input module 606 is configured to input the to-be-translated lyrics, the source tonal information, and the source alignment information into a song translation model, wherein the song translation model is used to translate the to-be-translated lyrics into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information;

[0144] An acquisition module 608 is configured to acquire target lyrics output by the song translation model and target alignment information between the target lyrics and the source tonal information;

[0145] A generation module 610 is configured to generate a target song according to the target lyrics, the source tonal information, and the target alignment information.

[0146] Optionally, the song translation model comprises an encoder, a decoder, a pooling embedding layer, and an alignment decoder; the input module 606 is further configured to:

[0147] input the source tonal information and the source alignment information into the pooling embedding layer to obtain a tonal matrix and an alignment matrix;

[0148] input the to-be-translated lyrics, the tonal matrix, and the alignment matrix into the encoder to obtain a lyric encoding matrix, a tonal encoding matrix, and an alignment encoding matrix;

[0149] inputting the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix into the decoder to obtain a target lyric decoding matrix and a target pitch decoding matrix;

[0150] determining target pitch information and target alignment information corresponding to the target pitch information according to the target pitch decoding matrix;

[0151] inputting the target pitch information and the target alignment information into the pooling embedding layer to obtain a target pitch matrix and a target alignment matrix;

[0152] inputting the target lyric decoding matrix, the target alignment information, the target pitch matrix and the target alignment matrix into the alignment decoder to obtain a target lyric and target alignment information between the target lyric and the source pitch information.

[0153] Optionally, the input module 606 is further configured to:

[0154] determining a preset lyric hidden vector corresponding to the target lyric decoding matrix;

[0155] inputting the preset lyric hidden vector, the target alignment information, the pitch matrix and the target alignment matrix into the alignment decoder to obtain a target lyric and target alignment information between the target lyric and the source pitch information.

[0156] Optionally, the input module 606 is further configured to:

[0157] determining target pitch information and target alignment information corresponding to the target pitch information according to the source pitch information and the target pitch decoding matrix.

[0158] Optionally, the source pitch information comprises pitch information and duration information.

[0159] Optionally, the decoder comprises m decoding layers, wherein m is a positive integer greater than or equal to 2.

[0160] The input module 606 is further configured to:

[0161] S61, obtaining a reference pitch matrix, a reference alignment matrix and a reference lyric matrix;

[0162] S62, inputting the reference pitch matrix, the reference alignment matrix, the reference lyric matrix and the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix into the first decoding layer to obtain a target lyric decoding matrix and a target pitch decoding matrix output by the first decoding layer;

[0163] S63, input the target lyric decoding matrix and the target pitch decoding matrix to the e-th decoding layer, to obtain the target lyric decoding matrix and the target pitch decoding matrix output by the e-th decoding layer, wherein 2≤e≤m;

[0164] S64, increase e by 1, and determine whether e is greater than m, if yes, obtain the target lyric decoding matrix and the target pitch decoding matrix, and if no, perform step S63.

[0165] Optionally, the song translation model is obtained based on the following steps:

[0166] Obtain song sample data, sample translation labels and sample alignment labels, wherein the song sample data, sample translation labels and sample alignment labels are generated based on single-language song data and / or bilingual song data;

[0167] Input the song sample data to the song translation model;

[0168] Receive the predicted translation lyrics and the predicted alignment information of the translation lyrics and the pitch information output by the song translation model;

[0169] Calculate a loss value according to the predicted translation lyrics, the predicted alignment information, the sample translation labels and the sample alignment labels;

[0170] Iteratively train the song translation model based on the loss value until a training stop condition is reached.

[0171] Optionally, the song sample data, sample translation labels and sample alignment labels are generated based on single-language song data through the following steps:

[0172] Obtain single-language song data;

[0173] Input the single-language song data to a preset song translation model;

[0174] Obtain translation lyrics and alignment information output by the preset song translation model;

[0175] Take the translation lyrics as song sample data, and take the translation lyrics and the alignment information as sample translation labels and sample alignment labels respectively.

[0176] The song processing device of the specification determines a song to be translated based on a received song translation request through a determination module, identifies song lyrics to be translated in the song to be translated, source tone information corresponding to the song lyrics to be translated, and source alignment information between the song lyrics to be translated and the source tone information through an identification module, inputs the song lyrics to be translated, the source tone information, and the source alignment information to a song translation model through an input module, wherein the song translation model is used to translate the song lyrics to be translated into corresponding target song lyrics and determine target alignment information between the target song lyrics and the source tone information, acquires the target song lyrics and the target alignment information between the target song lyrics and the source tone information output by the song translation model through an acquisition module, and generates a target song according to the target song lyrics, the source tone information, and the target alignment information through a generation module.

[0177] By determining the song to be translated, the song lyrics to be translated, the source tone information, and the alignment information between the song lyrics to be translated and the source tone information in the song to be translated are further determined, thereby facilitating further processing. The determined song lyrics to be translated, source tone information, and alignment information between the song lyrics to be translated and the source tone information are input into the song translation model, thereby acquiring the target song lyrics output by the song translation model and the alignment information between the target song lyrics and the source tone information. That is, in the case of obtaining the translated song lyrics, the translated song lyrics are aligned with the source tone, so that the alignment information between the song lyrics and the tone of the target song generated based on the target song lyrics and the target alignment information is more accurate.

[0178] The above is a schematic scheme of the song processing device of the embodiment. It should be noted that the technical scheme of the song processing device belongs to the same concept as the technical scheme of the song processing method described above, and the details of the technical scheme of the song processing device that are not described in detail can be referred to the description of the technical scheme of the song processing method.

[0179] Figure 7 A structural block diagram of a computing device 700 according to an embodiment of the specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 through a bus 730, and a database 750 is used to save data.

[0180] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 740 can include one or more of any type of network interface (for example, a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, Near Field Communication (NFC).

[0181] In one embodiment of the present specification, the above-described components of the computing device 700, as well as other components not shown in FIG. 7, can be connected to each other by a bus. It should be understood that the computing device structure diagram shown is for the purpose of example only and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art. Figure 7 In one embodiment of the present specification, the above-described components of the computing device 700, as well as other components not shown in FIG. 7, can be connected to each other by a bus. It should be understood that the computing device structure diagram shown is for the purpose of example only and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art. Figure 7 The computing device structure diagram shown is for the purpose of example only and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0182] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 can also be a mobile or stationary server.

[0183] The processor 720 is configured to execute computer-executable instructions to implement the steps of the song processing method. The above is a schematic solution of the computing device according to an embodiment of the present specification. It should be noted that the technical solution of the computing device and the technical solution of the song processing method belong to the same concept, and the details of the technical solution of the computing device not described in detail can be referred to the description of the technical solution of the song processing method.

[0184] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the song processing method.

[0185] The above is a schematic solution of the computer-readable storage medium according to an embodiment of the present specification. It should be noted that the technical solution of the storage medium and the technical solution of the song processing method belong to the same concept, and the details of the technical solution of the storage medium not described in detail can be referred to the description of the technical solution of the song processing method.

[0186] An embodiment of the present specification further provides a computer program, which, when executed in a computer, causes the computer to perform the steps of the song processing method.

[0187] The above is a schematic solution of the computer program according to an embodiment of the present specification. It should be noted that the technical solution of the computer program and the technical solution of the song processing method belong to the same concept, and the details of the technical solution of the computer program not described in detail can be referred to the description of the technical solution of the song processing method.

[0188] Corresponding to the method embodiments, the present specification also provides song processing system embodiments, Figure 8 A structural schematic diagram of a song processing system according to an embodiment of the present specification is shown. As shown in the figure, Figure 8 The song processing system 800 includes an end-side device and a cloud-side device.

[0189] The end-side device 802 is configured to generate a song translation request according to a song to be translated, and send the song translation request to the cloud-side device.

[0190] The cloud measurement device 804 is configured to determine a to-be-translated song based on the received song translation request, identify to-be-translated lyrics in the to-be-translated song, source tonal information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tonal information, input the to-be-translated lyrics, the source tonal information, and the source alignment information into a song translation model, where the song translation model is configured to translate the to-be-translated lyrics into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information, obtain the target lyrics and the target alignment information output by the song translation model, and generate a target song according to the target lyrics, the source tonal information, and the target alignment information, and send the target song to the terminal device.

[0191] The terminal device 802 is further configured to play the target song.

[0192] The song processing system of the present specification, the terminal device is configured to generate a song translation request according to a to-be-translated song and send the song translation request to a cloud measurement device; the cloud measurement device is configured to determine a to-be-translated song based on the received song translation request, identify to-be-translated lyrics in the to-be-translated song, source tonal information corresponding to the to-be-translated lyrics, and source alignment information between the to-be-translated lyrics and the source tonal information, input the to-be-translated lyrics, the source tonal information, and the source alignment information into a song translation model, where the song translation model is configured to translate the to-be-translated lyrics into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information, obtain the target lyrics and the target alignment information output by the song translation model, and generate a target song according to the target lyrics, the source tonal information, and the target alignment information, and send the target song to the terminal device; and the terminal device is further configured to play the target song.

[0193] By determining a to-be-translated song, further determining to-be-translated lyrics in the to-be-translated song, source tonal information, and alignment information between the to-be-translated lyrics and the source tonal information, the to-be-translated song can be further processed; the determined to-be-translated lyrics, source tonal information, and alignment information between the to-be-translated lyrics and the source tonal information are input into a song translation model, the song translation model can translate the to-be-translated lyrics, so that the translation of the lyrics can be realized;

[0194] Further, the target lyrics and the source tone information obtained by translation can be matched based on the source alignment information of the song to be translated to obtain target alignment information; and the target song is generated based on the target lyrics, the source tone information and the target alignment information, so that the translated song with matched lyrics and tone can be obtained, the accurate alignment between the lyrics and the tone is ensured based on the better translation result, and then the target song generated is consistent with the song to be translated in terms of playing effect.

[0195] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.

[0196] The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution package, etc. It should be noted that the computer readable medium can include or exclude certain contents according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunications signals.

[0197] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited to the order of the actions described, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all necessary for the embodiments of the present specification.

[0198] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0199] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A song processing method, comprising: determining a song to be translated based on a received song translation request; identifying lyrics to be translated in the song to be translated, source tone information corresponding to the lyrics to be translated, and source alignment information between the lyrics to be translated and the source tone information; inputting the lyrics to be translated, the source tone information, and the source alignment information into a song translation model, wherein the song translation model is configured to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tone information, the song translation model performs pooling processing on the source tone information and the source alignment information through a pooling embedding layer, and dynamically predicts the target alignment information between the target lyrics and the source tone information in the process of autoregressive translation through an alignment decoder; obtaining the target lyrics and the target alignment information between the target lyrics and the source tone information output by the song translation model; generating a target song according to the target lyrics, the source tone information, and the target alignment information.

2. The method of claim 1, wherein the song translation model comprises an encoder, a decoder, a pooling embedding layer, and an alignment decoder; inputting the lyrics to be translated, the source tone information, and the source alignment information into a song translation model, comprising: inputting the source tone information and the source alignment information into the pooling embedding layer to obtain a tone matrix and an alignment matrix; inputting the lyrics to be translated, the tone matrix, and the alignment matrix into the encoder to obtain a lyrics encoding matrix, a tone encoding matrix, and an alignment encoding matrix; inputting the lyrics encoding matrix, the tone encoding matrix, and the alignment encoding matrix into the decoder to obtain a target lyrics decoding matrix and a target tone decoding matrix; determining target tone information and target alignment information corresponding to the target tone information according to the target tone decoding matrix; inputting the target tone information and the target alignment information into the pooling embedding layer to obtain a target tone matrix and a target alignment matrix; inputting the target lyrics decoding matrix, the target alignment information, the target tone matrix, and the target alignment matrix into the alignment decoder to obtain target lyrics and target alignment information between the target lyrics and the source tone information.

3. The method of claim 2, wherein inputting the target lyrics decoding matrix, the target alignment information, the target tone matrix, and the target alignment matrix into the alignment decoder to obtain target lyrics and target alignment information between the target lyrics and the source tone information comprises: determining a preset lyrics hidden vector corresponding to the target lyrics decoding matrix; inputting the preset lyrics hidden vector, the target alignment information, the tone matrix, and the target alignment matrix into the alignment decoder to obtain target lyrics and target alignment information between the target lyrics and the source tone information.

4. The method of claim 2, wherein determining target tone information and target alignment information corresponding to the target tone information according to the target tone decoding matrix comprises: ​ Determine target pitch information and target alignment information corresponding to the target pitch information according to the source pitch information and the target pitch decoding matrix. 5.The method of claim 1, wherein the source pitch information comprises pitch information and duration information.

6. The method of claim 2, the decoder comprising m decoding layers, wherein, m is a positive integer greater than or equal to 2; Input the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix into the decoder to obtain a target lyric decoding matrix and a target pitch decoding matrix, comprising: S61, obtain a reference pitch matrix, a reference alignment matrix and a reference lyric matrix; S62, input the reference pitch matrix, the reference alignment matrix, the reference lyric matrix, the lyric encoding matrix, the pitch encoding matrix and the alignment encoding matrix into the first decoding layer to obtain a target lyric decoding matrix and a target pitch decoding matrix output by the first decoding layer; S63, input the target lyric decoding matrix and the target pitch decoding matrix and the target lyric decoding matrix and the target pitch decoding matrix output by the e-1th decoding layer into the e th decoding layer to obtain a target lyric decoding matrix and a target pitch decoding matrix output by the e th decoding layer, wherein 2≤e≤m; S64, increase e by 1, and determine whether e is greater than m, if yes, obtain the target lyric decoding matrix and the target pitch decoding matrix, if no, execute step S63. 7.The method of claim 1, wherein the song translation model is obtained by training based on the following steps: acquiring song sample data, sample translation labels, and sample alignment labels, wherein, The song sample data, sample translation label and sample alignment label are generated based on single-language song data and / or bilingual song data; Input the song sample data into the song translation model; Receive the predicted translation lyrics and the predicted alignment information of the translation lyrics and the pitch information output by the song translation model; Calculate a loss value according to the predicted translation lyrics, the predicted alignment information, the sample translation label and the sample alignment label; Iteratively train the song translation model based on the loss value until a training stop condition is reached. 8.The method of claim 7, wherein the song sample data, sample translation label and sample alignment label are generated based on single-language song data by the following steps: Obtain single-language song data; Input the single-language song data into a preset song translation model; Obtain translation lyrics and alignment information output by the preset song translation model; Take the translation lyrics as song sample data, and take the translation lyrics and the alignment information as sample translation label and sample alignment label respectively. 9.A song processing system, comprising an end-side device and a cloud-side device; The end-side device is configured to generate a song translation request according to a song to be translated, and send the song translation request to the cloud-side device; The cloud-side device is configured to determine the song to be translated based on the received song translation request; Identify the lyrics to be translated in the song to be translated, source pitch information corresponding to the lyrics to be translated and source alignment information between the lyrics to be translated and the source pitch information; input the lyrics to be translated, the source tonal information and the source alignment information to a song translation model, wherein the song translation model is configured to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information, the song translation model performs pooling processing on the source tonal information and the source alignment information through a pooling embedding layer, and dynamically predicts the target alignment information between the target lyrics and the source tonal information in the process of autoregressive translation through an alignment decoder; obtain the target lyrics output by the song translation model and the target alignment information between the target lyrics and the source tonal information; and generate a target song according to the target lyrics, the source tonal information and the target alignment information, and send the target song to the terminal device. The terminal device is further configured to play the target song. 10.A song processing apparatus, comprising: a determination module configured to determine a song to be translated based on a received song translation request; an identification module configured to identify lyrics to be translated in the song to be translated, source tonal information corresponding to the lyrics to be translated and source alignment information between the lyrics to be translated and the source tonal information; an input module configured to input the lyrics to be translated, the source tonal information and the source alignment information to a song translation model, wherein the song translation model is configured to translate the lyrics to be translated into corresponding target lyrics and determine target alignment information between the target lyrics and the source tonal information, the song translation model performs pooling processing on the source tonal information and the source alignment information through a pooling embedding layer, and dynamically predicts the target alignment information between the target lyrics and the source tonal information in the process of autoregressive translation through an alignment decoder; an obtaining module configured to obtain the target lyrics output by the song translation model and the target alignment information between the target lyrics and the source tonal information; a generating module configured to generate a target song according to the target lyrics, the source tonal information and the target alignment information. 11.A computing device, comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which implement the steps of the song processing method of any one of claims 1 to 8 when executed by the processor. 12.A computer readable storage medium storing computer executable instructions, which implement the steps of the song processing method of any one of claims 1 to 8 when executed by a processor. 13.A computer program product comprising computer instructions, which implement the steps of the song processing method of any one of claims 1 to 8 when executed by a processor.

Images