Piano accompaniment generation method and piano accompaniment generation model training method

Abstract

The invention discloses a piano accompaniment generation method and a piano accompaniment generation model training method, and belongs to the field of audio processing. The piano accompaniment generation method comprises the following steps: acquiring an audio data sequence corresponding to n sections of a target audio; through a piano accompaniment generation model, predicting piano data corresponding to the ith audio data in the audio data sequence at the ith time step based on a predicted piano data sequence corresponding to the ith sub-audio data sequence and the (i-1) th sub-audio data sequence in the audio data sequence; adding the piano data corresponding to the ith audio data to the predicted piano data sequence to obtain an updated predicted piano data sequence; making i = i + 1, and returning to execute the above steps; and generating a target piano accompaniment corresponding to the target audio based on the updated predicted piano data sequence obtained when i is equal to n. The piano accompaniment is generated in a bar-by-bar autoregression mode, and the generated piano accompaniment is more natural and smoother.

Description

Technical Field

[0001] This application relates to the field of audio processing, and in particular to a method for generating piano accompaniment and a training method for a piano accompaniment generation model. Background Technology

[0002] Piano accompaniment plays an important role in song performance and instrumental arrangement.

[0003] In related technologies, the original audio is decomposed and analyzed based on preset rules to generate a piano accompaniment. However, although the piano accompaniment generated based on preset rules may be completely correct in terms of technology and the harmony may be reasonable, the generated piano accompaniment is often rather stiff, lacking naturalness and expressiveness, and cannot meet the needs of practical applications. Summary of the Invention

[0004] This application provides a method for generating piano accompaniment and a training method for a piano accompaniment generation model. This application generates piano accompaniment through a measure-by-measure autoregression approach, resulting in a more natural and fluent piano accompaniment. The technical solution includes the following:

[0005] According to one aspect of this application, a method for generating piano accompaniment is provided, the method comprising the following: Obtain the audio data sequence corresponding to n sections of the target audio; Using a piano accompaniment generation model, at the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, the piano data corresponding to the i-th audio data is predicted; the initial value of i is one, and i is less than n. Add the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence; Let i = i + 1, and return to execute the piano accompaniment generation model. At the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, predict the piano data corresponding to the i-th audio data in the audio data sequence. Add the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence, until the piano data corresponding to the n-th audio data is obtained. Based on the updated predicted piano data sequence obtained when i=n, ​​the target piano accompaniment corresponding to the target audio is generated.

[0006] According to one aspect of this application, a training method for a piano accompaniment generation model is provided, the method comprising: Obtain the sample audio data sequence corresponding to m sample measures of the sample audio; and obtain the sample piano data sequence corresponding to m sample measures of the sample piano accompaniment, wherein the sample piano accompaniment is the piano accompaniment of the sample audio; Using the piano accompaniment generation model, at the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, the predicted piano data corresponding to the j-th sample measure is predicted; the initial value of j is one, and j is less than n; Let j = j + 1, and return to execute the piano accompaniment generation model. At the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, predict the predicted piano data corresponding to the j-th sample measure. The piano accompaniment generation model is trained based on the error between the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence.

[0007] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method for generating piano accompaniment or the method for training a piano accompaniment generation model.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program, the computer program being loaded and executed by a processor to implement the above-described method for generating piano accompaniment or the method for training a piano accompaniment generation model.

[0009] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned method for generating piano accompaniment or the method for training a piano accompaniment generation model.

[0010] The beneficial effects of the technical solutions provided in this application include at least the following: In this embodiment, an audio data sequence containing n measures is obtained. The piano data corresponding to the i-th sub-audio data sequence (the first i audio data) and the (i-1)-th sub-audio data sequence (the first i-1 audio data) are used to predict the piano data corresponding to the i-th audio data. This application provides a measure-by-measure autoregressive method for generating a target piano accompaniment. This application decomposes the task of predicting the piano data corresponding to the target piano accompaniment into the task of predicting the piano data corresponding to each measure. When predicting the piano data corresponding to each measure, it relies on the measure data (including audio data and piano data) of all preceding measures and the audio data corresponding to the current measure. This ensures that the final generated piano accompaniment for each measure continues the style of the previous measures, resulting in a more unified style and a more natural, smooth, and expressive overall piano accompaniment. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a method for generating piano accompaniment according to an embodiment of this application.

[0013] Figure 2 This is a flowchart of a method for generating piano accompaniment according to an embodiment of this application.

[0014] Figure 3 This is a flowchart of a method for generating piano accompaniment according to an embodiment of this application.

[0015] Figure 4 This is a flowchart of a method for generating piano data corresponding to the i-th section, provided in one embodiment of this application.

[0016] Figure 5 This is a schematic diagram of a method for generating piano data corresponding to the i-th measure provided in one embodiment of this application.

[0017] Figure 6 This is a flowchart of a method for generating piano accompaniment according to an embodiment of this application.

[0018] Figure 7 This is a schematic diagram of a method for generating piano accompaniment according to an embodiment of this application.

[0019] Figure 8This is a flowchart of a method for generating piano accompaniment according to an embodiment of this application.

[0020] Figure 9 This is a schematic diagram of a method for generating piano accompaniment according to an embodiment of this application.

[0021] Figure 10 This is a flowchart of a method for generating piano accompaniment according to an embodiment of this application.

[0022] Figure 11 This is a schematic diagram of a method for generating piano accompaniment according to an embodiment of this application.

[0023] Figure 12 This is a flowchart of a training method for a piano accompaniment generation model provided in one embodiment of this application.

[0024] Figure 13 This is a flowchart of a training method for a piano accompaniment generation model provided in one embodiment of this application.

[0025] Figure 14 This is a structural block diagram of a piano accompaniment generation device provided in one embodiment of this application.

[0026] Figure 15 This is a structural block diagram of a training device for a piano accompaniment generation model provided in one embodiment of this application.

[0027] Figure 16 This is a structural block diagram of a computer device provided in one embodiment of this application.

[0028] Figure 17 This is a structural block diagram of a computer device provided in another embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] It should be understood that "several" in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should be noted that the information (including but not limited to device information, personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the subject or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0032] First, the relevant terms in the embodiments of this application will be introduced.

[0033] Audio data refers to the information contained in the target audio at the data level. A synthesizer (player) can play the target audio based on this data, and audio extraction tools can extract the audio data from it. Optionally, audio data includes chord, pitch, and beat information. For example, tools like Chordino or Madmom can be used to extract chord information from the target audio; Melody Extraction algorithms, such as YourMT3+, can be used to extract the melody information, which includes pitch information; and beat information can be extracted using tools like Allin1 or Madmom.

[0034] Piano data refers to the information contained in the target piano accompaniment at the data level. The target piano accompaniment is the piano accompaniment for the target audio. A synthesizer (player) can play the target piano accompaniment based on the piano data. Furthermore, piano accompaniment extraction tools (such as Transkun, an open-source transcribing tool) can extract the corresponding piano data from the target audio. Optionally, the piano data includes chord information, pitch information, and beat information. Transkun can transcribe the target audio into a high-quality piano accompaniment MIDI file, which includes the piano data of the piano accompaniment.

[0035] Figure 1 A schematic diagram of a method for generating piano accompaniment provided in an exemplary embodiment of this application is shown.

[0036] In this embodiment of the application, audio data corresponding to n measures of the target audio is obtained. The target audio is the audio used to generate the target piano accompaniment. The target audio is divided into n measures. A placeholder element is inserted after the audio data corresponding to each measure to form a first sequence. The placeholder element refers to the element used to occupy the position in the sequence. For example, the placeholder element includes a matrix with empty matrix elements, a matrix with zero matrix elements, etc.

[0037] The piano accompaniment generation model generates piano data for each measure based on the first sequence. The model executes n time steps, predicting piano data for each measure at each time step.

[0038] like Figure 1 As shown, in the first time step, the piano accompaniment generation model 110 will input a first sequence 100, which includes 2n sequence elements. The first sequence 100 includes [audio data corresponding to the first measure, placeholder element 1, audio data corresponding to the second measure, placeholder element 2, ..., audio data corresponding to the nth measure, placeholder element n]. The piano accompaniment generation model 110 will predict the piano data 120 corresponding to the first measure based on the first sequence element in the first sequence 100. Then, replace the second sequence element (i.e., placeholder element 1) in the first sequence 100 with the piano data 120 corresponding to the first measure to obtain the replaced first sequence 100; In the second time step, the piano accompaniment generation model 110 inputs the replaced first sequence 100; based on the first sequence element, the second sequence element and the third sequence element in the first sequence 100, it predicts the piano data 130 corresponding to the second measure; Then, replace the fourth sequence element (i.e., placeholder element 2) in the first sequence 100 with the piano data 130 corresponding to the second measure to obtain the replaced first sequence 100; And so on, executing n time steps; After outputting the piano data corresponding to the nth measure at the nth time step, the final first sequence 100 is obtained; extract the even-numbered sequence elements from the final first sequence 100: the 2nd sequence element (piano data corresponding to the 1st measure), the 4th sequence element (piano data corresponding to the 2nd measure), ..., the 2nth sequence element (piano data corresponding to the nth measure), to form the second sequence 140; The target piano accompaniment is generated based on the second sequence 140, which includes piano data corresponding to n measures in the target piano accompaniment. Therefore, the target piano accompaniment can be generated based on the second sequence 140.

[0039] In this embodiment, by constructing a first sequence of audio data and placeholder elements corresponding to each measure, when predicting the piano data corresponding to the i-th measure at each prediction time step, the system relies on the sequence elements before the 2i-th sequence element in the first sequence (including measure data (including audio data and piano data) of all measures before the i-th measure and the audio data corresponding to the i-th measure). In other words, this application provides a scheme for generating a target piano accompaniment through measure-by-measure autoregression. This embodiment decomposes the prediction task of the piano data corresponding to the target piano accompaniment into a prediction task of the piano data corresponding to each measure. When predicting the piano data corresponding to each measure, the system relies on the measure data (including audio data and piano data) of all preceding measures and the audio data corresponding to the current measure. This ensures that the final generated piano accompaniment for each measure continues the style of the previous measures, resulting in a more unified style and a more natural, smooth, and expressive overall piano accompaniment.

[0040] In one embodiment, the piano accompaniment generation method provided in this application can be widely applied to AI-powered music adaptation scenarios. For example, the function of generating piano accompaniment through AI can enhance the user's music experience. Using the method proposed in this application embodiment, natural, smooth, and detailed piano accompaniment can be automatically generated for various audio creation scenarios. For instance, by retaining the original melody of a song and generating a completely new piano accompaniment, audio can be created, providing users with a fresh listening experience. Alternatively, by combining AI technology, the vocal track in the audio can be automatically converted into the performance style of instruments such as the dizi (Chinese flute) or hulusi (a type of flute), and then paired with the generated piano accompaniment to create a creative and expressive instrumental adaptation. Furthermore, using the method proposed in this application embodiment, a high-quality piano accompaniment track can also be generated separately for users to use as accompaniment for their own singing.

[0041] This application's embodiments can also be applied to intelligent music score scenarios, automatically generating complete and playable piano accompaniment scores, providing technical support for music creation and learning. This application's embodiments comprehensively consider both data quality and generation performance, providing more natural, diverse, and expressive piano accompaniments, and can be widely used in music composition, AI-based adaptation, intelligent music score generation, and music education.

[0042] In one embodiment, the piano accompaniment generation method provided in this application can be executed by a computer device, which includes at least one of a terminal device and a server; optionally, the piano accompaniment generation method is executed by the terminal device, or by the server, or by the terminal device and the server jointly.

[0043] In one embodiment, the device type of the terminal device includes at least one of the following: smartphone, smartwatch, in-vehicle terminal, wearable device, smart TV, tablet computer, e-book reader, MP3 player, MP4 player, laptop computer, and desktop computer.

[0044] In some embodiments, the server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0045] Figure 2 This is a flowchart of a method for generating piano accompaniment provided in an exemplary embodiment of this application, illustrated by an example of the method being executed by a computer device. The method includes: Step 210: Obtain the audio data sequence corresponding to n sections of the target audio; The audio data sequence consists of n subsections, each corresponding to one of the n audio data points.

[0046] n is a positive integer greater than one.

[0047] The target audio refers to the audio used to generate the target piano accompaniment, which is the piano accompaniment for the target audio. Optionally, the target audio can include songs, instrumental music, symphonies, background music, etc. The target audio has n measures, which can be divided into n regularly spaced units.

[0048] In one embodiment, target audio is acquired, and audio data corresponding to n measures is extracted from the target audio using an audio extraction tool. The audio extraction tool automatically divides the audio data into measures, obtaining audio data corresponding to n measures. Optionally, the audio data includes chord, pitch, and beat information.

[0049] For example, tools like Chordino or Madmom can be used to extract chord information for n measures from a target audio file. Another example is using Melody Extraction algorithms, such as YourMT3+, to extract the melody information for n measures, which includes pitch information. Yet another example is using tools like Allin1 or Madmom to extract the beat information for n measures from the target audio file. The chord information, pitch information, and beat information for a single audio measure constitute the audio data for that measure.

[0050] In one embodiment, the target audio is audio obtained from a music library, which is the source of music played by the audio player and contains a number of audio files. In another embodiment, the target audio is audio downloaded from the network; the user can download audio from the network to generate piano accompaniment. In yet another embodiment, the target audio is locally recorded audio, such as audio of a singer performing in a recording studio, audio of a singer performing live, or a user-recorded performance.

[0051] Step 220: Using the piano accompaniment generation model, at the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, predict the piano data corresponding to the i-th audio data in the audio data sequence. The initial value of i is 1, and i is less than n.

[0052] For example, the i-th audio data in the audio data sequence is the audio data corresponding to the i-th measure. The i-th sub-audio data sequence includes the first i audio data in the audio data sequence, which are the audio data of the first i measures. The (i-1)-th sub-audio data sequence includes the first i-1 audio data in the audio data sequence, which are the audio data of the first i-1 measures. The predicted piano data sequence corresponding to the (i-1)-th sub-audio data sequence includes the predicted piano data corresponding to the first i-1 measures.

[0053] Understandably, at the i-th time step, the piano accompaniment generation model will predict the predicted piano data corresponding to the i-th measure based on the audio data corresponding to the first i measures in the audio data sequence and the predicted piano data corresponding to the first i-1 measures. That is, this application provides a scheme for generating the target piano accompaniment in a measure-by-measure autoregressive manner.

[0054] This application breaks down the task of predicting the piano data corresponding to the target piano accompaniment into the task of predicting the piano data corresponding to each measure. When predicting the piano data corresponding to each measure, it will rely on the measure data (including audio data and piano data) of all previous measures and the audio data corresponding to the current measure. This ensures that the piano accompaniment of each measure generated in the end can continue the style of the previous measures. The final generated complete piano accompaniment is more unified in style, and the overall piano accompaniment is more natural, smooth and expressive.

[0055] Step 230: Add the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence; Add the piano data corresponding to the i-th audio data to the predicted piano data sequence. The resulting predicted piano data sequence is the predicted piano data sequence corresponding to the (i+1)-th audio data in the audio data sequence.

[0056] Step 240, let i = i + 1; Let i = i + 1, and return to execute steps 220 and 230 until the piano data corresponding to the nth audio data is obtained.

[0057] Step 250: Based on the updated predicted piano data sequence obtained when i=n, ​​generate the target piano accompaniment corresponding to the target audio.

[0058] Based on the updated predicted piano data sequence obtained when i=n, ​​the target piano accompaniment corresponding to the target audio is generated.

[0059] In one embodiment, the updated predicted piano data sequence obtained when i=n is input into a synthesizer, and the synthesizer produces the target piano accompaniment. The synthesizer includes an audio player.

[0060] based on Figure 2 In the alternative embodiments shown, such as Figure 3 As shown, step 220 includes step 211 before step 220, step 220 includes step 221, step 230 includes step 231, and step 250 includes steps 251 and 252. Step 211: Take the audio data corresponding to each of the n sections of the target audio as a sequence element, and insert a placeholder element after the audio data corresponding to each section to form the first sequence; In this embodiment of the application, the audio data corresponding to each of the n sections is taken as a sequence element, and a placeholder element is inserted after the audio data corresponding to each section. The placeholder element is used to occupy a sequence position to form a first sequence, which includes 2n sequence elements.

[0061] For example, the first sequence is an alternating sequence of M / X, such as M(1), X(1), M(2), X(2), ..., M(n), X(n), where M is the audio data corresponding to a measure and X is a placeholder element.

[0062] In one embodiment, a first switching symbol is inserted before the audio data corresponding to each measure, and a second switching symbol is inserted before each placeholder element. By inserting the first and second switching symbols, the piano accompaniment generation model can accurately distinguish each sequence element from the first sequence, avoiding the mistaken identification of the piano data corresponding to a measure as the content of a placeholder element, and the mistaken identification of the content of a placeholder element as the piano data corresponding to a measure.

[0063] For example, if the first switching symbol is represented by [TRACK M] and the second switching symbol is represented by [TRACK X], then the first sequence is represented as [TRACK M], M(1), [TRACK X], X(1)..., [TRACK M], M(n), [TRACK X], X(n). When the piano accompaniment generation model divides the sequence elements, it confirms that a sequence has been obtained each time it encounters the [TRACK] symbol.

[0064] In one embodiment, an end marker is inserted after the last placeholder element. When the piano accompaniment generation model reads the end marker, it stops predicting. The end marker is such as the EOS marker.

[0065] In one embodiment, after forming the first sequence, at least one of the following operations is performed on the first sequence: embedding, positional encoding, and dropout, to obtain an updated first sequence. Embedding increases the channel dimension of each sequence element in the first sequence, mapping discrete sequence element data to a continuous vector space. Through embedding, each sequence element in the first sequence can be represented as a high-dimensional vector. Positional encoding adds positional information to each sequence element, enabling the piano accompaniment generation model to understand the position of each sequence element in the first sequence. Optionally, positional encoding includes absolute positional encoding and relative positional encoding. In absolute positional encoding, each sequence element has a unique code for its position. In relative positional encoding, the position of each sequence element can be modeled through its positional relationship with other sequence elements. Dropout is a regularization technique that prevents overfitting by randomly shutting down a portion of neurons during training, thereby improving the model's generalization ability.

[0066] Step 221: Using the piano accompaniment generation model, at the i-th time step, based on the sequence elements before the 2i-th sequence element in the first sequence, predict the piano data corresponding to the i-th measure. The initial value of i is 1, and i is less than n; The piano accompaniment generation model will perform prediction operations at n time steps. In the i-th time step, the piano accompaniment generation model will predict the piano data corresponding to the i-th measure based on the sequence elements before the 2i-th sequence element in the first sequence.

[0067] Step 231: Replace the 2i-th sequence element with the piano data corresponding to the i-th measure; Replace the 2i-th sequence element with the piano data corresponding to the i-th measure.

[0068] Indicatively, in the first time step, the piano accompaniment generation model predicts the piano data corresponding to the first measure based on the first sequence element (the audio data corresponding to the first measure); and replaces the second sequence element with the piano data corresponding to the first measure to form the replaced first sequence. In the second time step, the piano accompaniment generation model will predict the piano data corresponding to the second measure based on the first to third sequence elements (including the audio data corresponding to the first measure, the piano data corresponding to the first measure, and the audio data corresponding to the second measure); and replace the fourth sequence element with the piano data corresponding to the second measure to form the first sequence after replacement. Continue in this manner until all blank elements are replaced with piano data.

[0069] Step 251: Extract even-numbered sequence elements from the first sequence when i=n to form the second sequence; After the above prediction and replacement operations, the final first sequence consists of 2n sequence elements, where the even-numbered sequence elements include the piano data corresponding to the 1st measure, the 2nd measure, ..., the nth measure. The even-numbered sequence elements extracted from the first sequence will form the second sequence, which includes the piano data corresponding to n measures.

[0070] Step 252: Based on the second sequence, generate the target piano accompaniment corresponding to the target audio.

[0071] In one embodiment, piano data from a second sequence is input into a synthesizer to obtain a target piano accompaniment. The synthesizer includes an audio player.

[0072] In summary, in this embodiment, by constructing a first sequence of audio data and placeholder elements corresponding to each measure, the prediction of the piano data corresponding to the i-th measure at each prediction time step will rely on the sequence elements before the 2i-th sequence element in the first sequence (including the measure data (including audio data and piano data) of all measures before the i-th measure and the audio data corresponding to the i-th measure). That is, this application provides a scheme for generating a target piano accompaniment through autoregression. This embodiment decomposes the prediction task of the piano data corresponding to the target piano accompaniment into the prediction task of the piano data corresponding to each measure. When predicting the piano data corresponding to each measure, it will rely on the measure data (including audio data and piano data) of all preceding measures and the audio data corresponding to the current measure. This ensures that the piano accompaniment of each measure generated can continue the style of the previous measures, and the final generated complete piano accompaniment is more stylistically unified, and the overall piano accompaniment is more natural, smooth, and expressive.

[0073] In addition, the piano data and audio data corresponding to each measure are cross-distributed in the first sequence. The piano data corresponding to each measure will follow the audio data corresponding to the current measure. This ensures that when predicting the piano data corresponding to the i-th measure, the style of the audio data corresponding to the i-th measure will be continued. As a result, the final generated complete piano accompaniment will be more consistent with the audio style of the target audio.

[0074] In one embodiment, the audio data corresponding to the i-th section of the target audio includes at least one of the following: 1. The bar number of the i-th bar in the target audio; 2. The subbeat number of the i-th measure in the target audio. 3. The speed (tempo) of the target audio; 4. The notes played in the i-th measure of the target audio. 5. The duration of the notes played in the i-th measure of the target audio. 6. The velocity of the subbeat notes in the i-th measure of the target audio. 7. Chords of the target audio.

[0075] To illustrate, the audio data examples for the first and second sections are given below.

[0076] [BAR 1] / / Section 1 [TEMPO 100] [CHORD G:maj] [SUBBEAT 0] [PITCH G4][DURATION 4] [velocity 60] [SUBBEAT 1] [PITCH A4][DURATION 4] [velocity 60] [SUBBEAT 2] [PITCH B4][DURATION 4] [velocity 60] [SUBBEAT 3] [PITCH D5][DURATION 4] [velocity 60] [BAR 2] / / Section 2 [TEMPO 100] [CHORD G:maj] [SUBBEAT 0] [PITCH E5][DURATION 4] [velocity 60] [SUBBEAT 1] [PITCH D5][DURATION 4] [velocity 60] [SUBBEAT 2] [PITCH B4][DURATION 4] [velocity 60] [SUBBEAT 3] [PITCH G4][DURATION 4] [velocity 60] As can be seen, the audio data corresponding to the first measure includes a target audio tempo of 100, a target audio chord of G:maj, a note played in the first subbeat of G4 with a duration of 4 and a velocity of 60; a note played in the second subbeat of A4 with a duration of 4 and a velocity of 60; a note played in the third subbeat of B4 with a duration of 4 and a velocity of 60; and a note played in the fourth subbeat of D5 with a duration of 4 and a velocity of 60. The audio data corresponding to the second section includes a target audio tempo of 100, a target audio chord of G:maj, and the following: the first subbeat has a pitch of E5, a duration of 4, and a velocity of 60; the second subbeat has a pitch of D5, a duration of 4, and a velocity of 60; the third subbeat has a pitch of B4, a duration of 4, and a velocity of 60; and the fourth subbeat has a pitch of G4, a duration of 4, and a velocity of 60.

[0077] In another embodiment, the piano data corresponding to the audio data of the i-th measure in the target audio includes at least one of the following: 1. The measure number of the i-th measure in the target piano accompaniment; 2. The subbeat number of the subbeat contained in the i-th measure of the target piano accompaniment; 3. The target tempo of the piano accompaniment; 4. The notes played in the i-th measure of the target piano accompaniment; 5. The duration of the notes played in the i-th measure of the target piano accompaniment; 6. The intensity of the subbeat notes contained in the i-th measure of the target piano accompaniment; 7. Chords for the target piano accompaniment.

[0078] Piano data is similar to audio data, both containing the above-mentioned types of information, which will not be elaborated on here.

[0079] based on Figure 3 In the optional embodiment shown, the piano accompaniment generation model includes a masked self-attention network and an output network; step 221, "using the piano accompaniment generation model, at the i-th time step, based on the sequence elements before the 2i-th sequence element in the first sequence, predicts the piano data corresponding to the i-th measure," includes, for example: Figure 4 Steps 410 and 420 are shown.

[0080] Step 410: In the i-th time step, the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the first sequence are masked by the masking self-attention network to form a masked sequence. Self-attention is calculated on the masked sequence to obtain the i-th feature. A masking self-attention network is a neural network used to perform masking and self-attention operations. For the i-th time step, the masking self-attention network masks the 2i-th sequence element and all subsequent sequence elements in the first sequence to obtain a masked sequence (i.e., the masking operation). Then, self-attention is calculated on the masked sequence to obtain the i-th feature. Optionally, the masking self-attention network sets the 2i-th sequence element and all subsequent sequence elements in the first sequence to 0 to achieve the masking effect.

[0081] Step 420: The piano data corresponding to the i-th measure is obtained by predicting the i-th feature through the output network.

[0082] The output network is the result layer network, used to predict piano data based on features. For example, the output network includes cascaded linear layers and activation function layers (such as softmax layers). The i-th feature is passed through the cascaded linear and activation function layers to obtain the probability that the piano data corresponding to the i-th measure belongs to each class of piano data. The piano data with the highest probability is determined as the piano data corresponding to the i-th measure. Alternatively, the output network can be a regression head. A regression head is a network structure used for regression tasks. The goal of regression tasks is to predict continuous values, rather than discrete class labels. Regression heads typically add a dense layer without a softmax activation function to output continuous values.

[0083] The following example uses the first time step of the piano accompaniment generation model. The first time step is used to predict the piano data corresponding to the first measure.

[0084] Indicative, for reference only Figure 5 The piano accompaniment generation model includes a masked self-attention network 510 and an output network 520. Figure 5 The execution process of the first time step is shown. The masked self-attention network 510 will receive a first sequence 500, which includes sequence elements 1 to 2n.

[0085] The masking self-attention network 510 masks all sequence elements except the first sequence element, resulting in a masked sequence 530. Optionally, the masking self-attention network 510 sets all sequence elements except the first sequence element to 0. Then, the masking self-attention network 510 performs self-attention operation on the masked sequence 530 to obtain a first feature 540. The output network 520 obtains the first feature 540 and predicts the piano data corresponding to the first section based on the first feature 540.

[0086] In one embodiment, the masking self-attention network comprises p cascaded masking self-attention networks, where p is a positive integer greater than one. Step 410 includes, for example,... Figure 6 Steps 411 to 413 are shown.

[0087] Step 411: In the i-th time step, the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the first sequence are masked by the first masking self-attention network to obtain the first masked sequence. Self-attention operation is performed on the first masked sequence to obtain the first feature sequence. In this embodiment, for the i-th time step, which is the time step used to predict the piano data corresponding to the i-th measure, the first sequence will be masked and self-attentioned through p stacked masking self-attention networks to obtain the i-th feature. The i-th feature will be passed to the output network, and the output network will predict the piano data corresponding to the i-th measure based on the i-th feature.

[0088] For the first masking self-attention network among p masking self-attention networks, the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the first sequence are masked to obtain the first masking sequence; then, self-attention operation is performed on the first masking sequence to obtain the first feature sequence.

[0089] Step 412: Merge the first sequence and the q-th feature sequence to obtain the q-th intermediate sequence; Since there are p masked self-attention networks, p masked self-attention operations will be performed. For the first masked self-attention network, its input is the first sequence and its output is the first feature sequence; while for the other p-1 masked self-attention networks, the input of each masked self-attention network is the intermediate sequence obtained by fusing the first sequence and the feature sequence output by the previous masked self-attention network.

[0090] The initial value of q is 1, and the maximum value of q is p.

[0091] In one embodiment, the sequence elements of the same order in the first sequence and the q-th feature sequence are weighted and summed to obtain the sequence element of the corresponding order in the q-th intermediate sequence. Optionally, the weighting weight is a hyperparameter. For example, the 5th sequence element in the first sequence and the 5th sequence element in the q-th feature sequence are directly added together to obtain the 5th sequence element of the q-th intermediate sequence.

[0092] Step 413: Mask the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the q-th intermediate sequence through the q+1-th masking self-attention network to obtain the q+1-th masking sequence. Perform self-attention operation on the q+1-th masking sequence to obtain the q+1-th feature sequence. When the value of q is p-1, the q+1-th feature sequence is the i-th feature.

[0093] For the (q+1)th masked self-attention network among the p masked self-attention networks, the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the q-th intermediate sequence are masked to obtain the (q+1)th masked sequence; then, self-attention operation is performed on the (q+1)th masked sequence to obtain the (q+1)th feature sequence.

[0094] When the value of q reaches p, the (q+1)th feature sequence is the i-th feature.

[0095] The processes of steps 411, 412, and 413 above can be achieved through... Figure 6 Example.

[0096] like Figure 7 As shown, in the i-th time step, the masking self-attention network 1 will acquire the first sequence 700, perform the masking self-attention operation on the first sequence 700, and obtain the first feature sequence 701; The first sequence 700 and the first feature sequence 701 will be merged to obtain the first intermediate sequence 702; The masked self-attention network 2 will obtain the first intermediate sequence 702, and perform a masked self-attention operation on the first intermediate sequence 702 to obtain the second feature sequence 703; The first sequence 700 and the second feature sequence 703 will be merged to obtain the second intermediate sequence 704; The masked self-attention network 3 will obtain the second intermediate sequence 704, and perform a masked self-attention operation on the second intermediate sequence 704 to obtain the third feature sequence 705; Similarly, the first sequence 700 and the (p-1)th feature sequence will be fused to obtain the (p-1)th intermediate sequence 706. The masking self-attention network p will obtain the (p-1)th intermediate sequence 706 and perform a masking self-attention operation on the (p-1)th intermediate sequence 706 to obtain the i-th feature 707 (i.e. the p-th feature sequence).

[0097] In summary, the above embodiments introduce a method for extracting the i-th feature by cascading multiple masking self-attention networks. In this case, the input of the first masking self-attention network is the first sequence, while the input of the other masking self-attention networks is an intermediate sequence obtained by fusing the first sequence and the feature sequence output by the previous masking self-attention network. By injecting the first sequence into each masking self-attention network through this residual fusion method, the gradient problem caused by the deepness of the network can be solved.

[0098] Optionally, each of the p masked self-attention networks can be replaced with any equivalent autoregressive structure, such as the standard Transformer Decoder Layer, an efficient Transformer network (such as Performer, Linformer), a convolutional autoregressive module, or other autoregressive networks.

[0099] based on Figure 6 In the alternative embodiment shown, each of the p masked self-attention networks includes a cascaded masked self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer.

[0100] Step 411 includes, for example: Figure 8 Steps 4111 to 4114 are shown.

[0101] Step 4111: Through the masking self-attention layer in the first masking self-attention network, the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the first sequence are masked to obtain the first masked sequence. Self-attention operation is performed on the first masked sequence to obtain the first feature sequence. Optionally, the masking self-attention layer sets the sequence elements after the 2ith sequence element in the first sequence to 0 to achieve the masking effect.

[0102] Optionally, during the self-attention operation on the first masking sequence, the masking self-attention layer calculates the correlation score between any two sequence elements and the correlation score between each sequence element and itself for the 2n sequence elements in the first masking sequence. For a sequence element, an attention score (attention weight) is obtained based on its 2n correlation scores. The attention score is then used to weight itself. The first feature sequence is obtained through the above steps.

[0103] Self-attention improves the global dependencies in the first masking sequence. No matter how far apart two sequence elements are in the first masking sequence, self-attention can capture their correlation, thereby constructing a higher quality feature representation and achieving better prediction results.

[0104] Step 4112: Map the first feature sequence to the first feedforward feature sequence through the first feedforward network layer in the masked self-attention network; A feedforward neural layer is a fundamental architecture for artificial intelligence neural networks. It consists of multiple neurons distributed across the input, hidden, and output layers. Data flows unidirectionally within the feedforward layer, from the input layer through the hidden layers to the output layer, without feedback loops. Each neuron receives the output from the previous layer, processes it through a weighted sum and activation function, and then passes the result to the next layer. Activation functions (such as ReLU and Sigmoid) introduce nonlinear characteristics, enabling the feedforward layer to learn complex nonlinear mappings.

[0105] Optionally, the feedforward network layer is a fully connected layer. By masking the feedforward network layer in the first self-attention network, the features of the first feature sequence can be further captured, mapping the first feature sequence into a higher-quality feature sequence and improving the feature representation capability.

[0106] Step 4113: The first sequence is combined with the first feedforward feature sequence through the residual connection layer in the first masked self-attention network to obtain the first residual feature sequence; When the piano accompaniment generation model includes a large number of masking self-attention networks (i.e., when the value of p is large), the piano accompaniment generation model is prone to degradation. That is, as the number of networks increases, the model performance may actually decrease. In this embodiment, a residual connection layer is introduced into the first masking self-attention network to combine the first sequence with the first feedforward feature sequence to obtain the first residual feature sequence. This embodiment can alleviate the performance degradation caused by a large number of masking self-attention network layers through this skip connection method.

[0107] Step 4114: Normalize the first residual feature sequence through the normalization layer in the first masked self-attention network to obtain the updated first feature sequence.

[0108] In this embodiment, by performing a normalization operation on the first residual feature sequence to obtain the updated first feature sequence, the convergence of the model can be accelerated, and some features can be prevented from dominating model learning due to their large values. Furthermore, the normalization operation can also reduce the problems of gradient vanishing or gradient exploding.

[0109] The process of steps 4111 to 4114 above can be achieved through... Figure 9 Example.

[0110] The first masked self-attention network includes a masked self-attention layer 910, a feedforward network layer 920, a residual connection layer 930, and a normalization layer 940.

[0111] The first masking self-attention network's masking self-attention layer 910 obtains the first sequence 900, performs a masking operation on the first sequence 900 to obtain the first masking sequence 901; then performs a self-attention operation on the first masking sequence 901 to obtain the first feature sequence 902; The first masked feedforward network layer 920 of the attention network obtains the first feature sequence 902 and maps the first feature sequence to the first feedforward feature sequence 903. The first residual connection layer 930 of the masked self-attention network obtains the first feedforward feature sequence 903, and combines the first sequence 900 with the first feedforward feature sequence 903 to obtain the first residual feature sequence 904. The first masking normalization layer 940 of the self-attention network obtains the first residual feature sequence 904, and performs a normalization operation on the first residual feature sequence 904 to obtain the updated first feature sequence 905.

[0112] In summary, the above embodiments introduce a first masked self-attention network comprising a cascaded masked self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer. Through the setting and connection of each network layer, the first sequence can be processed better, thereby extracting a first feature sequence of higher quality.

[0113] Optionally, each masked self-attention network may be followed by at least one of a cross-network attention layer and a convolutional layer after the normalization layer. For the first masked self-attention network, the cross-network attention layer is used to further perform attention operations on the first feature sequence obtained in the current masked self-attention network and the q-th feature sequence obtained in the other q-th masked self-attention network, and the result is used as the new first feature sequence generated by the current masked self-attention network. The convolutional layer is used to further extract features through convolution operations to improve feature quality.

[0114] Step 413 includes, for example: Figure 10 Steps 4131 to 4134 are shown.

[0115] Step 4131: Through the masking self-attention layer in the (q+1)th masking self-attention network, the 2ith sequence element and the sequence elements after the 2ith sequence element in the qth intermediate sequence are masked to obtain the (q+1)th masked sequence. Self-attention operation is performed on the (q+1)th masked sequence to obtain the (q+1)th feature sequence. Optionally, the masking self-attention layer sets the 2ith sequence element of the qth intermediate sequence and the sequence elements after the 2ith sequence element to 0 to achieve the masking effect.

[0116] Optionally, during the self-attention operation on the (q+1)th masking sequence, the masking self-attention layer calculates the correlation score between any two sequence elements and the correlation score between each sequence element and itself for the 2n sequence elements in the (q+1)th masking sequence. For a sequence element, an attention score (attention weight) is obtained based on its 2n correlation scores. The attention score is then used to weight itself. The (q+1)th feature sequence is obtained through the above steps.

[0117] Self-attention improves the global dependencies in the (q+1)th masked sequence. No matter how far apart two sequence elements are in the (q+1)th masked sequence, self-attention can capture their correlation, thereby constructing a higher quality feature representation and achieving better prediction results.

[0118] Step 4132: Map the (q+1)th feature sequence to the (q+1)th feedforward feature sequence by masking the feedforward network layer in the attention network. Optionally, the feedforward network layer is a fully connected layer. By masking the feedforward network layer in the attention network at the (q+1)th layer, the features of the (q+1)th feature sequence can be further captured, mapping the (q+1)th feature sequence to a higher quality feature sequence and improving the feature representation capability.

[0119] Step 4133: By using the (q+1)th masked residual connection layer in the attention network, the qth intermediate sequence is combined with the (q+1)th feedforward feature sequence to obtain the (q+1)th residual feature sequence. When the piano accompaniment generation model includes a large number of masking self-attention networks (i.e., when the value of p is large), the piano accompaniment generation model is prone to degradation. That is, as the number of networks increases, the model performance may actually decrease. In this embodiment, a residual connection layer is introduced into the (q+1)th masking self-attention network to combine the qth intermediate sequence with the (q+1)th feedforward feature sequence to obtain the (q+1)th residual feature sequence. This embodiment can alleviate the performance degradation caused by a large number of masking self-attention network layers by using skip connections.

[0120] Step 4134: Normalize the (q+1)th residual feature sequence through the normalization layer in the (q+1)th masked self-attention network to obtain the updated (q+1)th feature sequence.

[0121] In this embodiment, by performing a normalization operation on the (q+1)th residual feature sequence, an updated (q+1)th feature sequence is obtained. This can accelerate the convergence of the model, prevent certain features from dominating model learning due to their large values, and reduce the problems of gradient vanishing or gradient exploding.

[0122] The process of steps 4131 to 4134 above can be achieved through... Figure 11 Example.

[0123] The (q+1)th masked self-attention network includes a masked self-attention layer 1110, a feedforward network layer 1120, a residual connection layer 1130, and a normalization layer 1140.

[0124] The masking self-attention layer 1110 of the (q+1)th masking self-attention network obtains the qth intermediate sequence 1100, performs a masking operation on the qth intermediate sequence 1100 to obtain the (q+1)th masking sequence 1101; then performs self-attention operation on the (q+1)th masking sequence 1101 to obtain the (q+1)th feature sequence 1102. The (q+1)th masked feedforward network layer 1120 of the attention network obtains the (q+1)th feature sequence 1102 and maps the (q+1)th feature sequence 1102 to the (q+1)th feedforward feature sequence 1103. The (q+1)th masked residual connection layer 1130 of the attention network obtains the (q+1)th feedforward feature sequence 1103, and combines the (q)th intermediate sequence 1100 with the (q+1)th feedforward feature sequence 1103 to obtain the (q+1)th residual feature sequence 1104. The (q+1)th masking self-attention network normalization layer 1140 obtains the (q+1)th residual feature sequence 1104, and performs a normalization operation on the (q+1)th residual feature sequence 1104 to obtain the updated (q+1)th feature sequence 1105.

[0125] In summary, the above embodiments introduce a masked self-attention network comprising a cascaded masked self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer. Through the setting and connection of each network layer, the q-th intermediate sequence can be processed better, thereby extracting a higher quality q+1-th feature sequence.

[0126] Optionally, each masked self-attention network may be followed by at least one of a cross-network attention layer and a convolutional layer after the normalization layer. For the (q+1)th masked self-attention network, the cross-network attention layer is used to further perform attention operations on the (q+1)th feature sequence obtained in the current masked self-attention network and the feature sequences obtained from other masked self-attention networks, and the result is used as the new (q+1)th feature sequence generated by the current masked self-attention network. The convolutional layer is used to further extract features through convolution operations to improve feature quality.

[0127] The training method for the piano accompaniment generation model will be introduced next.

[0128] Figure 12 A flowchart illustrating a training method for a piano accompaniment generation model provided in an exemplary embodiment of this application is shown. The method is illustrated by way of example, performed by a computer device, which may include at least one of a server and a terminal device. The method includes: Step 1210: Obtain the sample audio data sequence corresponding to m sample sections of the sample audio; The sample audio data sequence consists of m sample segments, each corresponding to one of the m audio data segments.

[0129] m is a positive integer greater than one.

[0130] The sample audio is used to train the piano accompaniment generation model. The sample audio has m measures, which can be divided into m regularly ordered units by dividing the audio into measures.

[0131] In one embodiment, sample audio is acquired, and an audio extraction tool is used to extract audio data corresponding to m measures from the sample audio. The audio extraction tool automatically divides the audio data into measures, obtaining audio data corresponding to m measures. Optionally, the audio data includes chord, pitch, and beat information.

[0132] For example, tools like Chordino or Madmom can be used to extract chord information for m measures from sample audio. Another example is the Melody Extraction algorithm, such as YourMT3+, which extracts the melody information for m measures from sample audio, including pitch information. Yet another example is using tools like Allin1 or Madmom to extract beat information for m measures from sample audio. The chord information, pitch information, and beat information of an audio measure constitute the audio data for that measure.

[0133] For the audio data of the i-th section of the sample audio, please refer to the audio data of the i-th section of the target audio mentioned above, which will not be repeated here.

[0134] Step 1220: Obtain the sample piano data sequence corresponding to m sample measures of the sample piano accompaniment. The sample piano accompaniment is the piano accompaniment of the sample audio. In one embodiment, a piano accompaniment file (such as a MIDI file) is extracted from the sample audio using a piano accompaniment extraction tool (such as Transkun). The piano accompaniment file includes piano data corresponding to m sample measures. When generating the piano accompaniment file, the piano accompaniment extraction tool divides the file by measures, generating a piano accompaniment file that includes piano data corresponding to the m sample measures. The piano accompaniment file corresponds to the sample piano accompaniment.

[0135] Step 1230: Using the piano accompaniment generation model, at the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, predict the predicted piano data corresponding to the j-th sample measure. The initial value of j is 1, and j is less than n.

[0136] For example, the j-th audio data in the sample audio data sequence is the audio data corresponding to the j-th sample measure. The j-th sample sub-audio data sequence includes the first j audio data in the sample audio data sequence, where the first j audio data corresponds to the first j sample measures. The j-th piano data in the sample piano data sequence is the piano data corresponding to the j-th sample measure. The (j-1)-th sample sub-piano data sequence includes the first (j-1) piano data in the sample piano data sequence, where the first (j-1) piano data corresponds to the first j sample measures.

[0137] Understandably, at the j-th time step, the piano accompaniment generation model will predict the piano data corresponding to the j-th sample measure based on the audio data corresponding to the first j sample measures and the piano data corresponding to the first j-1 sample measures in the sample audio data sequence.

[0138] Step 1240, let j = j + 1; Let j = j + 1, and return to execute steps 1230 until the value of j is m.

[0139] Step 1250: Based on the error between the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence, train the piano accompaniment generation model.

[0140] By performing m prediction operations, m predicted piano data points will be obtained, and each of the m predicted piano data points corresponds one-to-one with a set of m sample measures. In one embodiment, based on the m predicted piano data points and the corresponding piano data points for the m sample measures, the conditional probability of each predicted piano data point is calculated. Calculate the negative log-likelihood loss based on m conditional probabilities; A piano accompaniment generation model was trained using negative log-likelihood loss.

[0141] Schematic representation: The conditional probability for each predicted piano data point is expressed by the following formula: ; Where T represents the total number of predicted piano data (i.e., T is equivalent to m). This represents the probability that the predicted piano data for the t-th sample measure is correct, given that the actual piano data for the t-1 sample measures are known.

[0142] Schematic, the negative log-likelihood loss is expressed by the following formula: ; in, This represents the negative log-likelihood loss. Let t represent the logarithm of the conditional probability corresponding to the t-th predicted piano data. The negative log-likelihood loss is obtained by accumulating the values ​​of T logarithms and then taking the inverse of the logarithm.

[0143] Based on negative log-likelihood loss, the parameters of the piano accompaniment generation model are adjusted through the error backpropagation algorithm to achieve training of the piano accompaniment generation model.

[0144] based on Figure 12 In the alternative embodiments shown, such as Figure 13 As shown, step 1230 includes step 1221 before step 1230, and step 1230 includes step 1231; Step 1221: Insert the piano data corresponding to the same sample measure after the audio data corresponding to each of the m sample measures to form the third sequence; In this embodiment of the application, the audio data corresponding to each sample measure in the m sample measures is taken as a sequence element, and a piano data corresponding to the same sample measure is inserted after the audio data corresponding to each sample measure as a sequence element to form a third sequence.

[0145] The third sequence consists of 2m sequence elements.

[0146] For example, the third sequence is an alternating sequence of M / X, such as M(1), X(1), M(2), X(2), ..., M(m), X(m), where M is the audio data corresponding to a sample measure and X is the piano data corresponding to a sample measure.

[0147] In one embodiment, a first switching symbol is inserted before the audio data corresponding to each sample measure, and a second switching symbol is inserted before the piano data corresponding to each sample measure. By inserting the first and second switching symbols, the piano accompaniment generation model can accurately distinguish each sequence element from the third sequence, avoiding the mistaken identification of the piano data corresponding to a sample measure as the content of the piano data, and the mistaken identification of the content of the piano data as the piano data corresponding to a sample measure.

[0148] In one embodiment, an end marker is inserted after the piano data corresponding to the last sample measure. When the piano accompaniment generation model reads the end marker, it stops predicting.

[0149] In one embodiment, after forming the third sequence, at least one of the following operations is performed on the third sequence: embedding, positional encoding, and dropout, to obtain an updated third sequence. Embedding increases the channel dimension of each sequence element in the third sequence, mapping discrete sequence element data to a continuous vector space. Through embedding, each sequence element in the third sequence can be represented as a high-dimensional vector. Positional encoding adds positional information to each sequence element, enabling the piano accompaniment generation model to understand the position of each sequence element in the third sequence. Optionally, positional encoding includes absolute positional encoding and relative positional encoding. In absolute positional encoding, each sequence element has a unique code for its position. In relative positional encoding, the position of each sequence element can be modeled through its positional relationship with other sequence elements. Dropout is a regularization technique that prevents overfitting by randomly shutting down a portion of neurons during training, thereby improving the model's generalization ability.

[0150] Step 1231: Using the piano accompaniment generation model, at the j-th time step, based on the first sequence element to the (2j-1)th sequence element in the third sequence, predict the predicted piano data corresponding to the j-th sample measure.

[0151] Where m is a positive integer, and j is a positive integer from 1 to m; The piano accompaniment generation model will perform m predictions to obtain m predicted piano data corresponding to m sample measures. The m predicted piano data correspond one-to-one with the m predictions.

[0152] For the j-th prediction, the piano accompaniment generation model will predict the piano data corresponding to the j-th sample measure based on the measure data (including audio data and piano data, if any) of all sample measures before the j-th sample measure and the audio data corresponding to the j-th sample measure.

[0153] For example, for the first prediction, the piano accompaniment generation model will predict the piano data corresponding to the first sample measure based on the first sequence element (the audio data corresponding to the first sample measure). For the second prediction, the piano accompaniment generation model will predict the piano data corresponding to the second sample measure based on the first to third sequence elements (including the audio data corresponding to the first sample measure, the piano data corresponding to the first sample measure, and the audio data corresponding to the second sample measure). This process continues until m predictions are completed.

[0154] It is important to note that the piano data from the previous sample measures used in each prediction are real piano data, not the piano data obtained from the previous prediction. This avoids accumulating the incorrect results of the previous prediction into the current prediction process, and each prediction process is independent of the others.

[0155] In summary, this application embodiment first constructs a third sequence, in which the audio data and piano data of sample measures are cross-distributed. When predicting the predicted piano data corresponding to the j-th sample measure, it will depend on the first sequence element to the (2j-1)-th sequence element in the third sequence. This application embodiment will train the piano accompaniment generation model based on the error between the m predicted piano data obtained from m predictions and the real piano data corresponding to the m sample measures. That is, this application embodiment provides a scheme to predict piano data measure by measure on the third sequence and train the piano accompaniment generation model based on the accumulated error of the piano data of the m measures. The piano accompaniment generation model can learn the dependency between the m measures, thereby improving the quality of the piano accompaniment generated by the piano accompaniment generation model. The final generated piano accompaniment is more natural, smooth and expressive.

[0156] In one embodiment, the piano accompaniment generation model includes a masked self-attention network and an output network.

[0157] Step 1231 includes steps 12311 and 12312.

[0158] Step 12311: Using a masking self-attention network, mask the 2jth sequence element and the sequence elements after the 2jth sequence element in the third sequence to form a sample masking sequence. Perform self-attention calculation on the sample masking sequence to obtain the features of the jth sample. For an explanation of what the masking self-attention network does for the third sequence, please refer to the explanation of what the masking self-attention network does for the first sequence in step 410 above.

[0159] In one embodiment, the masking self-attention network comprises p cascaded masking self-attention networks, where p is a positive integer greater than one. Step 12311 includes steps 1 to 3; Step 1: Mask the 2jth sequence element and the sequence elements after the 2jth sequence element in the third sequence through the first masking self-attention network to obtain the first sample masking sequence. Perform self-attention operation on the first sample masking sequence to obtain the first sample feature sequence. The first masking self-attention network comprises a cascaded masking self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer. In one embodiment, the masking self-attention layer in the first masking self-attention network masks the 2j-th sequence element and all subsequent sequence elements in the third sequence to obtain a first sample masking sequence. Self-attention is then performed on the first sample masking sequence to obtain a first sample feature sequence. The feedforward network layer in the first masking self-attention network maps the first sample feature sequence to a first sample feedforward feature sequence. The residual connection layer in the first masking self-attention network combines the third sequence with the first sample feedforward feature sequence to obtain a first sample residual feature sequence. The normalization layer in the first masking self-attention network performs a normalization operation on the first sample residual feature sequence to obtain an updated first sample feature sequence.

[0160] Step 2: Fuse the third sequence and the feature sequence of the qth sample to obtain the intermediate sequence of the qth sample; Optionally, the operation can be merged into an addition operation.

[0161] In one embodiment, the sequence elements of the same order in the third sequence and the feature sequence of the q-th sample are weighted and summed to obtain the sequence element of the corresponding order in the intermediate sequence of the q-th sample. Optionally, the weighting weight is a hyperparameter. For example, the 5th sequence element in the third sequence and the 5th sequence element in the feature sequence of the q-th sample can be directly added to obtain the 5th sequence element in the intermediate sequence of the q-th sample.

[0162] Step 3: Mask the 2jth sequence element and the sequence elements after the 2jth sequence element in the intermediate sequence of the qth sample using the (q+1)th masking self-attention network to obtain the (q+1)th sample masking sequence. Perform self-attention operation on the (q+1)th sample masking sequence to obtain the (q+1)th sample feature sequence. When q takes the value of p-1, the (q+1)th sample feature sequence is the jth sample feature.

[0163] In one embodiment, the (q+1)th masking self-attention network includes a cascaded masking self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer. The masking self-attention layer in the (q+1)th masking self-attention network masks the 2j-th sequence element and all subsequent sequence elements in the intermediate sequence of the q-th sample, obtaining the (q+1)th sample mask sequence. Self-attention is then performed on the (q+1)th sample mask sequence to obtain the (q+1)th sample feature sequence. The feedforward network layer in the (q+1)th masking self-attention network maps the (q+1)th sample feature sequence to a (q+1)th sample feedforward feature sequence. The residual connection layer in the (q+1)th masking self-attention network combines the intermediate sequence of the q-th sample with the (q+1)th sample feedforward feature sequence to obtain the (q+1)th sample residual feature sequence. The normalization layer in the (q+1)th masking self-attention network performs a normalization operation on the (q+1)th sample residual feature sequence to obtain the updated (q+1)th sample feature sequence.

[0164] Step 12312: The predicted piano data corresponding to the j-th sample measure is obtained by the output network based on the features of the j-th sample.

[0165] For an explanation of how the output network predicts the piano data corresponding to the j-th sample measure based on the j-th sample feature, please refer to the explanation of how the output network predicts the piano data corresponding to the i-th measure based on the i-th feature in step 420 above. It will not be repeated here.

[0166] based on Figure 13 In the optional embodiment shown, the sample audio is audio from the set of piano-dominant audio, which is piano-dominant audio, referring to audio in which piano accompaniment is the main accompaniment.

[0167] In one embodiment, the piano-dominant audio is obtained through a step-by-step screening process.

[0168] Step S1: Obtain multiple candidate audio files; Multiple candidate audios can be a large number of audios in the music library.

[0169] Step S2: Separate the tracks of each candidate audio from the multiple candidate audios to obtain the piano track and other accompaniment tracks for each candidate audio. For each candidate audio from multiple candidate audio samples, music multitrack separation technology is used to separate the instrument tracks of each candidate audio sample, resulting in piano accompaniment track, guitar accompaniment track, bass accompaniment track, drum accompaniment track, vocal track, and other independent tracks. All accompaniment tracks except the piano accompaniment track are used as other accompaniment tracks (the vocal track has been filtered out).

[0170] Step S3: By comparing the energy value of the piano track of each candidate audio with the energy value of other accompaniment tracks, the piano-dominant audio is selected from multiple candidate audios. The piano-dominant audio is the audio in which the piano accompaniment is dominant. The piano-dominant audio is added to the piano-dominant audio set.

[0171] In one embodiment, for each candidate audio track, a total accompaniment energy value is calculated based on the energy values ​​of the piano track and other accompaniment tracks. If the proportion of the piano track's energy value to the total accompaniment energy value is greater than a proportion threshold, the candidate audio track is designated as the piano-dominant audio. Optionally, the total accompaniment energy value is obtained by summing the temporal energy values ​​or RMS (Root Mean Square) energy values ​​of each track (including the piano accompaniment track and other accompaniment tracks). Optionally, the proportion threshold is 90%.

[0172] In another embodiment, for each candidate audio, a segmentation operation is performed on the piano track to obtain k piano track segments; optionally, the segmentation granularity is in units, and the segmentation operation is performed on the piano track of each candidate audio unit by unit to obtain k piano track segments.

[0173] Additionally, the other accompaniment tracks are segmented to obtain k other accompaniment track segments, k piano track segments, and k other accompaniment track segments are aligned; optionally, the segmentation granularity is at the measure level, and the segmentation operation is performed measure by measure on the other accompaniment tracks of each candidate audio to obtain k other accompaniment track segments.

[0174] For the t-th piano track segment out of k piano track segments, if the ratio of the energy value of the t-th piano track segment to the total energy value of the t-th segment exceeds a first ratio threshold, the t-th piano track segment is determined to be a piano-dominant segment; the total energy value of the t-th segment is the sum of the energy values ​​of the t-th piano track segment and the energy values ​​of the t-th other accompaniment track segments. For example, the first ratio threshold is 50%. If the ratio of the energy value of the t-th piano track segment to the total energy value of the t-th segment exceeds 50%, the t-th piano track segment is determined to be a piano-dominant segment. Optionally, the total energy value of the t-th segment can be obtained by calculating the sum of the time-domain energy values ​​of the t-th piano track segment and the t-th other accompaniment track segments. Optionally, the total energy value of the t-th segment can be obtained by calculating the sum of the RMS energy values ​​of the t-th piano track segment and the t-th other accompaniment track segments.

[0175] If the proportion of piano-dominant segments in the k piano track segments exceeds a second ratio threshold, the candidate audio is determined to be piano-dominant audio. For example, the second ratio threshold is 90%. If the proportion of piano-dominant segments in the k piano track segments exceeds 90%, then the candidate audio is considered to be piano-dominant audio.

[0176] The above embodiments provide a method for generating a training set for a piano accompaniment generation model. By segmenting the audio, determining whether each segment is a piano-dominant segment, and further determining the ratio of the piano-dominant segment to all segments, if the ratio exceeds a certain threshold, the current candidate audio is confirmed as the piano-dominant audio. This segment-by-segment determination of piano dominance in the above embodiments allows for finer-grained selection of piano-dominant audio, ensuring the quality of the selected audio. The method for generating the piano-dominant audio set provided by the above embodiments solves the problem of a lack of large-scale, high-quality data in the field of piano accompaniment, significantly improving the scale and quality of training data, thereby ensuring the performance of the piano accompaniment generation model trained based on the piano-dominant audio set.

[0177] Figure 14 This application shows a structural block diagram of a piano accompaniment generation apparatus provided in an exemplary embodiment. The apparatus includes: The acquisition module 1401 is used to acquire the audio data sequence corresponding to n sections of the target audio; Processing module 1402 is used to predict the piano data corresponding to the i-th audio data in the audio data sequence at the i-th time step based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, using a piano accompaniment generation model; the initial value of i is one, and i is less than n. Let i = i + 1, and return to execute the piano accompaniment generation model. At the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, predict the piano data corresponding to the i-th audio data in the audio data sequence, add the piano data corresponding to the i-th audio data to the predicted piano data sequence, and obtain the updated predicted piano data sequence, until the piano data corresponding to the n-th audio data is obtained. The accompaniment generation module 1403 is used to generate the target piano accompaniment corresponding to the target audio based on the updated predicted piano data sequence obtained when i=n.

[0178] In an optional embodiment, the apparatus further includes a sequence generation module 1404; The sequence generation module 1404 is used to take the audio data corresponding to each of the n sections of the target audio as a sequence element, and insert a placeholder element after the audio data corresponding to each section to form a first sequence; The processing module 1402 is used to predict the piano data corresponding to the i-th measure at the i-th time step based on the sequence elements before the 2i-th sequence element in the first sequence, using the piano accompaniment generation model; and replace the 2i-th sequence element with the piano data corresponding to the i-th measure, where the initial value of i is one and i is less than n. The accompaniment generation module 1403 is used to extract even-numbered sequence elements from the first sequence when i=n to form a second sequence; and to generate the target piano accompaniment corresponding to the target audio based on the second sequence.

[0179] In an optional embodiment, the piano accompaniment generation model includes a masking self-attention network and an output network; the processing module 1402 is further configured to, in the i-th time step, mask the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the first sequence through the masking self-attention network to form a masked sequence, and perform self-attention calculation on the masked sequence to obtain the i-th feature. By using the output network, the piano data corresponding to the i-th measure is predicted based on the i-th feature.

[0180] In an optional embodiment, the masking self-attention network includes p masking self-attention networks, where p is a positive integer greater than one; The processing module 1402 is also used to mask the 2ith sequence element and the sequence elements after the 2ith sequence element in the first sequence through the first masking self-attention network in the i-th time step to obtain the first masking sequence, and to perform self-attention operation on the first masking sequence to obtain the first feature sequence. The first sequence and the q-th feature sequence are merged to obtain the q-th intermediate sequence; The 2i-th sequence element and the sequence elements following the 2i-th sequence element in the q-th intermediate sequence are masked by the (q+1)-th masking self-attention network to obtain the (q+1)-th masking sequence. Self-attention operation is performed on the (q+1)-th masking sequence to obtain the (q+1)-th feature sequence. The initial value of q is 1. When the value of q is p-1, the (q+1)-th feature sequence is the i-th feature.

[0181] In an optional embodiment, each masking self-attention network includes a cascaded masking self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer.

[0182] In an optional embodiment, the processing module 1402 is further configured to mask the 2i-th sequence element and the sequence elements following the 2i-th sequence element in the first sequence through the masking self-attention layer in the first masking self-attention network to obtain a first masked sequence, and perform self-attention operation on the first masked sequence to obtain a first feature sequence. The first feature sequence is mapped to the first feedforward feature sequence through the first masked feedforward network layer in the self-attention network; The first sequence is combined with the first feedforward feature sequence through the residual connection layer in the first masked self-attention network to obtain the first residual feature sequence; The first residual feature sequence is normalized by passing through the normalization layer in the first masked self-attention network to obtain the updated first feature sequence.

[0183] In an optional embodiment, the processing module 1402 is further configured to mask the 2i-th sequence element and the sequence elements after the 2i-th sequence element in the q-th intermediate sequence through the masking self-attention layer in the q+1-th masking self-attention network to obtain the q+1-th masked sequence, and perform self-attention operation on the q+1-th masked sequence to obtain the q+1-th feature sequence. The (q+1)th feature sequence is mapped to the (q+1)th feedforward feature sequence through the (q+1)th masked feedforward network layer in the attention network. By combining the q-th intermediate sequence with the q+1-th feedforward feature sequence through the residual connection layer in the masked self-attention network, the q-th residual feature sequence is obtained. The normalization operation is performed on the (q+1)th residual feature sequence through the normalization layer in the masked self-attention network to obtain the updated (q+1)th feature sequence.

[0184] In an optional embodiment, each masking layer in the attention network is further cascaded with at least one of the attention layers and convolutional layers across the network.

[0185] In an optional embodiment, the processing module 1402 is further configured to perform at least one of an embedding operation, a position encoding operation, and a feature value random discarding operation on the first sequence to obtain an updated first sequence.

[0186] In an optional embodiment, the audio data corresponding to the i-th section of the target audio includes at least one of the following data: The measure number of the i-th measure in the target audio; The subbeat number of the subbeat contained in the i-th measure of the target audio; The speed of the target audio; The notes played in the subbeat of the i-th measure in the target audio; The duration of the subbeat notes contained in the i-th measure of the target audio; The intensity of the subbeat notes contained in the i-th measure of the target audio; The chords of the target audio.

[0187] In an optional embodiment, the piano data corresponding to the audio data of the i-th measure in the target audio includes at least one of the following: The measure number of the i-th measure in the target piano accompaniment; The subbeat number of the subbeat contained in the i-th measure of the target piano accompaniment; The target tempo for the piano accompaniment; The notes played in the subbeat of the i-th measure in the target piano accompaniment; The duration of the subbeat notes contained in the i-th measure of the target piano accompaniment; The intensity of the subbeat notes contained in the i-th measure of the target piano accompaniment; The target piano accompaniment chords.

[0188] In summary, in this embodiment, an audio data sequence containing n measures is obtained. The piano data corresponding to the i-th sub-audio data sequence (the first i audio data) and the (i-1)-th sub-audio data sequence (the first i-1 audio data) are used to predict the piano data corresponding to the i-th audio data. This provides a measure-by-measure autoregressive method for generating a target piano accompaniment. This application decomposes the task of predicting the piano data corresponding to the target piano accompaniment into the task of predicting the piano data corresponding to each measure. When predicting the piano data corresponding to each measure, it relies on the measure data (including audio data and piano data) of all preceding measures and the audio data corresponding to the current measure. This ensures that the final generated piano accompaniment for each measure continues the style of the previous measures, resulting in a more unified style and a more natural, smooth, and expressive overall piano accompaniment.

[0189] Figure 15 This application shows a structural block diagram of a training apparatus for a piano accompaniment generation model provided in an exemplary embodiment. The apparatus includes: The acquisition module 1501 is used to acquire the sample audio data sequence corresponding to m sample measures of the sample audio; and to acquire the sample piano data sequence corresponding to m sample measures of the sample piano accompaniment, wherein the sample piano accompaniment is the piano accompaniment of the sample audio. Processing module 1502 is used to predict the predicted piano data corresponding to the j-th sample measure at the j-th time step based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, using a piano accompaniment generation model; the initial value of j is one, and j is less than n. Let j = j + 1, and return to execute the piano accompaniment generation model. At the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, predict the predicted piano data corresponding to the j-th sample measure. Training module 1503 is used to train the piano accompaniment generation model based on the error between the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence.

[0190] In an optional embodiment, the device further includes a sequence generation module 1504; The sequence generation module 1504 is used to insert piano data corresponding to the same sample measure after the audio data corresponding to each of the m sample measures to form a third sequence; The processing module 1502 is used to generate a model through piano accompaniment. For each value of j, based on the first sequence element to the 2j-1th sequence element in the third sequence, the predicted piano data corresponding to the j-th sample measure is predicted, where m is a positive integer and j is a positive integer from 1 to m. Training module 1503 is used to train a piano accompaniment generation model based on the error between the predicted m piano data and the piano data corresponding to the m sample measures.

[0191] In an optional embodiment, the training module 1503 is further configured to calculate the conditional probability corresponding to each predicted piano data based on the predicted m predicted piano data and the piano data corresponding to the m sample measures. Calculate the negative log-likelihood loss based on m conditional probabilities; A piano accompaniment generation model was trained using negative log-likelihood loss.

[0192] In an optional embodiment, the sample audio is audio from a set of piano-dominant audio; the apparatus also includes an audio set generation module 1505: the audio set generation module 1505 is used to acquire multiple candidate audios; For each candidate audio among multiple candidate audios, separate the tracks to obtain the piano track and other accompaniment tracks for each candidate audio; By comparing the energy values ​​of the piano track and other accompaniment tracks of each candidate audio, piano-dominant audio is selected from multiple candidate audios. Piano-dominant audio is audio where the piano accompaniment is dominant. The piano-dominant audio is then added to the piano-dominant audio set.

[0193] In an optional embodiment, the audio set generation module 1505 is configured to perform a segmentation operation on the piano track for each candidate audio to obtain k piano track segments; and to perform a segmentation operation on other accompaniment tracks to obtain k other accompaniment track segments, and align the k piano track segments and the k other accompaniment track segments. For the t-th piano track segment among k piano track segments, if the ratio of the energy value of the t-th piano track segment to the total energy value of the t-th segment exceeds the first ratio threshold, the t-th piano track segment is determined to be the piano-dominant segment; the total energy value of the t-th segment is the sum of the energy value of the t-th piano track segment and the energy values ​​of the other t-th accompaniment track segments. If the proportion of piano-dominant segments in the k piano track segments exceeds the second ratio threshold, the candidate audio is determined to be piano-dominant audio.

[0194] In summary, the embodiments of this application provide a scheme for predicting piano data measure by measure and training a piano accompaniment generation model based on the error of accumulated piano data of m measures. The piano accompaniment generation model can learn the dependencies between m measures, thereby improving the quality of the piano accompaniment generated by the piano accompaniment generation model, and the final generated piano accompaniment is more natural, smooth and expressive.

[0195] Figure 16 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1600 includes a Central Processing Unit (CPU) 1601, a system memory 1604 including Random Access Memory (RAM) 1602 and Read-Only Memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the CPU 1601. The computer device 1600 also includes a basic input / output system (I / O system) 1606 to facilitate information transfer between various devices within the computer device, and a mass storage device 1607 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0196] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609 for user input, such as a mouse or keyboard. Both the display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include the input / output controller 1610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0197] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer device-readable media provide non-volatile storage for the computer device 1600. That is, the mass storage device 1607 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0198] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 1604 and mass storage device 1607 described above can be collectively referred to as memory.

[0199] According to various embodiments of this disclosure, the computer device 1600 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1600 can be connected to the network 1611 via a network interface unit 1612 connected to the system bus 1605, or the network interface unit 1612 can be used to connect to other types of networks or remote computer device systems (not shown).

[0200] The memory also includes one or more programs stored in the memory. The central processing unit 1601 executes the one or more programs to implement all or part of the steps of the above-mentioned method for generating piano accompaniment or training method for piano accompaniment generation model.

[0201] Figure 17 A structural block diagram of a computer device 1700 provided in an exemplary embodiment of this application is shown. The computer device 1700 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 1700 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0202] Typically, computer device 1700 includes a processor 1701 and a memory 1702.

[0203] Processor 1701 may include one or more processing cores, such as a 4-core processor, a 17-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0204] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1702 is used to store at least one instruction, which is executed by the processor 1701 to implement the piano accompaniment generation method or the piano accompaniment generation model training method provided in the method embodiments of this application.

[0205] In some embodiments, the computer device 1700 may also optionally include a peripheral device interface 1703 and at least one peripheral device. The processor 1701, memory 1702, and peripheral device interface 1703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1703 via a bus, signal line, or circuit board. For example, the peripheral device may include at least one of the following: a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707, and a power supply 1708.

[0206] Peripheral interface 1703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1701 and memory 1702. In some embodiments, processor 1701, memory 1702 and peripheral interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1701, memory 1702 and peripheral interface 1703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0207] The radio frequency (RF) circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0208] Display screen 1705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1701 for processing. In this case, display screen 1705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1705, disposed on the front panel of computer device 1700; in other embodiments, there may be at least two display screens, disposed on different surfaces of computer device 1700 or in a folded design; in still other embodiments, display screen 1705 may be a flexible display screen, disposed on a curved or folded surface of computer device 1700. Furthermore, display screen 1705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0209] The camera assembly 1706 is used to acquire images or videos. Optionally, the camera assembly 1706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0210] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 1701 for processing, or to the radio frequency circuit 1704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, positioned at different locations within the computer device 1700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0211] Power supply 1708 is used to supply power to the various components in computer device 1700. Power supply 1708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1708 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0212] In some embodiments, the computer device 1700 further includes one or more sensors 1709. The one or more sensors 1709 include, but are not limited to, an accelerometer 1710, a gyroscope 1711, a pressure sensor 1712, an optical sensor 1713, and a proximity sensor 1714.

[0213] Accelerometer 1710 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 1700. For example, accelerometer 1710 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1701 can control display screen 1705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1710. Accelerometer 1710 can also be used for games or for acquiring user motion data.

[0214] The gyroscope sensor 1711 can detect the orientation and rotation angle of the computer device 1700. The gyroscope sensor 1711 can work in conjunction with the accelerometer sensor 1710 to acquire 3D motion data from the user on the computer device 1700. Based on the data acquired by the gyroscope sensor 1711, the processor 1701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0215] Pressure sensor 1712 can be disposed on the side bezel of computer device 1700 and / or on the lower layer of display screen 1705. When pressure sensor 1712 is disposed on the side bezel of computer device 1700, it can detect the user's grip signal on computer device 1700, and processor 1701 can perform left / right hand recognition or quick operation based on the grip signal collected by pressure sensor 1712. When pressure sensor 1712 is disposed on the lower layer of display screen 1705, processor 1701 can control operable controls on the UI interface based on the user's pressure operation on display screen 1705. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0216] Optical sensor 1713 is used to collect ambient light intensity. In one embodiment, processor 1701 can control the display brightness of display screen 1705 based on the ambient light intensity collected by optical sensor 1713. For example, when the ambient light intensity is high, the display brightness of display screen 1705 is increased; when the ambient light intensity is low, the display brightness of display screen 1705 is decreased. In another embodiment, processor 1701 can also dynamically adjust the shooting parameters of camera assembly 1706 based on the ambient light intensity collected by optical sensor 1713.

[0217] The proximity sensor 1714, also known as a distance sensor, is typically located on the front panel of the computer device 1700. The proximity sensor 1714 is used to detect the distance between the user and the front of the computer device 1700. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually decreasing, the processor 1701 controls the display screen 1705 to switch from a screen-on state to a screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the computer device 1700 is gradually increasing, the processor 1701 controls the display screen 1705 to switch from a screen-off state to a screen-on state.

[0218] Those skilled in the art will understand that Figure 17 The structure shown does not constitute a limitation on the computer device 1700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0219] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the piano accompaniment generation method or the piano accompaniment generation model training method provided in the above method embodiments.

[0220] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the piano accompaniment generation method or the piano accompaniment generation model training method provided in the above-described method embodiments.

[0221] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0222] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0223] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating piano accompaniment, characterized in that, The method includes: Obtain the audio data sequence corresponding to n sections of the target audio; Using a piano accompaniment generation model, at the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, the piano data corresponding to the i-th audio data is predicted; the initial value of i is one, and i is less than n. Add the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence; Let i = i + 1, and return to execute the piano accompaniment generation model. At the i-th time step, based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence in the audio data sequence, predict the piano data corresponding to the i-th audio data in the audio data sequence. Add the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence, until the piano data corresponding to the n-th audio data is obtained. Based on the updated predicted piano data sequence obtained when i=n, ​​the target piano accompaniment corresponding to the target audio is generated.

2. The method according to claim 1, characterized in that, The method further includes: The audio data corresponding to each of the n sections of the target audio is taken as a sequence element, and a placeholder element is inserted after the audio data corresponding to each section to form a first sequence; The piano accompaniment generation model, at the i-th time step, predicts the piano data corresponding to the i-th audio data in the audio data sequence based on the predicted piano data sequence corresponding to the i-th sub-audio data sequence and the (i-1)-th sub-audio data sequence, including: Using the piano accompaniment generation model, at the i-th time step, the piano data corresponding to the i-th measure is predicted based on the sequence elements before the 2i-th sequence element in the first sequence. The step of adding the piano data corresponding to the i-th audio data to the predicted piano data sequence to obtain the updated predicted piano data sequence includes: Replace the 2i-th sequence element with the piano data corresponding to the i-th section; The process of generating the target piano accompaniment corresponding to the target audio based on the updated predicted piano data sequence obtained when i=n includes: Based on the fact that when i=n, ​​the even-numbered sequence elements are extracted from the first sequence to form the second sequence; Based on the second sequence, a target piano accompaniment corresponding to the target audio is generated.

3. The method according to claim 2, characterized in that, The piano accompaniment generation model includes a masked self-attention network and an output network; The process of predicting the piano data corresponding to the i-th measure using the piano accompaniment generation model at the i-th time step based on the sequence elements preceding the 2i-th sequence element in the first sequence includes: In the i-th time step, the masking self-attention network is used to mask the 2i-th sequence element and the sequence elements following the 2i-th sequence element in the first sequence to form a masked sequence. Self-attention calculation is performed on the masked sequence to obtain the i-th feature. The output network is used to predict the piano data corresponding to the i-th measure based on the i-th feature.

4. The method according to claim 3, characterized in that, The masked self-attention network includes p masked self-attention networks, where p is a positive integer greater than one; In the i-th time step, the masking self-attention network masks the 2i-th sequence element and the sequence elements following the 2i-th sequence element in the first sequence to form a masked sequence. Self-attention calculation is then performed on the masked sequence to obtain the i-th feature, including: In the i-th time step, the 2i-th sequence element and the sequence elements following the 2i-th sequence element in the first sequence are masked by the first masking self-attention network to obtain the first masked sequence. Self-attention operation is performed on the first masked sequence to obtain the first feature sequence. The first sequence and the q-th feature sequence are fused to obtain the q-th intermediate sequence; The 2i-th sequence element and the sequence elements following the 2i-th sequence element in the q-th intermediate sequence are masked by the (q+1)-th masking self-attention network to obtain the (q+1)-th masking sequence. Self-attention operation is performed on the (q+1)-th masking sequence to obtain the (q+1)-th feature sequence. The initial value of q is one. When the value of q is p-1, the (q+1)-th feature sequence is the i-th feature.

5. The method according to claim 4, characterized in that, Each of the masked self-attention networks includes a cascaded masked self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer.

6. The method according to claim 5, characterized in that, The process involves masking the 2i-th sequence element and subsequent sequence elements in the first sequence using a first masking self-attention network to obtain a first masked sequence. Self-attention operations are then performed on the first masked sequence to obtain a first feature sequence, including: The first masked sequence is obtained by masking the 2ith sequence element and the sequence elements after the 2ith sequence element in the first sequence through the masking self-attention layer in the first masking self-attention network, and then performing self-attention operation on the first masked sequence to obtain the first feature sequence. The first feature sequence is mapped to a first feedforward feature sequence through the feedforward network layer in the first masking self-attention network; The first sequence is combined with the first feedforward feature sequence through the residual connection layer in the first masked self-attention network to obtain the first residual feature sequence; The first residual feature sequence is normalized by the normalization layer in the first masked self-attention network to obtain the updated first feature sequence.

7. The method according to claim 5, characterized in that, The process involves masking the 2i-th sequence element and the sequence elements following the 2i-th sequence element in the q-th intermediate sequence using the (q+1)-th masking self-attention network to obtain the (q+1)-th masked sequence. Self-attention operations are then performed on the (q+1)-th masked sequence to obtain the (q+1)-th feature sequence, which includes: By using the masking self-attention layer in the (q+1)th masking self-attention network, the 2ith sequence element and the sequence elements following the 2ith sequence element in the qth intermediate sequence are masked to obtain the (q+1)th masked sequence. Self-attention operation is then performed on the (q+1)th masked sequence to obtain the (q+1)th feature sequence. The (q+1)th feature sequence is mapped to the (q+1)th feedforward feature sequence through the (q+1)th masked self-attention network layer. By combining the q-th intermediate sequence with the q+1-th feedforward feature sequence through the residual connection layer in the q+1-th masked self-attention network, the q+1-th residual feature sequence is obtained. The normalization operation is performed on the (q+1)th residual feature sequence through the normalization layer in the masked self-attention network to obtain the updated (q+1)th feature sequence.

8. The method according to claim 5, characterized in that, Each of the masking layers in the attention network is further cascaded with at least one of the attention layers and convolutional layers across the network.

9. The method according to any one of claims 2 to 8, characterized in that, After taking the audio data corresponding to each of the n sections of the target audio as a sequence element and inserting a placeholder element after the audio data corresponding to each section to form a first sequence, the method further includes: Perform at least one of the following operations on the first sequence: embedding operation, position encoding operation, and feature value random discarding operation, to obtain the updated first sequence.

10. The method according to any one of claims 1 to 8, characterized in that, The audio data corresponding to the i-th section of the target audio includes at least one of the following data: The section number of the i-th section in the target audio; The subbeat number of the subbeat contained in the i-th section of the target audio; The speed of the target audio; The notes played in the subbeat of the i-th measure in the target audio; The duration of the subbeat notes contained in the i-th measure of the target audio; The intensity of the subbeat notes contained in the i-th measure of the target audio; The chords of the target audio.

11. The method according to any one of claims 1 to 8, characterized in that, The piano data corresponding to the audio data of the i-th measure in the target audio includes at least one of the following: The measure number of the i-th measure in the target piano accompaniment; The subbeat number of the subbeat contained in the i-th measure in the target piano accompaniment; The speed of the target piano accompaniment; The notes played in the subbeat of the i-th measure in the target piano accompaniment; The duration of the subbeat notes contained in the i-th measure of the target piano accompaniment; The intensity of the subbeat notes contained in the i-th measure of the target piano accompaniment; The target piano accompaniment chords.

12. A training method for a piano accompaniment generation model, characterized in that, The method includes: Obtain the sample audio data sequence corresponding to m sample measures of the sample audio; and obtain the sample piano data sequence corresponding to m sample measures of the sample piano accompaniment, wherein the sample piano accompaniment is the piano accompaniment of the sample audio; Using the piano accompaniment generation model, at the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, the predicted piano data corresponding to the j-th sample measure is predicted; the initial value of j is one, and j is less than n; Let j = j + 1, and return to execute the piano accompaniment generation model. At the j-th time step, based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, predict the predicted piano data corresponding to the j-th sample measure. The piano accompaniment generation model is trained based on the error between the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence.

13. The method according to claim 12, characterized in that, The method further includes: After the audio data corresponding to each of the m sample measures, insert the piano data corresponding to the same sample measure to form a third sequence; The piano accompaniment generation model, at the j-th time step, predicts the predicted piano data corresponding to the j-th sample measure in the audio data sequence based on the j-th sample sub-audio data sequence in the sample audio data sequence and the (j-1)-th sample sub-piano data sequence in the sample piano data sequence, including: Using the piano accompaniment generation model, at the j-th time step, based on the first sequence element to the (2j-1)th sequence element in the third sequence, the predicted piano data corresponding to the j-th sample measure is predicted.

14. The method according to claim 12, characterized in that, The piano accompaniment generation model is trained based on the error between the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence, including: Based on the m predicted piano data obtained when j=m and the piano data corresponding to the m sample measures in the sample piano data sequence, calculate the conditional probability corresponding to each predicted piano data. Calculate the negative log-likelihood loss based on m conditional probabilities; The piano accompaniment generation model is trained based on the negative log-likelihood loss.

15. The method according to claim 13 or 14, characterized in that, The sample audio is audio from a set of piano-dominant audio; the method further includes: Get multiple candidate audio; Each candidate audio among the multiple candidate audios is separated into tracks to obtain a piano track and other accompaniment tracks for each candidate audio. By comparing the energy value of the piano track of each candidate audio with the energy value of other accompaniment tracks, a piano-dominant audio is selected from the multiple candidate audios. The piano-dominant audio is the audio in which the piano accompaniment is dominant. The piano-dominant audio is then added to the set of piano-dominant audios.

16. The method according to claim 15, characterized in that, The process of filtering out the piano-dominant audio from the multiple candidate audios by comparing the energy values ​​of the piano track and other accompaniment tracks for each candidate audio includes: For each candidate audio track, a segmentation operation is performed on the piano track to obtain k piano track segments; and a segmentation operation is performed on the other accompaniment tracks to obtain k other accompaniment track segments, wherein the k piano track segments and the k other accompaniment track segments are aligned. For the t-th piano track segment among the k piano track segments, if the ratio of the energy value of the t-th piano track segment to the total energy value of the t-th segment exceeds a first ratio threshold, the t-th piano track segment is determined to be the piano-dominant segment; the total energy value of the t-th segment is the sum of the energy value of the t-th piano track segment and the energy values ​​of the t-th other accompaniment track segments. If the proportion of the number of piano-dominant segments in the k piano track segments exceeds the second ratio threshold, the candidate audio is determined to be the piano-dominant audio.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the piano accompaniment generation method as described in any one of claims 1 to 11, or the piano accompaniment generation model training method as described in any one of claims 12 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by the processor to implement the method for generating piano accompaniment as described in any one of claims 1 to 11, or the method for training a piano accompaniment generation model as described in any one of claims 12 to 16.

Images