Music generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a music generation method and device, equipment and a storage medium. The method comprises the following steps: based on a received music generation request, constructing a target mask corresponding to the music generation request; generating an intermediate encoded representation based on the mask matrix and the first encoded representation of the reference music content; processing the intermediate coded representation using a target model to generate a second coded representation, the second coded representation being generated by replacing a target portion in the intermediate coded representation, the target portion being indicated by the mask matrix; and generating the target music content based on the second coded representation. Therefore, according to the embodiment of the invention, expected music content can be generated by utilizing the mask matrix.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for music generation. Background Technology

[0002] With the development of computer technology, music generation technology, as a cutting-edge innovation in the field of artificial intelligence, is gradually changing the traditional model of music creation and generation. Traditional music generation models have limited functionality and cannot guarantee that background sounds or vocals remain unchanged during the music generation process. Summary of the Invention

[0003] In a first aspect of this disclosure, a music generation method is provided. The method includes: constructing a target mask corresponding to a received music generation request; generating an intermediate encoded representation based on the target mask and a first encoded representation of reference music content; processing the intermediate encoded representation using a target model to generate a second encoded representation, the second encoded representation being generated by replacing a target portion in the intermediate encoded representation, the target portion being indicated by the target mask; and generating target music content based on the second encoded representation.

[0004] In a second aspect of this disclosure, an apparatus for music generation is provided. The apparatus includes: a target mask construction module configured to construct a target mask corresponding to a received music generation request; an intermediate encoded representation generation module configured to generate an intermediate encoded representation based on the target mask and a first encoded representation of reference music content; a second encoded representation generation module configured to process the intermediate encoded representation using a target model to generate a second encoded representation, the second encoded representation being generated by replacing a target portion in the intermediate encoded representation, the target portion being indicated by the target mask; and a music content generation module configured to generate target music content based on the second encoded representation.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A flowchart illustrating music generation according to some embodiments of this disclosure is shown;

[0011] Figures 3A to 3E Example diagrams of target masks according to some embodiments of the present disclosure are shown;

[0012] Figures 4A to 4D A flowchart illustrating the generation of music based on different target masks according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A flowchart for training a music generation model according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A block diagram of an apparatus for music generation according to some embodiments of the present disclosure is shown; and

[0015] Figure 7 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0021] In this specification and the embodiments, the data involved in model training and inference (including but not limited to the data itself, data acquisition, and / or use) comply with the requirements of relevant laws and regulations. As briefly mentioned above, with the development of computer technology, music generation technology, as a cutting-edge innovation in the field of artificial intelligence, is gradually changing the traditional mode of music creation and generation. Traditional music generation models have limited music generation functions and cannot guarantee that background sounds or vocals remain unchanged during the music generation process.

[0022] Therefore, embodiments of this disclosure propose a scheme for music generation. According to various embodiments of this disclosure, a target mask corresponding to a received music generation request is constructed; an intermediate encoded representation is generated based on the target mask and a first encoded representation of reference music content; the intermediate encoded representation is processed using a target model to generate a second encoded representation, the second encoded representation being generated by replacing a target portion in the intermediate encoded representation, the target portion being indicated by the target mask; and target music content is generated based on the second encoded representation.

[0023] In this way, the embodiments of this disclosure can construct a corresponding mask matrix based on the music generation request, and can further utilize the mask matrix to generate corresponding music content. Therefore, the embodiments of this disclosure can enrich music generation scenarios and improve the quality of music content.

[0024] Furthermore, embodiments of this disclosure utilize a mask matrix to indicate the target portion in the intermediate encoded representation corresponding to the content to be generated. Therefore, embodiments of this disclosure can ensure that other audio data are not interfered with during music generation, thus maintaining the consistency of the overall music style.

[0025] Example embodiments of this disclosure are described below with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110 and target model 136.

[0028] exist Figure 1 In environment 100, electronic device 110 can receive input data 130 from user 140, which may include audio data and / or text data. Further, electronic device 110 can invoke target model 136 to process the input data 130 to obtain target audio 120. The target model may be, for example, a non-causal deep learning model such as a diffusion model, which can generate synthesized audio content based on the input text content or lyrics.

[0029] In some embodiments, electronic device 110 may include various types of computing systems / servers capable of providing computing power, and electronic device 110 may include terminal devices. Such terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), voice / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Electronic device 110 may, for example, include various types of computing systems / servers capable of providing computing power, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, etc. Although shown as a single device, electronic device 110 may include multiple physical devices.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example of music generation process

[0033] Figure 2 A flowchart of a voice processing procedure 200 according to some embodiments of the present disclosure is shown. Procedure 200 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 200.

[0034] like Figure 2 As shown in box 210, electronic device 110 can construct a target mask corresponding to the received music generation request based on the received music generation request.

[0035] In some embodiments, the target mask, also known as a mask matrix, may be constructed accordingly based on the content to be generated in the music generation request. In some of these targets, the target mask may correspond to a preset mask pattern corresponding to different types of generation requests. In some embodiments, the mask pattern includes, but is not limited to, the first mask pattern, the second mask pattern, the third mask pattern, the fourth mask pattern, and the fifth mask pattern described below.

[0036] In some embodiments, the first mask pattern, for example, indicates the generation of subsequent audio track data corresponding to an existing track based on reference music content. In some scenarios, the first mask pattern may also be referred to as a continuation mask. Figure 3AA schematic diagram of a continuation mask according to some embodiments of the present disclosure is shown. For example... Figure 3A As shown, the continuation mask can mask all audio tracks in the time dimension and instruct the model to generate subsequent music content, thereby achieving the ability to continue writing music content.

[0037] In some embodiments, the second mask pattern indicates that additional audio track data corresponding to the new audio track should be generated based on reference music content. In some scenarios, the second mask pattern may also be referred to as an audio track conversion mask. Figure 3B A schematic diagram of an audio track conversion mask according to some embodiments of the present disclosure is shown. For example... Figure 3B As shown, an audio track conversion mask can convert other audio tracks based on one or more audio tracks. Such music generation scenarios could include adding an instrumental track based on a vocal track, or adding a vocal track based on an instrumental track.

[0038] In some embodiments, the third mask pattern can instruct the generation of subsequent track data corresponding to at least one existing track and additional track data corresponding to a new track based on reference music content. In some scenarios, the third mask pattern may also be referred to as an extension mask. Figure 3C A schematic diagram of an expansion mask according to some embodiments of the present disclosure is shown. For example... Figure 3C As shown, the extension mask can be used to continue writing content on the main audio track and supplement the content of other audio tracks. Such music generation scenarios may include, for example, generating subsequent music content and adding an accompaniment track based on a given humming content.

[0039] In some embodiments, the fourth masking mode can instruct the generation of additional audio track data corresponding to multiple audio tracks based on existing track data of multiple audio tracks based on reference music content, with the additional audio track data of different tracks corresponding to different time periods. In some scenarios, the fourth masking mode can also be referred to as a hybrid mask. Figure 3D A schematic diagram of a hybrid mask according to some embodiments of the present disclosure is shown. For example... Figure 3D As shown, a blending mask can indicate that the content of one audio track is used as the starting content and the content of another audio track is used as the ending content, and the intermediate content between the two audio tracks is generated accordingly. Such music generation scenarios can include, for example, transitioning from one specified music segment to another specified music segment.

[0040] In some embodiments, the fifth masking mode can instruct the generation of intermediate track data corresponding to at least one existing track and additional track data corresponding to a new track, based on reference music content. In some scenarios, unlike the third masking mode, the track content generated by the fifth masking mode is the intermediate portion of a specific track. In some scenarios, the fifth masking mode is also referred to as a time-track extension mask.

[0041] Figure 3E A schematic diagram of a time-track extension mask according to some embodiments of the present disclosure is shown. For example... Figure 3E As shown, a time-track augmentation mask can indicate intermediate content for generating a specific track segment and add other tracks. Such music generation scenarios could include, for example, augmenting content within an instrumental track and adding other tracks.

[0042] The target model can process different input audio data based on different mask patterns to generate different music content, thus enriching the music generation function of the target model.

[0043] In some embodiments, the electronic device 110 may, for example, match the corresponding mask pattern according to the music generation request and generate the corresponding target mask.

[0044] Specifically, electronic device 110 can determine at least one audio track and / or at least one audio track segment to be generated based on a music generation request. Taking a music extension scenario as an example, when electronic device 110 receives a music extension request, it can determine the audio track to be generated based on the intro audio in the music extension request. For example, as Figure 3C As shown, the audio track to be generated can be as follows: Figure 3C The audio track corresponding to the extended mask in the text.

[0045] Furthermore, the electronic device 110 can construct a target mask based on at least one audio track and / or at least one audio track segment. In some examples, the mask portion of the target mask corresponding to at least one audio track and / or at least one audio track segment is set to a preset value (e.g., 0). Continuing with the example of a music extension scenario, the electronic device 110 can construct a target mask based on the received intro audio encoding and the determined audio track to be generated, such as... Figure 3C The extended mask shown.

[0046] The following will further define the specific process for determining the audio tracks and / or track segments to be generated based on the music generation request.

[0047] In some embodiments, a music generation request may, for example, indicate lyrics to be added. Taking a music continuation request as an example, the user may, for instance, indicate lyrics corresponding to the subsequent music content to be generated.

[0048] Accordingly, the electronic device 110 can determine the newly added first lyrics content based on the music generation request. Further, the electronic device 110 can determine the first time length corresponding to the first lyrics content based on the speech rate information of the reference music content.

[0049] For example, electronic device 110 can determine the singing speed in the reference music content by sampling, and can determine the expected time length corresponding to the newly added first lyrics content based on the singing speed.

[0050] Accordingly, electronic device 100 can determine that the at least one audio track segment to be generated includes a first audio track segment from the human voice track with the first time length. Taking a music continuation scenario as an example, electronic device 110 can, for example, construct a music continuation based on the determined first time length. Figure 3A The continuation mask shown is 300A.

[0051] In some scenarios, the reference music content may not include a vocal track. Accordingly, if the music generation request indicates the addition of lyrics, the electronic device can determine the added second lyrics content based on the music generation request. Further, the electronic device 110 can determine, in response to the reference music content not including a vocal track, that the at least one audio track to be generated may include the vocal track.

[0052] As an example, such a music generation request could instruct the generation of a vocal track corresponding to the second set of lyrics based on existing accompaniment content. Accordingly, electronic device 110 could construct, for example... Figure 3B The audio track conversion mask shown is 300B.

[0053] In some embodiments, a music generation request may, for example, instruct the expansion of specific content on a certain audio track. Specifically, the electronic device 110 may, based on the music generation request, obtain the reference music content, which includes multiple track segments corresponding to the target audio track. Further, the electronic device 110 may determine that the at least one track segment to be generated includes a track segment corresponding to a preset time period of the target audio track.

[0054] Taking accompaniment expansion as an example, the music generation request may specify the start and end segments of the input accompaniment track, and may specify a preset time period to be expanded. Accordingly, the electronic device 110 may, for example, construct... Figure 3E The time-track extension mask shown.

[0055] In some scenarios, the preset time period to be expanded can also be determined based on the lyrics content indicated by the music generation request. For example, a user can instruct the insertion of third lyrics content into existing lyrics content, and accordingly, the electronic device 110 can determine that the preset time period to be expanded is the time period corresponding to the third lyrics content.

[0056] In some embodiments, with Figure 3EThe time-track expansion mask shown is an example. In the accompaniment expansion scenario, the music generation request also indicates a set of tracks to be added, such as vocal tracks.

[0057] In some embodiments, the music generation request may indicate audio track segments from two different audio tracks to indicate a transition between the two audio track segments. Specifically, the electronic device 110 may, for example, obtain the reference music content based on the music generation request, the reference music content including at least a second audio track segment corresponding to the first audio track and a third audio track segment corresponding to the second audio track.

[0058] For example, the first audio track can be a vocal track, and the second audio track can be an accompaniment track. The music generation request can, for example, specify the starting audio segment of the vocal track and the ending audio segment of the accompaniment track.

[0059] Accordingly, the electronic device 110 can determine that the at least one audio track segment to be generated includes a fourth audio track segment (e.g., the subsequent content of the second audio track segment) corresponding to the first audio track (e.g., human voice) and a fifth audio track segment (e.g., the preceding content of the third audio track segment) corresponding to the second audio track.

[0060] In some embodiments, the duration of the fourth audio track segment to be expanded may be determined, for example, based on the indicated new lyrics content (also referred to as the fourth lyrics content). Accordingly, the electronic device 110 may construct, based on the fourth and fifth audio track segments to be generated, such as... Figure 3D The hybrid mask shown is 300D.

[0061] Based on the above mask construction process, the embodiments of this disclosure can generate corresponding masks according to different music generation needs, thereby enriching the music generation scenarios.

[0062] Continue to refer to Figure 2 In box 220, electronic device 110 can generate an intermediate encoded representation based on the mask matrix and a first encoded representation of the reference music content.

[0063] In some embodiments, the electronic device 110 may, in response to the reference audio content including multiple audio tracks, use a separation module to determine the audio track data corresponding to the multiple audio tracks, and may use an audio encoder to process the audio track data corresponding to the multiple audio tracks to determine a first encoded representation.

[0064] The following will combine Figures 4A to 4D To describe the music generation process corresponding to different music generation scenarios.

[0065] In some scenarios, such as Figure 4A As shown, Figure 4AThis illustration shows a music generation process in a music expansion scenario according to one embodiment of the present invention. Electronic device 110 can receive a music generation request, which may indicate the lyrics to be added. Electronic device 110 can acquire intro audio 401 (i.e., reference music content), the music expansion request, and the first lyrics content 407 added by user 140. It should be understood that the intro audio 401 may contain multi-track data or only single-track data. When the intro audio 401 includes multi-track data, electronic device 110 may also include a target track specified by user 140, which may be, for example, a vocal track.

[0066] After receiving the intro audio track 401, the electronic device 110 can input the intro audio 401 into the audio separation model 402 to separate multiple audio tracks in the intro audio 401, obtaining the intro audio code 403 (i.e., the first code representation) corresponding to the vocal track. Furthermore, the electronic device 110 can determine the singing speed in the intro audio 401 based on the average of the number of words appearing in the reference music content and the duration of the reference music content.

[0067] Furthermore, the electronic device 110 can calculate the duration of subsequent audio based on the singing speed of the intro audio 401 and the length of the first lyrics content 407, and generate a corresponding extension mask based on the duration of the subsequent audio and the music extension request. After generating the extension mask, the electronic device 110 can combine the intro audio encoding 403 and the extension mask to obtain the intermediate encoding representation 406.

[0068] In other scenarios, such as Figure 4B As shown, Figure 4B This illustration shows a music generation process in a track conversion scenario according to one embodiment of the present invention. The electronic device 110 can receive a music generation request, which may indicate the vocal track to be generated. The electronic device 110 can acquire accompaniment audio 421 (i.e., reference music content), second lyrics content 426, and the track conversion request. The electronic device 110 can input the accompaniment audio track 421 into an audio separation model to separate multiple tracks in the accompaniment audio 421, and encode each track in the accompaniment audio using an audio encoder 423 to obtain an accompaniment audio track code 424 (i.e., a first encoding representation). It should be understood that the accompaniment audio 421 may include multiple tracks, for example, the accompaniment audio 421 may include a piano track, a violin track, and a cello track, etc.

[0069] Furthermore, the electronic device 110 can determine the human voice track to be generated based on the audio track conversion request, and generate an audio track conversion mask based on the human voice track to be generated.

[0070] Finally, the electronic device 110 can combine the accompaniment track encoding 424 with the track conversion mask to obtain the intermediate encoding representation 425.

[0071] In other scenarios, such as Figure 4C As shown, Figure 4C This illustration shows a music generation process in an accompaniment expansion scenario according to one embodiment of the present invention. Electronic device 110 can receive a music generation request, which may specify the expansion of specific content of a certain audio track. Electronic device 110 can obtain reference music content and a preset time period to be expanded specified by user 140. The reference music content includes the start audio segment 431-1 and the end audio segment 431-2 of the vocal track. Electronic device 110 can input the start audio segment 431-1 and the end audio segment 431-2 to audio encoders 432-1 and 432-2 respectively, to obtain start audio code 433-1 and end audio code 433-2. Electronic device 110 can combine the start audio code 433-1 and the end audio code 433-2 into a first encoded representation.

[0072] Furthermore, the electronic device 110 can generate a corresponding time-track expansion mask based on a preset time period specified by the user 140. The time-track expansion mask may include an accompaniment mask and intermediate audio codes between the start audio code 433-1 and the end audio code 433-2.

[0073] Finally, the electronic device 110 can combine the start audio code 433-1, the end audio code 433-2, and the time-track extension mask to obtain the intermediate code representation 435.

[0074] In other scenarios, such as Figure 4D As shown, Figure 4D This illustration shows a music generation method in an audio mixing scenario according to one embodiment of the present invention. The electronic device 110 can receive a music generation request, which can indicate audio segments from two different audio tracks to indicate a transition between the two track segments. The electronic device 110 can acquire reference music content, which includes a starting audio track segment 441-1 and an ending audio track segment 441-2. The starting audio track segment 441-1 indicates a vocal track, and the ending audio track segment 441-2 indicates an accompaniment track.

[0075] After acquiring the starting audio track segment 441-1 and the ending audio track segment 441-2, the electronic device 110 can input the starting audio track segment 441-1 and the ending audio track segment 441-2 into the audio encoder 442-1 and the audio encoder 442-2 respectively, to obtain the starting audio track code 443-1 and the ending audio track code 443-2. ​​The electronic device 110 can combine the starting audio track code 443-1 and the ending audio track code 443-2 into a first encoding representation.

[0076] Furthermore, based on the starting track code 443-1 and the ending track code 443-2, the electronic device 110 can determine that the vocal track needs to generate the subsequent part of the starting track code 443-1, and the accompaniment track needs to generate the preceding part of the ending track code 443-2. ​​Based on this, the electronic device 110 can generate a corresponding mixing mask to indicate the part of the track to be generated.

[0077] Finally, the electronic device 110 can fill the mixed mask into the part of the audio track to be generated, to obtain the intermediate encoded representation 435.

[0078] In some embodiments, the electronic device 110 may also generate a second coded representation based on the text content indicated by the music generation request, wherein the text content description includes lyrics, style information, timbre information, and melody information.

[0079] In box 230, electronic device 110 can use the target model to process the intermediate coded representation to generate a second coded representation, which is generated by replacing the target portion in the intermediate coded representation, the target portion being indicated by a target mask.

[0080] In some embodiments, the electronic device 110 may also generate a second coded representation based on the text content indicated by the music generation request, wherein the text content description includes lyrics, style information, timbre information and melody information.

[0081] For example, electronic device 110 can receive a music description input by user 140. When generating the second coded representation, electronic device 110 can input the music description content into the target model to instruct the target model to generate the second coded representation with reference to the music description content.

[0082] As an example, such as Figure 4A As shown, in the music enhancement scenario, the electronic device 110 can combine the acquired first lyrics content 407 with the second lyrics content to obtain the third lyrics content 408. Further, the electronic device 110 can input the third lyrics content 408 and the intermediate mask representation 406 into the target model 409 to obtain the second encoded representation. The second encoded representation includes the intro audio encoding 403, the subsequent audio encoding of the vocal track, and the audio encoding of the accompaniment track.

[0083] As an example, such as Figure 4B As shown, in the audio track conversion scenario, the electronic device 110 can input the second lyrics content 426 and the intermediate encoded representation 425 input by the user into the target model 427 to obtain the second encoded representation. The second encoded representation includes the accompaniment audio track encoding 424 and the human voice track generated by the target model 427 based on the audio track mask and the second lyrics content 426.

[0084] As an example, such as Figure 4C As shown, in the accompaniment expansion scenario, the electronic device 110 can input the intermediate surface representation 435 and the third lyrics content 436 input by the user into the target model to obtain the second encoded representation. The second encoded representation includes the starting audio encoding 433-1, the ending audio encoding 433-2, and the audio encoding of one or more tracks generated by the target model 437 based on the time-track expansion mask.

[0085] As an example, such as Figure 4D As shown, in an audio mixing scenario, the electronic device 110 can input the intermediate surface representation 435 and the third lyrics content 436 input by the user into the target model to obtain a second encoded representation. The second encoded representation includes the starting audio track encoding 443-1, the ending audio track encoding 443-2, the audio track encoding of the preceding part of the ending audio track encoding 443-2 generated by the target model 447 based on the mixing mask, and the audio track encoding of the subsequent part of the starting audio track encoding 443-1 generated by the target model 447 based on the mixing mask.

[0086] Continue to refer to Figure 2 In box 240, electronic device 110 can generate target music content based on the second encoded representation.

[0087] Specifically, the electronic device 110 can invoke any audio decoder to decode the second encoded representation to generate the target music content. This disclosure is not intended to limit the specific decoding process.

[0088] In some embodiments, the target model may be trained based on the following process: First, training audio track data corresponding to multiple audio tracks of the music sample is determined; further, the training audio track data is encoded to generate a first audio feature; then, a second audio feature is generated based on a constructed training mask and the first audio feature, the training mask indicating the audio track and / or audio track segment to be generated; further, the second audio feature is processed using the target model to generate a third audio feature; finally, the target model is trained based on a comparison of the first audio feature and the third audio feature.

[0089] In some embodiments, the third audio feature is also generated based on the text content associated with the music sample.

[0090] The training process of the target model discussed above will be further described below. Figure 5 The diagram illustrates a training process 500 for a target model according to an embodiment of the present invention. This training process 500 can be performed by a suitable training device, which may be the same as or different from the electronic device 110.

[0091] like Figure 5 As shown, the training device can acquire audio data 510 and text data 550 for training the target model 570. For the text data 550, the training device can encode the text data 550 through a text encoder 560 to obtain text features 555.

[0092] For audio data 510, the training device inputs the audio data into the track separation model 520 to separate multiple tracks of the audio data 510, obtaining multi-track audio data 514 (i.e., training track data). Further, the electronic device 100 can call the audio encoder 530 to encode the multi-track audio data 514, obtaining multi-track audio features 516 (i.e., first audio features). Even further, the training device can perform multi-functional masking processing 540 on the multi-track audio features 516 using different types of masks to obtain the audio training features of the target model 570 (i.e., second audio features).

[0093] Furthermore, the training device can input the audio training data and text features 555 into the target model 570 to train the target model 570, obtaining multitrack audio features 575 generated by the target model 570. Finally, the training device can calculate the loss function 580 of the target model 570 based on the multitrack audio features 516 and the generated multitrack audio features 575 (i.e., the third audio feature). The training device can further adjust the parameters of the target model 570 according to the loss function 580.

[0094] In this way, the embodiments of this disclosure can construct a corresponding mask matrix based on the music generation request, and can further utilize the mask matrix to generate corresponding music content. Therefore, the embodiments of this disclosure can enrich music generation scenarios and improve the quality of music content.

[0095] Furthermore, embodiments of this disclosure utilize a mask matrix to indicate the target portion in the intermediate encoded representation corresponding to the content to be generated. Therefore, embodiments of this disclosure can ensure that other audio data are not interfered with during music generation, thus maintaining the consistency of the overall music style.

[0096] Example devices and equipment

[0097] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6A schematic structural block diagram of an apparatus 600 for music generation according to certain embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0098] like Figure 6 As shown, the apparatus 600 includes a target mask generation module 610, configured to construct a target mask corresponding to a received music generation request; an intermediate encoding representation generation module 620, configured to generate an intermediate encoding representation based on the target mask and a first encoding representation of reference music content; a second encoding representation generation module 630, configured to process the intermediate encoding representation using a target model to generate a second encoding representation, wherein the second encoding representation is generated by replacing a target portion in the intermediate encoding representation, the target portion being indicated by the target mask; and a music content generation module 640, configured to generate target music content based on the second encoding representation.

[0099] In some embodiments, the target mask generation module 610 is further configured to: determine at least one audio track and / or at least one audio track fragment to be generated based on a music generation request; and construct a target mask based on at least one audio track and / or at least one audio track fragment, wherein the mask portion of the target mask corresponding to at least one audio track and / or at least one audio track fragment is set to a preset value.

[0100] In some embodiments, the target mask generation module 610 is further configured to: determine newly added first lyrics content based on a music generation request; determine a first time length corresponding to the first lyrics content based on the speech rate information of the reference music content; and determine that at least one audio track segment to be generated includes a first audio track segment in the human voice track with the first time length.

[0101] In some embodiments, the target mask generation module 610 is further configured to: determine the newly added second lyrics content based on the music generation request; and determine that at least one audio track to be generated includes a human voice track in response to the reference music content not including a human voice track.

[0102] In some embodiments, the target mask generation module 610 is further configured to: obtain reference music content based on a music generation request, the reference music content including multiple audio track segments corresponding to the target audio track; and determine that at least one audio track segment to be generated includes an audio track segment corresponding to a preset time period of the target audio track.

[0103] In some embodiments, the preset time period includes: a user-specified time period; or a time period determined based on third lyrics content indicated by the music generation request.

[0104] In some embodiments, the music generation request also indicates a set of audio tracks to be added.

[0105] In some embodiments, the target mask generation module 610 is further configured to: obtain reference music content based on a music generation request, wherein the reference music content includes at least a second audio track segment corresponding to the first audio track and a third audio track segment corresponding to the second audio track; and determine that at least one audio track segment to be generated includes a fourth audio track segment corresponding to the first audio track and a fifth audio track segment corresponding to the second audio track.

[0106] In some embodiments, the first audio track is a human voice track, and the duration of the fourth audio track segment is determined based on the fourth lyrics content indicated by the music generation request.

[0107] In some embodiments, the target mask corresponds to a mask pattern determined based on a music generation request, the mask pattern including: a first mask pattern indicating the generation of subsequent audio track data corresponding to an existing audio track based on reference music content; a second mask pattern indicating the generation of additional audio track data corresponding to a new audio track based on reference music content; a third mask pattern indicating the generation of subsequent audio track data corresponding to at least one existing audio track and additional audio track data corresponding to a new audio track based on reference music content; a fourth mask pattern indicating the generation of additional audio track data corresponding to multiple audio tracks based on existing audio track data of multiple audio tracks of reference music content, wherein the additional audio track data of different audio tracks corresponds to different time periods; or a fifth mask pattern indicating the generation of intermediate audio track data corresponding to at least one existing audio track and additional audio track data corresponding to a new audio track based on reference music content.

[0108] In some embodiments, the apparatus 600 further includes a first encoded representation generation module configured to: in response to the reference audio content including multiple audio tracks, use a separation module to determine audio track data corresponding to the multiple audio tracks; and use an audio encoder to process the audio track data corresponding to the multiple audio tracks to determine a first encoded representation.

[0109] In some embodiments, the second encoding representation is also generated based on the text content indicated by the music generation request, the text content describing at least one of the following: lyrics, style information, timbre information, and melody information.

[0110] In some embodiments, the target model is trained based on the following process: determining training audio track data corresponding to multiple audio tracks of a music sample; encoding the training audio track data to generate a first audio feature; generating a second audio feature based on a constructed training mask and the first audio feature, the training mask indicating the audio track and / or audio track segment to be generated; processing the second audio feature using the target model to generate a third audio feature; and training the target model based on a comparison of the first audio feature and the third audio feature.

[0111] In some embodiments, the third audio feature is also generated based on text content associated with the music sample.

[0112] The units included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 600 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0113] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 Electronic devices 110.

[0114] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0115] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0116] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0117] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0118] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0119] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0120] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0122] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for music generation, comprising: constructing a mask matrix corresponding to a received music generation request based on the music generation request; generating an intermediate encoded representation based on the target mask and a first encoded representation of a reference music content; processing the intermediate encoded representation with a target model to generate a second encoded representation, the second encoded representation being generated by replacing a target portion in the intermediate encoded representation, the target portion being indicated by the target mask; and generating a target music content based on the second encoded representation. 2.The method of claim 1, wherein the constructing a target mask corresponding to a received music generation request based on the music generation request comprises: determining at least one audio track and / or at least one audio track segment to be generated based on the music generation request; and constructing the target mask based on the at least one audio track and / or the at least one audio track segment, wherein a mask portion in the target mask corresponding to the at least one audio track and / or the at least one audio track segment is set to a preset value. 3.The method of claim 2, wherein the determining at least one audio track and / or at least one audio track segment to be generated based on the music generation request comprises: determining a first lyric content to be added based on the music generation request; determining a first time length corresponding to the first lyric content based on tempo information of the reference music content; and determining the at least one audio track segment to be generated comprises a first audio track segment in a vocal audio track with the first time length. 4.The method of claim 2, wherein the determining at least one audio track and / or at least one audio track segment to be generated based on the music generation request comprises: determining a second lyric content to be added based on the music generation request; and determining the at least one audio track to be generated comprises a vocal audio track in response to the reference music content not including the vocal audio track. 5.The method of claim 2, wherein the determining at least one audio track and / or at least one audio track segment to be generated based on the music generation request comprises: obtaining the reference music content comprising a plurality of audio track segments corresponding to a target audio track based on the music generation request; and determining the at least one audio track segment to be generated comprises an audio track segment corresponding to a preset time period of the target audio track. 6.The method of claim 5, wherein the preset time period comprises: a time period specified by a user; or a time period determined based on a third lyric content indicated by the music generation request. 7.The method of claim 5, wherein the music generation request further indicates a set of audio tracks to be added. 8.The method of claim 2, wherein the determining at least one audio track and / or at least one audio track segment to be generated based on the music generation request comprises: obtaining the reference music content comprising at least a second audio track segment corresponding to a first audio track and a third audio track segment corresponding to a second audio track based on the music generation request; and determining the at least one audio track segment to be generated comprises the second audio track segment and the third audio track segment. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ determining the at least one audio track segment to be generated comprises a fourth audio track segment corresponding to the first audio track and a fifth audio track segment corresponding to the second audio track. 9.The method of claim 8, wherein the first audio track is a human voice track, and a time length of the fourth audio track segment is determined based on fourth lyric content indicated by the music generation request. 10.The method of claim 1, wherein the target mask corresponds to a mask pattern determined based on the music generation request, the mask pattern comprising: a first mask pattern indicating to generate subsequent audio track data corresponding to an existing audio track based on the reference music content; a second mask pattern indicating to generate additional audio track data corresponding to a new audio track based on the reference music content; a third mask pattern indicating to generate subsequent audio track data corresponding to at least one existing audio track and additional audio track data corresponding to a new audio track based on the reference music content; a fourth mask pattern indicating to generate additional audio track data corresponding to a plurality of audio tracks of the reference music content based on existing audio track data of the plurality of audio tracks, the additional audio track data of different audio tracks corresponding to different time periods; or a fifth mask pattern indicating to generate intermediate audio track data corresponding to at least one existing audio track and additional audio track data corresponding to a new audio track based on the reference music content. 11.The method of claim 1, further comprising: determining audio track data corresponding to a plurality of audio tracks using a separation module in response to the reference audio content comprising the plurality of audio tracks; and processing the audio track data corresponding to the plurality of audio tracks using an audio encoder to determine the first encoded representation. 12.The method of claim 1, wherein the second encoded representation is further generated based on text content indicated by the music generation request, the text content describing at least one of: lyric content, style information, timbre information, melody information. 13.The method of claim 1, wherein the target model is trained based on a process comprising: determining training audio track data corresponding to a plurality of audio tracks of a music sample; encoding the training audio track data to generate first audio features; generating second audio features based on a constructed training mask and the first audio features, the training mask indicating audio tracks and / or audio track segments to be generated; processing the second audio features using the target model to generate third audio features; and training the target model based on a comparison of the first audio features and the third audio features. 14.The method of claim 13, wherein the third audio features are further generated based on text content associated with the music sample. 15.An apparatus for music generation, comprising: a target mask construction module configured to construct a target mask corresponding to a received music generation request based on the music generation request; an intermediate encoded representation generation module configured to generate an intermediate encoded representation based on the target mask and a first encoded representation of reference music content; and an audio generation module configured to generate a second encoded representation based on the intermediate encoded representation. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ a second encoding representation generation module configured to process the intermediate encoding representation with a target model to generate a second encoding representation, the second encoding representation being generated by replacing a target portion in the intermediate encoding representation, the target portion being indicated by the target mask; and a target music generation module configured to generate target music content based on the second encoding representation.

16. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1-14.

17. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-14.

Images