Method and apparatus for music editing, device, and storage medium

Abstract

A method and apparatus for music editing, a device, and a storage medium. The method comprises: receiving a music editing request for reference music content, wherein the editing request is used for instructing to edit a target portion of the reference music content (210); generating an intermediate encoded representation on the basis of the target portion to be edited and a first encoded representation of the reference music content (220); using a target model to generate a second encoded representation on the basis of the intermediate encoded representation, wherein the second encoded representation is generated by replacing a partial encoded representation corresponding to the target portion in the intermediate encoded representation (230); and generating target music content on the basis of the second encoded representation (240).

Description

Method, device, equipment and storage medium for music editing

[0001] The present application claims priority to the Chinese Patent Application No. 202411252042.4, filed on September 6, 2024, and entitled "Method, device, equipment and storage medium for music editing", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, device, equipment and computer readable storage medium for music editing. BACKGROUND

[0003] With the development of computer technology, music editing technology, as a frontier innovation in the field of artificial intelligence, is gradually changing the traditional mode of music creation. The traditional music editing method cannot guarantee not to affect the unedited audio area in the editing process, and is not convenient and flexible. SUMMARY

[0004] In a first aspect of the present disclosure, a method for music editing is provided. The method comprises: receiving a music editing request for a reference music content, the editing request indicating a target part of the reference music content to be edited; generating an intermediate coded representation based on the target part to be edited and a first coded representation of the reference music content; generating a second coded representation based on the intermediate coded representation using a target model, the second coded representation being generated by replacing a part coded representation in the intermediate coded representation corresponding to the target part; and generating a target music content based on the second coded representation.

[0005] In a second aspect of the present disclosure, a device for music editing is provided. The device comprises: a request receiving module configured to receive a music editing request for a reference music content, the editing request indicating a target part of the reference music content to be edited; an intermediate coded representation generating module configured to generate an intermediate coded representation based on the target part to be edited and a first coded representation of the reference music content; a second coded representation generating module configured to generate a second coded representation based on the intermediate coded representation using a target model, the second coded representation being generated by replacing a part coded representation in the intermediate coded representation corresponding to the target part; and a music content generating module configured to generate a target music content based on the second coded representation.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. The instructions, when executed by the at least one processor, cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] In a fifth aspect of the disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer- executable instructions that, when executed by a device, cause the device to perform the method according to the first aspect.

[0009] It should be understood that the contents described in this section are not intended to limit the key features or important features of the embodiments of the disclosure, nor are they used to limit the scope of the disclosure. Other features of the disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other features, advantages, and aspects of embodiments of the disclosure will become more apparent by describing in detail some embodiments thereof with reference to the annexed drawings in which:

[0011] FIG. 1 shows a schematic diagram of an example environment in which embodiments of the disclosure can be implemented;

[0012] FIG. 2 shows a flowchart of music editing according to some embodiments of the disclosure;

[0013] FIGS. 3A-3B show example diagrams of a mask matrix according to some embodiments of the disclosure;

[0014] FIGS. 4A-4B show flowcharts of editing music for different audio tracks according to some embodiments of the disclosure;

[0015] FIG. 5 shows a flowchart of training a music editing model according to some embodiments of the disclosure;

[0016] FIG. 6 shows a block diagram of an apparatus for music editing according to some embodiments of the disclosure; and

[0017] FIG. 7 shows a block diagram of an electronic device capable of implementing various embodiments of the disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be noted that the titles of any sections / sub-sections provided herein are not limiting. Various embodiments are described throughout this document and any type of embodiment can be included under any section / sub-section. Furthermore, embodiments described in any section / sub-section can be combined with any other embodiments described in the same section / sub-section and / or different section / sub-section in any manner.

[0020] In the description of embodiments of the present disclosure, the term "includes" and its derivatives, such as "including," should be understood in an open, inclusive sense, that is, "including, but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "an embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicitly and implicitly recited definitions can also be found below. The terms "first," "second," and the like can refer to different or identical objects. Other explicit and implicit definitions can also be found below.

[0021] Data of users, acquisition and / or use of data, etc. can be involved in embodiments of the present disclosure. These aspects all comply with corresponding laws and regulations and relevant provisions. In embodiments of the present disclosure, all collection, acquisition, processing, processing, forwarding, use, etc. of data are performed on the premise that users are aware of and confirm. Accordingly, when implementing embodiments of the present disclosure, the type of data or information that can be involved, the range of use, the scenario of use, etc. should be informed to users and authorized by users in a proper manner according to relevant laws and regulations. The specific informing and / or authorization manner can vary according to actual situations and application scenarios, and the scope of the present disclosure is not limited in this respect.

[0022] In the present specification and embodiments, if personal information processing is involved, it will be processed on the premise of legality (for example, obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and only within the prescribed or agreed range. Users refuse to process personal information other than the necessary information required for basic functions, which will not affect the user's use of basic functions.

[0023] The data involved in the schemes in the specification and examples, such as the training and inference of the model, including but not limited to the data itself, the acquisition and / or use of the data, complies with the requirements of the corresponding laws and regulations.

[0024] As briefly mentioned above, with the development of computer technology, music editing technology, as a frontier innovation in the field of artificial intelligence, is gradually changing the traditional mode of music creation. Traditional music editing methods cannot guarantee that the unedited audio area will not be affected during editing, and are not convenient and flexible.

[0025] To this end, an embodiment of the present disclosure proposes a scheme for music editing. According to various embodiments of the present disclosure, a music editing request for a reference music content is received, the editing request indicating a target part of the reference music content to be edited; based on the target part to be edited and a first encoded representation of the reference music content, an intermediate encoded representation is generated; a target model is used to generate a second encoded representation based on the intermediate encoded representation, the second encoded representation being generated by replacing the part encoded representation corresponding to the target part in the intermediate encoded representation; and based on the second encoded representation, a target music content is generated.

[0026] In this way, the embodiments of the present disclosure can implement editing of music content by modifying the encoded representation, thereby improving the efficiency of music editing.

[0027] The example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0028] Example environment

[0029] FIG. 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in FIG. 1, the example environment 100 can include an electronic device 110 and a target model 136.

[0030] In the environment 100 of FIG. 1, the electronic device 110 can receive input data 130 from a user 140, the input data 130 can include audio data and / or text data. Further, the electronic device 110 can invoke the target model 136 to process the input data 130 to obtain target audio 120. The target model may, for example, be a non-causal deep learning model such as a diffusion model, which can generate synthesized audio content based on input text content or lyrics content.

[0031] In some embodiments, the electronic device 110 can include various types of computing systems / servers capable of providing computing capabilities, and the electronic device 110 can include an end device. Such an end device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile handset, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), a voice / video recorder, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. The electronic device 110 may, for example, include various types of computing systems / servers capable of providing computing capabilities, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, and the like. Although shown as a single device, the electronic device 110 can include multiple physical devices.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0033] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0034] Music editing process example

[0035] FIG. 2 shows a flowchart of a music editing process 200 according to some embodiments of the present disclosure. The process 200 can be implemented at the electronic device 110. The process 200 is described below with reference to FIG. 1.

[0036] As shown in FIG. 2, at block 210, the electronic device 110 can receive a music editing request for reference music content, the editing request indicating a target portion of the reference music content to be edited.

[0037] In some scenarios, as shown in FIG. 3A, based on receiving a single-track music editing request for reference music content, the electronic device 110 can determine a target portion 310-1 in the reference music content 310 that needs to be edited. After the electronic device 110 determines the target portion, a corresponding mask matrix can be generated based on the target portion to indicate the audio region that needs to be edited by the target model 136. The target portion that needs to be edited can be specified by a user or calculated by the electronic device 110 based on the received corresponding data, without limitation of the present disclosure.

[0038] In some scenarios, as shown in FIG. 3A, the electronic device 110 can determine a target portion 310-1 in the reference music content that needs to be edited based on receiving a multi-track music editing request for the reference music content. The target portion 310-1 includes a partial audio region of a main track and a partial audio region of another track.

[0039] With continued reference to FIG. 2, at block 220, the electronic device 110 can generate an intermediate encoded representation based on the target portion to be edited and the first encoded representation of the reference music content.

[0040] As an example, as shown in FIG. 3A and FIG. 3B, FIG. 3A and FIG. 3B illustrate an example process in which the electronic device 110 generates an intermediate encoded representation. Taking single-track editing as an example, the electronic device 110 can overlay a mask matrix generated based on the target portion to the target portion 310-1 in the reference music content 310, thereby generating an intermediate encoded representation 320.

[0041] In some embodiments, the electronic device 110 can construct a target mask based on the target portion to be edited, in which a mask portion corresponding to the target portion is set to a preset value. As an example, as shown in FIG. 3, after determining the target portion 310-1 to be edited, the electronic device 110 can generate a target mask 320-1 corresponding to the target portion. The value in the target mask 320-1 can be set to 0 to indicate that the target model 136 should edit the audio region corresponding to the target mask. The target portion can be a partial region in a single track, such as a partial region in a vocal track, or a partial region in multiple tracks, such as a partial region in a vocal track and an accompaniment track.

[0042] Further, the electronic device 110 can generate an intermediate encoded representation based on the target mask and the first encoded representation of the reference music content. As an example, after the electronic device 110 generates the target mask based on the target portion, the electronic device 110 can generate an intermediate mask representation 320 as shown in FIG. 3.

[0043] With continued reference to FIG. 2, at block 230, the electronic device 110 can generate a second encoded representation based on the intermediate encoded representation using the target model, the second encoded representation being generated by replacing a portion of the intermediate encoded representation corresponding to the target portion.

[0044] Specifically, the target model 136, upon receiving the intermediate coded representation, can determine the audio encoding to be generated (i.e., the partial coded representation corresponding to the target portion) based on the mask matrix in the intermediate coded representation. Further, the target model 136 can generate the target portion in the intermediate coded representation based on the first coded representation and the determined audio encoding to be generated. Finally, the target model 136 can combine or replace the first coded representation and the generated target portion in the intermediate coded representation to obtain the second coded representation.

[0045] In some embodiments, the electronic device 110 can further generate the second coded representation based on text content indicated by the music generation request, the text content describing including lyric content, style information, timbre information, and melody information.

[0046] With reference back to FIG. 2, at block 240, the electronic device 110 can generate the target music content based on the second coded representation. Specifically, the electronic device 110 can invoke an arbitrary audio decoder to decode the second coded representation to generate the target music content.

[0047] In some embodiments, the electronic device 110 can determine the at least one audio track segment to be edited based on the music editing request. Taking single-audio track editing as an example, upon receiving the music editing request, the electronic device 110 can determine the audio track segment to be edited based on the audio track indicated by the music editing request. For example, as shown in FIG. 3A, the audio track segment to be edited can be, for example, a partial audio track segment in the main audio track.

[0048] Further, the electronic device 110 can determine the target portion of the reference music content to be edited based on the at least one audio track segment to be edited. Taking single-audio track editing as an example, as shown in FIG. 3A, the electronic device 110 can determine the region in the main audio track where the target portion 310-1 to be edited is located based on the determined audio track segment to be edited in the main audio track.

[0049] After determining the target portion of the reference music content to be edited, the target model 136 can edit the target portion without wasting time on irrelevant portions, improving the editing efficiency. This targeted music editing manner can also reduce unnecessary modifications and attempts, making the editing process more smooth and efficient.

[0050] In some embodiments, the electronic device 110 can determine the first lyric content based on the music editing request. The user can indicate, for example, the lyric content corresponding to the target portion to be edited.

[0051] Accordingly, the electronic device 110 can determine a target lyric part in the second lyric content based on the first lyric content and the second lyric content of the reference music lyric content. As an example, the electronic device 110 can determine a difference part between the first lyric content and the second lyric content based on the first lyric content and the second lyric content, and take the difference part as the target lyric part.

[0052] Further, the electronic device 110 can determine a first time period corresponding to the target lyric part. As an example, the electronic device 110 can determine the first time period corresponding to the target lyric part based on a time point at which an initial lyric part corresponding to the target lyric part in the second lyric content is located in the reference music content.

[0053] Finally, the electronic device 110 can determine at least one audio track segment to be edited based on the first time period. As an example, the electronic device 110 can determine an audio track segment corresponding to the first time period as the audio track segment to be edited based on the first time period, in which the reference music content matches the audio track segment corresponding to the first time period.

[0054] In some embodiments, the electronic device 110 can determine speech speed information associated with the reference music content. In some scenarios, the speech speed information can indicate an average speech speed in the reference music content corresponding to the target lyric part. For example, the electronic device 110 can determine the singing speed in the reference music content by sampling.

[0055] Accordingly, the electronic device 110 can adjust the first time period based on the speech speed information and the first lyric content to determine a second time period. For example, if the number of words of the lyrics in the first lyric content corresponding to the target lyric part is more than the number of words of the target lyric part, the electronic device 110 can appropriately increase the time length of the first time period to determine the second time period associated with the first time period after the time length is increased. The time length of the first time period that is increased can be determined by the electronic device 110 according to the number of words that are increased in the first lyric content, or can be determined by the electronic device 110 through an algorithm and / or model for counting the time length.

[0056] Further, the electronic device 110 can determine at least one audio track segment to be edited based on the second time period. As an example, the electronic device 110 can match the second time period with a time period associated with the reference music content, find a third time period in the reference music content that matches the second time period, and take an audio track segment corresponding to the third time period as the audio track segment to be edited. The audio track segment to be edited can be a single-audio-track audio track segment or a multi-audio-track audio track segment, which can be determined by the electronic device 110 according to a music editing request received by the electronic device 110.

[0057] As an example, the electronic device 110 can obtain the custom lyrics (i.e., the first lyrics content) input by the user 140. After obtaining the custom lyrics, the electronic device 110 can align the custom lyrics with the original lyrics (i.e., the second lyrics content) in the reference music content, and determine the to-be-edited part in the original lyrics. Further, the electronic device 110 can adjust the target part to be edited in the reference music content based on the average speed of the to-be-edited part and the speed at which the custom lyrics need to be filled in the to-be-edited part. Thus, the generated target music content is ensured to be consistent with the speed of the reference music content in the case of modifying the number of lyrics, and the effect of the generated target music content is better. In some scenarios, the electronic device 110 can also obtain audio data input by the user 140 to adjust the target part to be edited in the reference music content based on the audio data. The audio data may, for example, be vocal melody data, instrument melody data, chord melody data, and the like.

[0058] In some embodiments, the electronic device 110 can determine the number of lyrics for replacing the target lyrics part in the first lyrics content. As an example, the electronic device 110 can match the first lyrics content with the second lyrics content to determine the number of lyrics for replacing the target lyrics part.

[0059] Further, the electronic device 110 can adjust the first time period to determine the second time period based on the number of lyrics and the speed information. As an example, the electronic device 110 can determine the time required for each word in the lyrics based on the speed information. Accordingly, the electronic device 110 can determine the time period required for the lyrics for replacing the target lyrics part based on the time required for each word. Finally, the electronic device 110 can adjust the first time period to determine the second time period based on the time period required for the lyrics for replacing the target lyrics part. In some scenarios, the electronic device 110 can also call a related model to determine the time required for each word in the lyrics through the related model. The related model may, for example, be any model for calculating the time required for each word in the lyrics.

[0060] In some embodiments, the electronic device 110 can determine the time distribution of the second lyrics content based on the reference music content, the time distribution indicating the time of a group of characters in the second lyrics content. For example, the second lyrics content is “apple, orange, red, color”, and the electronic device 110 can respectively determine the time nodes of “apple”, “orange”, “red”, and “color” in the reference music content.

[0061] Further, the electronic device 110 can determine, based on the time distribution, a first time of a starting character and a second time of an ending character of the target lyric portion. For example, the electronic device 110 can determine the first time corresponding to "apple" and the second time corresponding to "color".

[0062] Finally, the electronic device 110 can determine, based on the first time and the second time, a first time period corresponding to the target lyric portion. For example, the first time is one minute and twenty seconds, the second time is three minutes and fifteen seconds, and the first time period is a time period between one minute and twenty seconds and three minutes and fifteen seconds.

[0063] In some embodiments, the electronic device 110 can process the reference music content by using a separation module to obtain audio track data corresponding to a plurality of audio tracks. As an example, the separation module can include, for example, an audio track separation module. The electronic device 110 can input the reference music content into the audio track separation module to obtain the audio track data corresponding to the plurality of audio tracks in the reference music content.

[0064] Further, the electronic device 110 can encode the audio track data corresponding to the plurality of audio tracks by using an audio encoder to determine a first encoded representation. Specifically, the electronic device 110 can obtain, by the audio encoder, audio encoded representations corresponding to the plurality of audio tracks, and combine the audio encoded representations corresponding to the plurality of audio tracks to obtain the first encoded representation.

[0065] Different music editing processes corresponding to different music editing scenarios will be described below in connection with FIGS. 4A-4B.

[0066] FIG. 4A illustrates an example process of multi-audio track editing in an embodiment of the present application. The electronic device 110 can receive a music editing request, which can indicate, for example, editing all audio tracks. As an example, as shown in FIG. 4A, the electronic device 110 can receive reference music content 410. After receiving the reference music content 410, the electronic device 110 inputs the reference music content 410 into an audio separation model 420 and a text recognition model 430 to obtain a first encoded representation 422 and lyric time points 435 associated with the reference music content, respectively. The lyric time points 435 include time points corresponding to each character in the lyrics of the reference music content.

[0067] Further, the electronic device 110 can receive a custom lyric 440 (i.e., first lyric content) input by the user 140, and perform audio-text alignment between the custom lyric 440 and the lyric time points 435 to determine a target portion to be edited in the first encoded representation 422. The electronic device 110 can generate a first mask matrix based on the target portion to be edited in the first encoded representation 422, and add the first mask matrix to the target portion in the first encoded representation 422 to obtain an initial intermediate encoded representation 424.

[0068] Still further, the electronic device 110 can also obtain the start time and end time of the target lyric paragraph to be edited in the lyrics of the reference music content when performing audio-text alignment of the custom lyrics 440 and the lyric time points 435. The electronic device 110 can adjust the length of the target portion to be edited in the first encoded representation 422 according to the average speech speed of the target lyric paragraph and the number of words to be filled in the target lyric paragraph. The electronic device 110 can construct a second mask matrix 460 based on the audio sampling rate and the length of the target portion to be edited in the adjusted first encoded representation 422. The electronic device 110 can then replace the first mask matrix in the initial intermediate encoded representation 424 with the second mask matrix to obtain a final intermediate encoded representation 426.

[0069] Finally, the electronic device 110 can invoke the target model 470 to process the final intermediate encoded representation 426 and the custom lyrics 440 input by the user 140 to obtain a second encoded representation. The electronic device 110 can invoke the audio decoder 480 to decode the second encoded representation to obtain the target music content 490.

[0070] FIG. 4B illustrates an example process of specifying a track for editing in an embodiment of the present application. The electronic device 110 can receive a music editing request, which can indicate, for example, to edit a specific track.

[0071] As an example, as shown in FIG. 4B, the electronic device 110 can receive a reference music content 410, input the reference music content 410 into the audio separation model 420 and the text recognition model 430 to obtain a first encoded representation 422 and lyric time points 435 associated with the reference music content, respectively.

[0072] Further, the electronic device 110 can receive custom lyrics 440-1 input by the user 140 and a specified track 440-2 to be edited. The electronic device 110 can perform audio-text alignment of the custom lyrics 440-1, the specified track 440-2 to be edited, and the lyric time points 435 to determine a target portion to be edited in the first encoded representation 422. In some scenarios, the electronic device 110 can also determine the target portion to be edited in the first encoded representation 422 based on audio data input by the user 140.

[0073] Still further, the electronic device 110 can construct a mask matrix based on the audio sampling rate and the target portion to be edited in the first encoded representation 422, and add the mask matrix to the target portion of the first encoded representation 422 to obtain an intermediate encoded representation 460.

[0074] Finally, the electronic device 110 can invoke the target model 470 to process the intermediate coded representation 460 and the custom lyrics 440 input by the user 140 to obtain a second coded representation. The electronic device 110 can invoke the audio decoder 480 to decode the second coded representation to obtain the target music content 490.

[0075] In some embodiments, the target model can be trained based on the following procedure: first, determining training track data corresponding to a plurality of tracks of a music sample; further, encoding the training track data to generate first audio features; then, generating second audio features based on a constructed training mask and the first audio features, the training mask indicating tracks and / or track segments to be generated; further, processing the second audio features using the target model to generate third audio features; finally, training the target model based on a comparison of the first audio features and the third audio features.

[0076] In some embodiments, the third audio features are further generated based on text content associated with the music sample.

[0077] The training procedure of the target model discussed above will be further introduced below. FIG. 5 illustrates a training procedure 500 of a target model of an embodiment of the present application. The training procedure 500 can be performed by a suitable training device, which can be the same as or different from the electronic device 110.

[0078] As shown in FIG. 5, the training device can obtain audio data 510 and text data 550 for training of the target model 570. For the text data 550, the training device can encode the text data 550 through a text encoder 560 to obtain text features 555.

[0079] For the audio data 510, the training device inputs the audio data to a track separation model 520 to separate a plurality of tracks of the audio data 510 to obtain multi-track audio data 514 (i.e., training track data). Further, the electronic device 100 can invoke an audio encoder 530 to encode the multi-track audio data 514 to obtain multi-track audio features 516 (i.e., first audio features). Further, the training device can perform multi-functional mask processing 540 on the multi-track audio features 516 through different kinds of masks to obtain audio training features of the target model 570 (i.e., second audio features).

[0080] Further, the training device can input the audio training data and the text features 555 into the target model 570 to train the target model 570, to obtain the multi-track audio features 575 generated by the target model 570. Finally, the training device can calculate the loss function 580 of the target model 570 based on the multi-track audio features 516 and the generated multi-track audio features 575 (i.e., the third audio features). The training device can further adjust the parameters of the target model 570 according to the loss function 580.

[0081] In this way, the embodiments of the present disclosure can indicate the target region that needs to be edited in the reference music content by the mask matrix, enable the music editing model to edit only the target part of the reference music content, and thus improve the efficiency of music editing.

[0082] Example apparatus and device

[0083] Embodiments of the present disclosure also provide a corresponding apparatus for implementing the above method or process. FIG. 6 shows a schematic structural block diagram of an apparatus 600 for music editing according to some embodiments of the present disclosure. The apparatus 600 can be implemented as or included in the electronic device 110. Various modules / components in the apparatus 600 can be implemented by hardware, software, firmware, or any combination thereof.

[0084] As shown in FIG. 6, the apparatus 600 includes an editing request receiving module 610 configured to receive a music editing request for a reference music content, the editing request indicating a target part of the reference music content to be edited; an intermediate coded representation generating module 620 configured to generate an intermediate coded representation based on the target part to be edited and a first coded representation of the reference music content; a second coded representation generating module 630 configured to generate a second coded representation based on the intermediate coded representation by using a target model, the second coded representation being generated by replacing a part coded representation corresponding to the target part in the intermediate coded representation; and a music content generating module 640 configured to generate a target music content based on the second coded representation.

[0085] In some embodiments, the apparatus 600 further includes a target part determining module configured to determine at least one audio track segment to be edited based on the music editing request; and determine the target part of the reference music content to be edited based on the at least one audio track segment to be edited.

[0086] In some embodiments, the target part determining module is further configured to determine a first lyric content based on the music editing request; determine a target lyric part in the second lyric content based on the first lyric content and a second lyric content of the reference music content; determine a first time period corresponding to the target lyric part; and determine the at least one audio track segment to be edited based on the first time period.

[0087] In some embodiments, the target portion determination module is further configured to: determine speech rate information associated with the reference music content; adjust the first time period based on the speech rate information and the first lyrics content to determine a second time period; and determine the at least one track segment to be edited based on the second time period.

[0088] In some embodiments, the speech rate information indicates an average speech rate in the reference music content corresponding to the target lyrics portion.

[0089] In some embodiments, the target portion determination module is further configured to: determine a number of lyrics words in the first lyrics content for replacing the target lyrics portion; and adjust the first time period based on the number of lyrics words and the speech rate information to determine a second time period.

[0090] In some embodiments, the target portion determination module is further configured to: determine a time distribution of the second lyrics content based on the reference music content, the time distribution indicating times of a set of characters in the second lyrics content; determine a first time of a starting character and a second time of an ending character of the target lyrics portion based on the time distribution; and determine the first time period corresponding to the target lyrics portion based on the first time and the second time.

[0091] In some embodiments, the intermediate coded representation generation module 620 is further configured to: construct a target mask based on the target portion to be edited, wherein a mask portion in the target mask corresponding to the target portion is set to a preset value; and generate the intermediate coded representation based on the target mask and the first coded representation of the reference music content.

[0092] In some embodiments, the apparatus 600 further comprises a first coded representation generation module configured to process the reference music content with a separation module to obtain track data corresponding to a plurality of tracks; and encode the track data corresponding to the plurality of tracks with an audio encoder to determine the first coded representation.

[0093] In some embodiments, the second coded representation is further generated based on text content indicated by the music editing request, the text content describing at least one of: lyrics content, style information, timbre information, melody information.

[0094] In some embodiments, the target model is trained based on a process of: determining training track data corresponding to a plurality of tracks of a music sample; encoding the training track data to generate first audio features; generating second audio features based on a constructed training mask and the first audio features, the training mask indicating tracks and / or track segments to be generated; processing the second audio features with the target model to generate third audio features; and training the target model based on a comparison of the first audio features and the third audio features.

[0095] In some embodiments, the third audio feature is further generated based on textual content associated with the music sample.

[0096] The units included in the apparatus 600 can be implemented utilizing various means including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, e.g., machine-executable instructions stored on a storage medium. In addition to or alternatively, some or all of the units in the apparatus 600 can be implemented at least partially by one or more hardware logic components. As an example and not by way of limitation, specific types of hardware logic components that can be used include a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SOC), a complex programmable logic device (CPLD), etc.

[0097] FIG. 7 shows a block diagram of an electronic device 700 in which one or more embodiments of the disclosure can be implemented. It should be understood that the electronic device 700 illustrated in FIG. 7 is merely an example and should not be construed as limiting on the functionality and scope of the embodiments described herein. The electronic device 700 illustrated in FIG. 7 can be used to implement the electronic device 110 of FIG. 1.

[0098] As shown in FIG. 7, the electronic device 700 is in the form of a general electronic device. Components of the electronic device 700 can include, but are not limited to, one or more processors 710 or processing units, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processor 710 can be a real or virtual processor and is capable of performing various processing according to programs stored in the memory 720. In a multi-processor system, multiple processors perform computer-executable instructions in parallel to improve parallel processing capabilities of the electronic device 700.

[0099] The electronic device 700 typically includes a number of computer storage media. Such media can be any available media that is accessible by the electronic device 700 and includes both volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 can be a removable or non-removable media and can include machine-readable media, such as a flash drive, a magnetic disk, or any other medium that can be used to store information and / or data and that can be accessed by the electronic device 700.

[0100] The electronic device 700 can further include additional detachable / non-detachable, volatile / non-volatile storage media. Although not shown in FIG. 7, a disk drive for reading from or writing to a detachable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a detachable, non-volatile optical disk (e.g., a CD-ROM) can be provided. In these cases, each drive can be connected to the bus (not shown) by one or more data media interfaces. The memory 720 can include a computer program product 725 having one or more program modules configured to carry out the various methods or acts of the various embodiments of the present disclosure.

[0101] The communication unit 740 enables communication with other electronic devices through communication media. Additionally, the functionality of the components of the electronic device 700 can be implemented in a single computing cluster or a plurality of computer machines capable of communicating over a communication connection. As such, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0102] The input device 750 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 760 can be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc., one or more devices that enable a user to interact with the electronic device 700, or any devices (e.g., a network card, a modem, etc.) that enable the electronic device 700 to communicate with one or more other electronic devices, through the communication unit 740, as desired. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0103] According to an example implementation of the present disclosure, a computer readable storage medium having computer executable instructions stored thereon is provided, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0104] Various aspects of the disclosure can be described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0105] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0106] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method of music editing, comprising receiving a music editing request for a reference music content, the editing request indicating a target portion of the reference music content to be edited; generating an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content; generating a second encoded representation based on the intermediate encoded representation by replacing a portion of the intermediate encoded representation corresponding to the target portion with a target model; and generating a target music content based on the second encoded representation. 2.The method of claim 1, further comprising: determining at least one audio track segment to be edited based on the music editing request; and determining the target portion of the reference music content to be edited based on the at least one audio track segment to be edited. 3.The method of claim 2, wherein determining at least one audio track segment to be edited based on the music editing request comprises: determining a first lyric content based on the music editing request; determining a target lyric portion in a second lyric content of the reference music content based on the first lyric content and the second lyric content; determining a first time period corresponding to the target lyric portion; and determining the at least one audio track segment to be edited based on the first time period. 4.The method of claim 3, wherein determining the at least one audio track segment to be edited based on the first time period comprises: determining a speech rate information associated with the reference music content; adjusting the first time period based on the speech rate information and the first lyric content to determine a second time period; and determining the at least one audio track segment to be edited based on the second time period. 5.The method of claim 4, wherein the speech rate information indicates an average speech rate in the reference music content corresponding to the target lyric portion. 6.The method of claim 5, wherein adjusting the first time period based on the speech rate information and the first lyric content to determine a second time period comprises: determining a number of words in the first lyric content to be replaced by the target lyric portion; and adjusting the first time period based on the number of words and the speech rate information to determine the second time period. 7.The method of claim 3, wherein determining a first time period corresponding to the target lyric portion comprises: determining a time distribution of the second lyric content based on the reference music content, the time distribution indicating times of a set of characters in the second lyric content; determining a first time of a starting character and a second time of an ending character of the target lyric portion based on the time distribution; and determining the first time period corresponding to the target lyric portion based on the first time and the second time. 8.The method of claim 1, wherein generating an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content comprises: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ constructing a target mask based on the target part to be edited, wherein a mask part corresponding to the target part in the target mask is set as a preset value; and generating the intermediate encoded representation based on the target mask and a first encoded representation of the reference music content.

9. The method of any one of claims 1-8, further comprising: processing the reference music content with a separation module to obtain track data corresponding to a plurality of audio tracks; and encoding the track data corresponding to the plurality of audio tracks with an audio encoder to determine the first encoded representation.

10. The method of claim 1, wherein the second encoded representation is further generated based on text content indicated by the music editing request, the text content describing at least one of: lyrics content, style information, timbre information, melody information.

11. The method of claim 1, wherein the target model is trained based on a process comprising: determining training track data corresponding to a plurality of audio tracks of a music sample; encoding the training track data to generate first audio features; generating second audio features based on a constructed training mask and the first audio features, the training mask indicating audio tracks and / or audio track segments to be generated; processing the second audio features with the target model to generate third audio features; and training the target model based on a comparison of the first audio features and the third audio features.

12. The method of claim 11, wherein the third audio features are further generated based on text content associated with the music sample.

13. An apparatus for music generation, comprising: a request receiving module configured to receive a music editing request for a reference music content, the editing request indicating a target part of the reference music content to be edited; an intermediate encoded representation generating module configured to generate an intermediate encoded representation based on the target part to be edited and a first encoded representation of the reference music content; a second encoded representation generating module configured to generate a second encoded representation based on the intermediate encoded representation with a target model, the second encoded representation being generated by replacing a part encoded representation in the intermediate encoded representation corresponding to the target part; a music content generating module configured to generate a target music content based on the second encoded representation.

14. An electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, cause the electronic device to perform the method of any one of claims 1-12.

15. A computer-readable storage medium having computer-executable instructions stored thereon that are executable by a processor to implement the method of any one of claims 1-12.

16. A computer program product, the computer program product being tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 12.

Images