Music editing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a music editing method and device, equipment and a storage medium. The method comprises the following steps: receiving a music editing request for reference music content, wherein the editing request indicates to edit a target part of the reference music content; generating an intermediate encoded representation based on the target portion to be edited and the first encoded representation of the reference music content; generating a second coded representation based on the intermediate coded representation by using the target model, wherein the second coded representation is generated by replacing a partial coded representation corresponding to the target part in the intermediate coded representation; and generating the target music content based on the second coded representation. Therefore, according to the embodiment of the invention, the music content editing efficiency is improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and computer-readable storage media for music editing. Background Technology

[0002] With the development of computer technology, music editing technology, as a cutting-edge innovation in the field of artificial intelligence, is gradually changing the traditional model of music creation. Traditional music editing methods cannot guarantee that unedited audio areas will not be affected during the editing process, and are not convenient or flexible enough. Summary of the Invention

[0003] In a first aspect of this disclosure, a music editing method is provided. The method includes: receiving a music editing request for reference music content, the editing request indicating editing a target portion of the reference music content; generating an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content; generating a second encoded representation based on the intermediate encoded representation using a target model, the second encoded representation being generated by replacing a portion of the intermediate encoded representation corresponding to the target portion; and generating target music content based on the second encoded representation.

[0004] In a second aspect of this disclosure, an apparatus for music editing is provided. The apparatus includes: a request receiving module configured to receive a music editing request for reference music content, the editing request indicating a target portion of the reference music content to be edited; an intermediate encoding representation generation module configured to generate an intermediate encoding representation based on the target portion to be edited and a first encoding representation of the reference music content; a second encoding representation generation module configured to generate a second encoding representation based on the intermediate encoding representation using a target model, the second encoding representation being generated by replacing a portion of the intermediate encoding representation corresponding to the target portion; and a music content generation module configured to generate target music content based on the second encoding representation.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A flowchart of music editing according to some embodiments of this disclosure is shown;

[0011] Figures 3A to 3B An example diagram of a mask matrix according to some embodiments of the present disclosure is shown;

[0012] Figures 4A to 4B A flowchart illustrating the editing of music for different audio tracks according to some embodiments of this disclosure is shown;

[0013] Figure 5 A flowchart for training a music editing model according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A block diagram of an apparatus for music editing according to some embodiments of the present disclosure is shown; and

[0015] Figure 7 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0021] In this specification and the embodiments, if the training and inference of the model are involved, the data involved (including but not limited to the data itself, the acquisition and / or use of the data) comply with the requirements of relevant laws and regulations.

[0022] As briefly mentioned earlier, with the development of computer technology, music editing technology, as a cutting-edge innovation in the field of artificial intelligence, is gradually changing the traditional model of music creation. Traditional music editing methods cannot guarantee that unedited audio areas will not be affected during the editing process, and are not convenient or flexible enough.

[0023] Therefore, embodiments of this disclosure propose a scheme for music editing. According to various embodiments of this disclosure, a music editing request for reference music content is received, the editing request instructing editing a target portion of the reference music content; an intermediate encoded representation is generated based on the target portion to be edited and a first encoded representation of the reference music content; a second encoded representation is generated based on the intermediate encoded representation using a target model, the second encoded representation being generated by replacing the portion of the intermediate encoded representation corresponding to the target portion; and the target music content is generated based on the second encoded representation.

[0024] In this way, the embodiments of this disclosure can achieve the editing of music content by modifying the encoded representation, thereby improving the efficiency of music editing.

[0025] Example embodiments of this disclosure are described below with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110 and target model 136.

[0028] exist Figure 1 In environment 100, electronic device 110 can receive input data 130 from user 140, which may include audio data and / or text data. Further, electronic device 110 can invoke target model 136 to process the input data 130 to obtain target audio 120. The target model may be, for example, a non-causal deep learning model such as a diffusion model, which can generate synthesized audio content based on the input text content or lyrics.

[0029] In some embodiments, electronic device 110 may include various types of computing systems / servers capable of providing computing power, and electronic device 110 may include terminal devices. Such terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), voice / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Electronic device 110 may, for example, include various types of computing systems / servers capable of providing computing power, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, etc. Although shown as a single device, electronic device 110 may include multiple physical devices.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Music editing process example

[0033] Figure 2 A flowchart of a music editing process 200 according to some embodiments of the present disclosure is shown. Process 200 can be implemented at an electronic device 110. Reference is made below. Figure 1 To describe process 200.

[0034] like Figure 2 As shown in block 210, electronic device 110 can receive a music editing request for reference music content, the editing request indicating the target part of the reference music content to be edited.

[0035] In some scenarios, such as Figure 3A As shown, the electronic device 110 can determine the target portion 310-1 that needs to be edited in the reference music content 310 based on receiving a single-track music editing request for the reference music content. After determining the target portion, the electronic device 110 can generate a corresponding mask matrix based on the target portion to indicate the audio region that needs to be edited in the target model 136. The target portion to be edited can be specified by the user or calculated by the electronic device 110 based on the corresponding received data; this invention does not impose any limitations on this.

[0036] In other scenarios, such as Figure 3BAs shown, the electronic device 110 can also determine the target portion 330-1 of the reference music content that needs to be edited based on receiving a multi-track music editing request for the reference music content. The target portion 330-1 includes a portion of the audio area of ​​the main track and portions of the audio areas of other tracks.

[0037] Continue to refer to Figure 2 In box 220, electronic device 110 can generate an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content.

[0038] As an example, such as Figure 3A and Figure 3B As shown, Figure 3A and Figure 3B An example process for generating an intermediate encoded representation by electronic device 110 is shown. Taking single-track editing as an example, electronic device 110 can overlay a mask matrix generated based on the target portion onto the target portion 310-1 in the reference music content 310, thereby generating an intermediate encoded representation 320.

[0039] In some embodiments, the electronic device 110 can construct a target mask based on the target portion to be edited, wherein the mask portion corresponding to the target portion is set to a preset value. As an example, as shown in FIG3, after determining the target portion 310-1 to be edited, the electronic device 110 can generate a target mask 320-1 corresponding to the target portion. The value in the target mask 320-1 can be set to 0 to indicate that the target model 136 should edit the audio region corresponding to the target mask. The target portion can be a portion of a single audio track, such as a portion of a vocal track, or a portion of multiple audio tracks, such as a portion of a vocal track and a accompaniment track.

[0040] Furthermore, the electronic device 110 can generate an intermediate coded representation based on the target mask and the first coded representation of the reference music content. As an example, after generating the target mask based on the target portion, the electronic device 110 can generate an intermediate mask representation 320 as shown in Figure 3.

[0041] Continue to refer to Figure 2 In box 230, electronic device 110 can use the target model to generate a second encoding representation based on the intermediate encoding representation, the second encoding representation being generated by replacing the portion of the intermediate encoding representation corresponding to the target portion.

[0042] Specifically, after receiving the intermediate encoded representation, the target model 136 can determine the audio code to be generated (i.e., the partial encoded representation corresponding to the target part) based on the mask matrix in the intermediate encoded representation. Further, the target model 136 can generate the target part in the intermediate encoded representation based on the first encoded representation and the determined audio code to be generated. Finally, the target model 136 can combine or replace the target part in the first encoded representation and the generated intermediate encoded representation to obtain the second encoded representation.

[0043] In some embodiments, the electronic device 110 may also generate a second coded representation based on the text content indicated by the music generation request, wherein the text content description includes lyrics, style information, timbre information and melody information.

[0044] Continue to refer to Figure 2 In box 240, electronic device 110 can generate target music content based on the second encoded representation. Specifically, electronic device 110 can invoke any audio decoder to decode the second encoded representation to generate the target music content.

[0045] In some embodiments, the electronic device 110 can determine at least one audio track segment to be edited based on a music editing request. Taking single-track editing as an example, when the electronic device 110 receives a music editing request, it can determine the audio track segment to be edited based on the audio track indicated in the music editing request. For example, as Figure 3A As shown, the audio track segment to be edited can be, for example, a portion of the main audio track.

[0046] Furthermore, the electronic device 110 can determine the target portion of the reference music content to be edited based on at least one audio track segment to be edited. Continuing with the example of single-track editing, such as... Figure 3A As shown, the electronic device 110 can determine the area of ​​the target part 310-1 to be edited in the main audio track based on the determined audio track segment to be edited in the main audio track.

[0047] Once the target portion of the reference audio content to be edited is determined, the target model 136 can edit that target portion without wasting time on irrelevant parts, thus improving editing efficiency. This targeted music editing method also reduces unnecessary modifications and attempts, making the editing process smoother and more efficient.

[0048] In some embodiments, the electronic device 110 may determine the first lyrics content based on a music editing request. For example, the user may specify the lyrics content corresponding to the target section to be edited.

[0049] Accordingly, the electronic device 110 can determine the target lyric portion in the second lyric content based on the first lyric content and the second lyric content referencing the musical lyric content. As an example, the electronic device 110 can determine the difference between the first and second lyric content based on the first and second lyric content, and use the difference portion as the target lyric portion.

[0050] Furthermore, the electronic device 110 can determine a first time period corresponding to the target lyrics portion. As an example, the electronic device 110 can determine the first time period corresponding to the target lyrics portion based on the time point in the reference music content where the initial lyrics portion corresponding to the target lyrics portion in the second lyrics content is located.

[0051] Finally, the electronic device 110 can determine at least one audio track segment to be edited based on the first time period. As an example, the electronic device 110 can match the audio track segment corresponding to the first time period with the reference music content based on the first time period, and use the audio track segment corresponding to the first time period as the audio track segment to be edited.

[0052] In some embodiments, the electronic device 110 can determine speech rate information associated with reference musical content. In some scenarios, the speech rate information can indicate the average speech rate corresponding to the target lyrics portion of the reference musical content. For example, the electronic device 110 can determine the singing speech rate in the reference musical content by sampling.

[0053] Accordingly, the electronic device 110 can adjust the first time period based on the speech rate information and the first lyrics content to determine the second time period. For example, if the number of characters in the lyrics corresponding to the target lyrics section in the first lyrics content is greater than the number of characters in the target lyrics section, the electronic device 110 can appropriately increase the length of the first time period to determine the second time period associated with the increased first time period. The increased length of the first time period can be determined by the electronic device 110 based on the increased number of characters in the first lyrics content, or it can be determined by the electronic device 110 through an algorithm and / or model used to calculate the length of the time period.

[0054] Furthermore, the electronic device 110 can determine at least one audio track segment to be edited based on the second time period. As an example, the electronic device 110 can match the second time period with a time period associated with reference music content, find a third time period in the reference music content that matches the second time period, and use the audio track segment corresponding to the third time period as the audio track segment to be edited. The audio track segment to be edited can be a single-track segment or a multi-track segment, specifically determined by the music editing request received by the electronic device 110.

[0055] As an example, electronic device 110 can acquire custom lyrics (i.e., first lyrics content) input by user 140. After acquiring the custom lyrics, electronic device 110 can align the custom lyrics with the original lyrics (i.e., second lyrics content) in the reference music content to determine the part to be edited in the original lyrics. Furthermore, electronic device 110 can adjust the target part to be edited in the reference music content based on the average speech rate of the part to be edited and the speed at which the custom lyrics need to be filled into the part to be edited. This ensures that even when the number of words in the lyrics is modified, the speech rate of the generated target music content is consistent with that of the reference music content, thereby improving the effect of the generated target music content. In some scenarios, electronic device 110 can also acquire audio data input by user 140 to adjust the target part to be edited in the reference music content based on the audio data. The audio data can be, for example, vocal melody data, instrumental melody data, chord melody data, etc.

[0056] In some embodiments, the electronic device 110 can determine the number of lyrics characters in the first lyrics content used to replace the target lyrics portion. As an example, the electronic device 110 can match the first lyrics content with second lyrics content to determine the number of lyrics characters used to replace the target lyrics portion.

[0057] Furthermore, the electronic device 110 can adjust the first time period to determine the second time period based on the number of words in the lyrics and the speech rate information. As an example, the electronic device 110 can determine the time required for each word in the lyrics based on the speech rate information. Accordingly, the electronic device 110 can determine the time period required for the lyrics used to replace the target lyrics section based on the time required for each word. Finally, the electronic device 110 can adjust the first time period based on the time period required for the lyrics used to replace the target lyrics section to determine the second time period. In some scenarios, the electronic device 110 can also invoke a correlation model to determine the time required for each word in the lyrics. The correlation model can be, for example, any model used to calculate the time required for each word in the lyrics.

[0058] In some embodiments, the electronic device 110 can determine the time distribution of the second lyrics content based on reference music content, wherein the time distribution indicates the time of a group of characters in the second lyrics content. For example, if the second lyrics content is "apple, orange, red, color", the electronic device 110 can determine the time points of "apple", "orange", "red" and "color" in the reference music content, respectively.

[0059] Furthermore, the electronic device 110 can determine the first time of the starting character and the second time of the ending character of the target lyrics based on the time distribution. For example, the electronic device 110 can determine the first time corresponding to "apple" and the second time corresponding to "color".

[0060] Finally, the electronic device 110 can determine the first time segment corresponding to the target lyrics based on the first time and the second time. For example, if the first time is one minute and twenty seconds and the second time is three minutes and fifteen seconds, the first time segment is the period between one minute and twenty seconds and three minutes and fifteen seconds.

[0061] In some embodiments, the electronic device 110 can utilize a separation module to process reference music content to obtain audio track data corresponding to multiple audio tracks. As an example, the separation module may include an audio track separation module. The electronic device 110 can input the reference music content into the audio track separation module to obtain audio track data corresponding to multiple audio tracks in the reference music content.

[0062] Furthermore, the electronic device 110 can use an audio encoder to encode audio track data corresponding to multiple audio tracks to determine a first encoded representation. Specifically, the electronic device 110 can obtain audio encoded representations corresponding to multiple audio tracks through an audio encoder, and combine the audio encoded representations corresponding to multiple audio tracks to obtain the first encoded representation.

[0063] The following will combine Figures 4A to 4B To describe the music editing process corresponding to different music editing scenarios.

[0064] Figure 4A An example process for multi-track editing according to one embodiment of the present invention is illustrated. Electronic device 110 can receive music editing requests, which may, for example, instruct the editing of all tracks. As an example, such as... Figure 4A As shown, the electronic device 110 can receive reference music content 410. After receiving the reference music content 410, the electronic device 110 inputs the reference music content 410 into the audio separation model 420 and the text recognition model 430, respectively obtaining a first encoded representation 422 associated with the reference music content and lyrics time points 435. The lyrics time points 435 include the time point corresponding to each character in the lyrics of the reference music content.

[0065] Furthermore, the electronic device 110 can receive custom lyrics 440 (i.e., first lyrics content) input by the user 140, and perform audio-text alignment of the custom lyrics 440 with the lyrics time point 435 to determine the target portion to be edited in the first encoded representation 422. The electronic device 110 can generate a first mask matrix based on the target portion to be edited in the first encoded representation 422, and add the first mask matrix to the target portion in the first encoded representation 422 to obtain an initial intermediate encoded representation 424.

[0066] Furthermore, when aligning the custom lyrics 440 with the lyrics time point 435 using audio-text, the electronic device 110 also obtains the start and end times of the target lyric segment to be edited in the lyrics of the reference music content. The electronic device 110 can adjust the length of the target segment to be edited in the first encoding representation 422 based on the average speech rate of the target lyric segment and the number of words to be added to the target lyric segment. The electronic device 110 can construct a second mask matrix 460 based on the audio sampling rate and the adjusted length of the target segment to be edited in the first encoding representation 422. Then, the electronic device 110 can replace the first mask matrix in the initial intermediate encoding representation 424 with the second mask matrix to obtain the final intermediate encoding representation 426.

[0067] Finally, the electronic device 110 can call the target model 470 to process the final intermediate encoded representation 426 and the custom lyrics 440 input by the user 140 to obtain the second encoded representation. The electronic device 110 can call the audio decoder 480 to decode the second encoded representation to obtain the target music content 490.

[0068] Figure 4B An example process for editing a specified audio track is illustrated in one embodiment of the present invention. Electronic device 110 can receive music editing requests, which may, for example, instruct the editing of a specific audio track.

[0069] As an example, such as Figure 4B As shown, the electronic device 110 can receive reference music content 410 and input the reference music content 410 into the audio separation model 420 and the text recognition model 430 to obtain the first encoded representation 422 and the lyrics time point 435 associated with the reference music content, respectively.

[0070] Furthermore, the electronic device 110 can receive custom lyrics 440-1 and a specified audio track 440-2 to be edited input by the user 140. The electronic device 110 can perform audio-text alignment of the custom lyrics 440-1, the audio track 440-2 to be edited, and the lyrics time point 435 to determine the target portion to be edited in the first coded representation 422. In some scenarios, the electronic device 110 can also determine the target portion to be edited in the first coded representation 422 based on the audio data input by the user 140.

[0071] Furthermore, the electronic device 110 can construct a mask matrix based on the audio sampling rate and the target portion to be edited in the first coded representation 422, and add the mask matrix to the target portion of the first coded representation 422 to obtain the intermediate coded representation 460.

[0072] Finally, the electronic device 110 can call the target model 470 to process the intermediate encoded representation 460 and the custom lyrics 440 input by the user 140 to obtain the second encoded representation. The electronic device 110 can call the audio decoder 480 to decode the second encoded representation to obtain the target music content 490.

[0073] In some embodiments, the target model may be trained based on the following process: First, training audio track data corresponding to multiple audio tracks of the music sample is determined; further, the training audio track data is encoded to generate a first audio feature; then, a second audio feature is generated based on a constructed training mask and the first audio feature, the training mask indicating the audio track and / or audio track segment to be generated; further, the second audio feature is processed using the target model to generate a third audio feature; finally, the target model is trained based on a comparison of the first audio feature and the third audio feature.

[0074] In some embodiments, the third audio feature is also generated based on the text content associated with the music sample.

[0075] The training process of the target model discussed above will be further described below. Figure 5 The diagram illustrates a training process 500 for a target model according to an embodiment of the present invention. This training process 500 can be performed by a suitable training device, which may be the same as or different from the electronic device 110.

[0076] like Figure 5 As shown, the training device can acquire audio data 510 and text data 550 for training the target model 570. For the text data 550, the training device can encode the text data 550 through a text encoder 560 to obtain text features 555.

[0077] For audio data 510, the training device inputs the audio data into the track separation model 520 to separate multiple tracks of the audio data 510, obtaining multi-track audio data 514 (i.e., training track data). Further, the electronic device 100 can call the audio encoder 530 to encode the multi-track audio data 514, obtaining multi-track audio features 516 (i.e., first audio features). Even further, the training device can perform multi-functional masking processing 540 on the multi-track audio features 516 using different types of masks to obtain the audio training features of the target model 570 (i.e., second audio features).

[0078] Furthermore, the training device can input the audio training data and text features 555 into the target model 570 to train the target model 570, obtaining multitrack audio features 575 generated by the target model 570. Finally, the training device can calculate the loss function 580 of the target model 570 based on the multitrack audio features 516 and the generated multitrack audio features 575 (i.e., the third audio feature). The training device can further adjust the parameters of the target model 570 according to the loss function 580.

[0079] In this way, embodiments of the present disclosure use a mask matrix to indicate the target area in the reference music content that needs to be edited, enabling the music editing model to edit only the target part of the reference music content, thereby improving the efficiency of music editing.

[0080] Example devices and equipment

[0081] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an apparatus 600 for music editing according to certain embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in an electronic device 110. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0082] like Figure 6 As shown, the apparatus 600 includes an edit request receiving module 610 configured to receive a music editing request for reference music content, the edit request indicating the target portion of the reference music content to be edited; an intermediate encoding representation generation module 620 configured to generate an intermediate encoding representation based on the target portion to be edited and a first encoding representation of the reference music content; a second encoding representation generation module 630 configured to generate a second encoding representation based on the intermediate encoding representation using a target model, the second encoding representation being generated by replacing the portion of the intermediate encoding representation corresponding to the target portion; and a music content generation module 640 configured to generate the target music content based on the second encoding representation.

[0083] In some embodiments, the apparatus 600 further includes a target portion determination module configured to determine at least one audio track segment to be edited based on a music editing request; and to determine a target portion of reference music content to be edited based on the at least one audio track segment to be edited.

[0084] In some embodiments, the target portion determination module is further configured to: determine first lyrics content based on a music editing request; determine a target lyric portion in the second lyrics content based on the first lyric content and second lyrics content of a reference music content; determine a first time period corresponding to the target lyric portion; and determine at least one audio track segment to be edited based on the first time period.

[0085] In some embodiments, the target portion determination module is further configured to: determine speech rate information associated with reference music content; adjust a first time period based on the speech rate information and first lyrics content to determine a second time period; and determine at least one audio track segment to be edited based on the second time period.

[0086] In some embodiments, the speech rate information indicates the average speech rate corresponding to the target lyrics portion in the reference music content.

[0087] In some embodiments, the target portion determination module is further configured to: determine the number of lyrics characters in the first lyrics content used to replace the target lyrics portion; and adjust the first time period based on the number of lyrics characters and speech rate information to determine the second time period.

[0088] In some embodiments, the target portion determination module is further configured to: determine the time distribution of the second lyrics content based on reference music content, the time distribution indicating the time of a group of characters in the second lyrics content; determine the first time of the start character and the second time of the end character of the target lyrics portion based on the time distribution; and determine the first time period corresponding to the target lyrics portion based on the first time and the second time.

[0089] In some embodiments, the intermediate encoding representation generation module 620 is further configured to: construct a target mask based on the target portion to be edited, wherein the mask portion corresponding to the target portion in the target mask is set to a preset value; and generate an intermediate encoding representation based on the target mask and a first encoding representation of the reference music content.

[0090] In some embodiments, the apparatus 600 further includes a first encoded representation generation module configured to process reference music content using a separation module to obtain audio track data corresponding to multiple audio tracks; and to encode the audio track data corresponding to the multiple audio tracks using an audio encoder to determine a first encoded representation.

[0091] In some embodiments, the second encoding representation is also generated based on the text content indicated by the music editing request, which describes at least one of the following: lyrics, style information, timbre information, and melody information.

[0092] In some embodiments, the target model is trained based on the following process: determining training audio track data corresponding to multiple audio tracks of a music sample; encoding the training audio track data to generate a first audio feature; generating a second audio feature based on a constructed training mask and the first audio feature, the training mask indicating the audio track and / or audio track segment to be generated; processing the second audio feature using the target model to generate a third audio feature; and training the target model based on a comparison of the first audio feature and the third audio feature.

[0093] In some embodiments, the third audio feature is also generated based on text content associated with the music sample.

[0094] The units included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 600 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0095] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 Electronic devices 110.

[0096] like Figure 7As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0097] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0098] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0099] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0100] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0102] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for music editing, comprising: receiving a music editing request for a reference music content, the editing request indicating a target portion of the reference music content to be edited; generating an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content; generating a second encoded representation based on the intermediate encoded representation by replacing a portion of the intermediate encoded representation corresponding to the target portion with a target model; and generating a target music content based on the second encoded representation. 2.The method of claim 1, further comprising: determining at least one audio track segment to be edited based on the music editing request; and determining the target portion of the reference music content to be edited based on the at least one audio track segment to be edited. 3.The method of claim 2, wherein determining at least one audio track segment to be edited based on the music editing request comprises: determining a first lyric content based on the music editing request; determining a target lyric portion in the second lyric content based on the first lyric content and a second lyric content of the reference music content; determining a first time period corresponding to the target lyric portion; and determining the at least one audio track segment to be edited based on the first time period. 4.The method of claim 3, wherein determining the at least one audio track segment to be edited based on the first time period comprises: determining a speech rate information associated with the reference music content; adjusting the first time period based on the speech rate information and the first lyric content to determine a second time period; and determining the at least one audio track segment to be edited based on the second time period. 5.The method of claim 4, wherein the speech rate information indicates an average speech rate in the reference music content corresponding to the target lyric portion. 6.The method of claim 5, wherein adjusting the first time period based on the speech rate information and the first lyric content to determine a second time period comprises: determining a number of words in the first lyric content to be replaced by the target lyric portion; and adjusting the first time period based on the number of words and the speech rate information to determine the second time period. 7.The method of claim 3, wherein determining a first time period corresponding to the target lyric portion comprises: determining a time distribution of the second lyric content based on the reference music content, the time distribution indicating times of a set of characters in the second lyric content; determining a first time of a starting character and a second time of an ending character of the target lyric portion based on the time distribution; and determining the first time period corresponding to the target lyric portion based on the first time and the second time. 8.The method of claim 1, wherein generating an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content comprises: ​ ​ ​ ​ ​ constructing a target mask based on the target portion to be edited, wherein a mask portion corresponding to the target portion in the target mask is set as a preset value; and generating the intermediate encoded representation based on the target mask and a first encoded representation of the reference music content.

9. The method of claim 1, further comprising: processing the reference music content with a separation module to obtain track data corresponding to a plurality of audio tracks; and encoding the track data corresponding to the plurality of audio tracks with an audio encoder to determine the first encoded representation.

10. The method of claim 1, wherein the second encoded representation is further generated based on text content indicated by the music editing request, the text content describing at least one of: lyrics content, style information, timbre information, melody information.

11. The method of claim 1, wherein the target model is trained based on a process comprising: determining training track data corresponding to a plurality of audio tracks of a music sample; encoding the training track data to generate first audio features; generating second audio features based on a constructed training mask and the first audio features, the training mask indicating audio tracks and / or audio track segments to be generated; processing the second audio features with the target model to generate third audio features; and training the target model based on a comparison of the first audio features and the third audio features.

12. The method of claim 11, wherein the third audio features are further generated based on text content associated with the music sample.

13. An apparatus for music generation, comprising: a request receiving module configured to receive a music editing request for a reference music content, the editing request indicating a target portion of the reference music content to be edited; an intermediate encoded representation generating module configured to generate an intermediate encoded representation based on the target portion to be edited and a first encoded representation of the reference music content; a second encoded representation generating module configured to generate a second encoded representation based on the intermediate encoded representation with a target model, the second encoded representation being generated by replacing a portion encoded representation in the intermediate encoded representation corresponding to the target portion; a music content generating module configured to generate a target music content based on the second encoded representation.

14. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-12.

15. A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-12.

Images