Data processing method, data processing device, electronic equipment and storage medium

Abstract

The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, and relates to the technical field of data processing, in particular to an audio generation technology. The implementation scheme is as follows: obtaining a generation condition for music generation; performing autoregressive coding and decoding on the condition vector corresponding to the generation condition to obtain a reference audio mixing feature of the audio to be generated; performing autoregression coding on the condition vector and the reference sound mixing feature to obtain a sharing feature at least aiming at the first sub-track and the second sub-track at the first moment; performing first decoding on the shared feature at the first moment to obtain a feature vector of a first sub-track at the first moment; performing second decoding on the shared feature at the first moment to obtain a feature vector of a second sub-track at the first moment; and fusing at least one of the feature vector of the first sub-track at the first moment, the sound mixing feature at the first moment and the feature vector of the second sub-track at the first moment to obtain the lexical element of the first sub-track at the first moment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to an audio generation technique, and more particularly to a data processing method, a data processing apparatus, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.), which has both hardware and software technologies. Artificial intelligence hardware technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0003] In recent years, there has been significant progress in condition-controlled (text, audio or image) music generation.

[0004] The methods described in this section can not necessarily be the methods previously conceived or employed. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, unless otherwise indicated, matters discussed in this section should not be assumed to have been admitted to be prior art in any jurisdiction. SUMMARY

[0005] The present disclosure provides a data processing method, a data processing apparatus, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to an aspect of the present disclosure, a data processing method is provided, comprising: obtaining a generation condition for music generation; performing autoregressive coding and decoding on a condition vector corresponding to the generation condition to obtain a reference mix feature of audio to be generated, wherein the audio is formed by mixing at least a first track and a second track; performing autoregressive coding on the condition vector and the reference mix feature to obtain a shared feature at a first time instance for at least the first track and the second track, wherein the reference mix feature at least includes a mix feature at the first time instance; performing first decoding on the shared feature at the first time instance to obtain a feature vector of the first track at the first time instance; performing second decoding on the shared feature at the first time instance to obtain a feature vector of the second track at the first time instance; fusing at least one of the feature vector of the first track at the first time instance and the mix feature at the first time instance and the feature vector of the second track at the first time instance to obtain a word piece of the first track at the first time instance; generating a word piece of the second track at the first time instance according to the feature vector of the second track; and generating audio of the first track and audio of the second track at the first time instance based on the word piece of the first track at the first time instance and the word piece of the second track at the first time instance, respectively.

[0007] According to another aspect of the present disclosure, there is provided a data processing apparatus, comprising: a condition obtaining unit configured to obtain a generation condition for music generation; a mix generation unit configured to perform auto-regressive coding on a condition vector corresponding to the generation condition to obtain a reference mix feature of audio to be generated, wherein the audio is formed by mixing at least a first track and a second track; a shared feature encoding unit configured to perform auto-regressive encoding on the condition vector and the reference mix feature to obtain a shared feature at least for the first track and the second track at a first time; a first track decoding unit configured to perform first decoding on the shared feature at the first time to obtain a feature vector of the first track at the first time; a second track decoding unit configured to perform second decoding on the shared feature at the first time to obtain a feature vector of the second track at the first time; a first track generation unit configured to fuse at least one of the feature vector of the first track at the first time and the mix feature at the first time and the feature vector of the second track at the first time to obtain a word piece of the first track at the first time; a second track generation unit configured to generate a word piece of the second track at the first time according to the feature vector of the second track; and an audio generation unit configured to generate audio of the first track and audio of the second track at the first time based on the word piece of the first track at the first time and the word piece of the second track at the first time, respectively.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to an embodiment of the present disclosure.

[0011] According to one or more embodiments of the present disclosure, the feature vector of another track can be referenced when generating audio tokens of different tracks and / or information of generating the mix feature in the first stage, so that the information of another track and / or the overall mix information can be simultaneously referenced when predicting the audio tokens of different tracks, to enhance the coordination between different tracks and the generation effect of each track.

[0012] It should be understood that the matters described herein are intended to be illustrative rather than limiting. For example, while the implementation of the present disclosure is described in terms of specific embodiments, it should be apparent that other embodiments can be devised without departing from the scope of the disclosure. For example, described features can be combined in any combination or sub-combination. Where compositions are described, aspects of the disclosure include related systems, devices, and methods. As will be apparent, one skilled in the art can readily BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the disclosure. The illustrated embodiments are exemplary only and not limiting of the scope of the claims. In all

[0014] Figure 1 shows a schematic diagram of an exemplary system in which various methods described herein can be implemented, according to an embodiment of the present disclosure; Figure 2 shows an exemplary flowchart of a data processing method, according to an embodiment of the present disclosure; Figure 3 shows an exemplary process of an audio generation manner, according to the present disclosure; Figure 4 shows an exemplary block diagram of a data processing apparatus, according to an embodiment of the present disclosure; Figure 5 shows a structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification. The

[0016] In the present disclosure, the terms “first”, “second”, etc. used in the description of various described examples are not intended to limit the positional relationship, the timing relationship or the importance relationship of the elements, and such terms are only used to distinguish one element from another. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0017] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing the specific examples, and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term “and / or” used in the present disclosure encompasses any one of the listed items and all possible combinations.

[0018] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0019] Figure 1 A schematic diagram of an example system 100 in which various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0020] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the execution of methods of data processing according to embodiments of the present disclosure.

[0021] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0022] In Figure 1In the illustrated configuration, server 120 can include one or more components that implement functionality performed by server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that a wide variety of system configurations are possible, which can differ from system 100. Thus, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0023] A user can use client devices 101, 102, 103, 104, 105, and / or 106 to receive user input and provide output results for the user. The client devices can provide an interface that enables a user of the client device to interact with the client device. The client devices can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the present disclosure can support any number of client devices.

[0024] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices can be capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0025] Network 110 can be any type of network familiar to those skilled in the art, which can use any of a variety of available protocols (including, without limitation, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wireless network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network, a WIFI network), and / or any combination of these and / or other networks.

[0026] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers), server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can run one or more services or software applications that provide the functionality described below.

[0027] Computing units in server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0028] In some implementations, server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0029] In some implementations, server 120 can be a server of a distributed system, or a server combined with a blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) service.

[0030] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network- based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0031] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores supported by file systems.

[0032] Figure 1 The system 100 of FIG. 1 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0033] Music generation in related art mainly relies on the paradigm of autoregressive Transformer, diffusion model, and non-autoregressive mask modeling. In an exemplary two-stage autoregressive music generation framework, the mix tokens are first generated in the first stage, and then the audio of different tracks (such as vocals, accompaniment, etc.) is respectively predicted according to the feature vectors of the mix tokens in the second stage. However, there are deficiencies in the collaborative modeling and prediction of vocals and accompaniment in the above music generation process, and there is task antagonism and representation competition between the prediction branches of different tracks. For example, in the time period when vocals appear, the energy of the accompaniment audio is systematically weakened, which manifests as weakened rhythm, missing frequency bands, etc., and thus affects the sound quality of the finally generated audio and the consistency between different audio tracks.

[0034] To solve the above problems, the present application provides a new data processing method for audio generation.

[0035] Figure 2 An exemplary flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0036] In step S201, a generation condition for music generation is obtained.

[0037] In step S202, autoregressive coding is performed on a condition vector corresponding to the generation condition to obtain a reference mix feature of the audio to be generated, wherein the audio is formed by mixing at least the first sub-track and the second sub-track.

[0038] In step S203, autoregressive coding is performed on the condition vector and the reference mix feature to obtain a shared feature at the first time instance for at least the first sub-track and the second sub-track, wherein the reference mix feature at least includes a mix feature at the first time instance.

[0039] In step S204, first decoding is performed on the shared feature at the first time instance to obtain a feature vector of the first sub-track at the first time instance.

[0040] In step S205, second decoding is performed on the shared feature at the first time instance to obtain a feature vector of the second sub-track at the first time instance.

[0041] In step S206, at least one of the feature vector of the first sub-track at the first time instance and the mix feature at the first time instance and the feature vector of the second sub-track at the first time instance is fused to obtain a word piece of the first sub-track at the first time instance.

[0042] In step S207, a word piece of the second sub-track at the first time instance is generated according to the feature vector of the second sub-track.

[0043] In step S208, audio of the first sub-track and audio of the second sub-track are respectively generated based on the word piece of the first sub-track at the first time instance and the word piece of the second sub-track at the first time instance.

[0044] By using the data processing method provided by the embodiments of the present disclosure, when generating word pieces of different sub-tracks, the feature vector of another sub-track and / or the information of the mix feature generated in the first stage can be referred to, so that when predicting the word pieces of the sub-tracks, the information of another sub-track and / or the overall mix information can be referred to at the same time, thereby enhancing the coordination between different sub-tracks and the generation effect of each sub-track.

[0045] The specific examples of the present disclosure will be described in detail below.

[0046] In step S201, a generation condition for music generation is obtained. The generation condition includes at least one of text, image, audio, and video related to the audio to be generated. For example, the generation condition can describe the music style to be generated in the form of text, can indicate to generate music with a similar style to a specified image, can specify the human voice tone present in the generated audio by using audio or video, etc. The specific form and content of the generation condition are not limited herein.

[0047] In step S202, the condition vector corresponding to the generation condition is auto-regressively coded to obtain a reference mix feature of the audio to be generated, wherein the audio is formed by mixing at least the first sub-track and the second sub-track. The token of the mix audio can be generated by auto-regressively decoding the reference mix feature, and the mix audio can be further generated. The mix token referred to herein refers to the token of the audio formed by mixing (rather than synthesizing) the audio content of at least the first sub-track and the second sub-track. Although the auditory effect of mixing different sub-tracks is included therein, the individual audio of different sub-tracks cannot be obtained, and thus the audio of different sub-tracks cannot be processed separately. In the following, the principle of the present disclosure will be described by taking an example in which the first sub-track is an accompaniment sub-track and the second sub-track is a vocal sub-track. However, it can be understood by those skilled in the art that, according to different actual application scenarios, the first sub-track and the second sub-track can also include other forms of content.

[0048] Therefore, it is necessary to further predict the individual audio of different sub-tracks according to the mix audio in subsequent steps.

[0049] The condition vector corresponding to the generation condition can be obtained by vectorizing the generation condition obtained in step S201. The specific vectorization means of the generation condition is not limited herein. The corresponding vectorization means can be determined according to the specific form of the generation condition. For example, the text condition can be vectorized by using word embedding, the image condition can be vectorized by using image block embedding, the audio condition can be vectorized by using time-frequency feature extraction, and the video condition can be vectorized by using spatio-temporal block embedding.

[0050] In step S202, the hidden state of the generation condition can be determined by auto-regressively encoding the input generation condition. Further, the reference mix feature can be generated by auto-regressively decoding the hidden state of the generation condition. In some examples, the auto-regressive encoding and decoding involved in the embodiments of the present disclosure can be implemented by using a Transformer model. In other examples, any other suitable model can be used to implement the auto-regressive encoding and decoding, and the specific form of the auto-regressive model is not limited herein.

[0051] In step S203, the condition vector and the reference mix feature can be auto-regressively encoded to obtain the shared feature H k of at least the first sub-track and the second sub-track at the first time k, wherein the reference mix feature at least includes the mix feature t k of the first sub-track and the second sub-track at the first time k. k The condition vector of the generation condition and the mix feature sequence t0, t1…t kare concatenated together as input of the auto-regressive encoding, and the shared feature H for the first sub-track and the second sub-track obtained from the output of the auto-regressive encoding k Alternatively, the reference mix feature can also include a sequence of partial mix features before the first time instant k, for example, t k-M， t k-M+1 …t k。 The specific value of M can be set according to actual conditions.

[0052] In some examples, the reference mix feature further includes a mix feature at a second time instant, where the second time instant is located after the first time instant in time sequence. For example, the reference mix feature can include a sequence of mix features t0, t1…t k …t k+N where N represents the Nth future time instant after the time instant k, and the specific value of N can be set according to actual conditions, and can be adjusted by adjusting the value of N. In the above manner, when predicting the shared feature H k of the first sub-track and the second sub-track at the time instant k, not only the mix information of the historical time instants before the time instant k is considered, but also the mix information of the future time instants after the time instant k is considered, so that the continuity of the mix audio before and after the time instant k is more considered when generating the shared feature of the first sub-track and the second sub-track, thereby improving the consistency in time sequence of the audio of the first sub-track and the second sub-track generated thereafter.

[0053] In step S204, the shared feature at the first time instant can be first decoded to obtain the feature vector H k_B of the first sub-track at the first time instant. Various feature transformation methods (such as using a Transformer structure or any other suitable neural network structure) can be used to realize the first decoding.

[0054] In step S205, the shared feature at the first time instant can be second decoded to obtain the feature vector H k_V of the second sub-track at the first time instant. Various feature transformation methods (such as using a Transformer structure or any other suitable neural network structure) can be used to realize the second decoding.

[0055] Using steps S204 and S205, the corresponding feature vectors of the first sub-track and the second sub-track can be generated from the shared feature generated by the mix feature, so as to further generate the audio of the first sub-track and the second sub-track.

[0056] In step S206, at least one of the feature vector of the first sub-track at the first time instant and the mix feature at the first time instant and the feature vector of the second sub-track at the first time instant is fused to obtain the word piece of the first sub-track at the first time instant.

[0057] Fusing the feature vector of the first track at the first time instant and the mix feature at the first time instant can include determining a first cross-attention prediction result of the feature vector of the first track at the first time instant and the mix feature at the first time instant. For example, the feature vector H k_B As a query (Q) of the cross-attention mechanism, the mix feature t k As a key (K) and a value (V) of the cross-attention mechanism, the feature vector H k_B of the first track and the mix feature t k to obtain a first cross-attention prediction result therebetween.

[0058] Fusing the feature vector of the first track at the first time instant and the feature vector of the second track at the first time instant can include determining a second cross-attention prediction result of the feature vector of the first track at the first time instant and the feature vector of the second track at the first time instant. For example, the feature vector H k_B As a query (Q) of the cross-attention mechanism, the feature vector H k_V As a key (K) and a value (V) of the cross-attention mechanism, the feature vector H k_B of the first track and the feature vector H k_V of the second track to obtain a second cross-attention prediction result therebetween.

[0059] Step S206 can include determining the token T k_B of the first track at the first time instant according to at least one of the first cross-attention prediction result and the second cross-attention prediction result. For example, the feature vector of the first track at the first time instant can be transformed with at least one of the first cross-attention prediction result and the second cross-attention prediction result to obtain the token of the first track at the first time instant. The transformation of the feature vector of the first track can be implemented based on a multi-head attention mechanism. For example, the feature vector H k_BThe different subspaces are mapped, and weights for each subspace are determined according to at least one of the first cross-attention prediction result and the second cross-attention prediction result, and then the transformation of the feature vector of the first track is realized by summing the products of the mapping results of the different subspaces and the corresponding weights. For example, the first cross-attention prediction result can be taken as the first weight of the first subspace, and the second cross-attention prediction result can be taken as the second weight of the second subspace. The number of subspaces can be determined according to actual conditions. The first subspace or the second subspace can be omitted by modifying the number of parameters of the multi-head matrix in the case of using only the first cross-attention prediction result or using only the second cross-attention prediction result according to actual conditions. It can be understood that at least one of the first cross-attention prediction result and the second cross-attention prediction result can be applied to the feature vector H k_B of the first track at the first moment in other ways without departing from the principles of the present disclosure, to obtain the word token T k_B of the first track at the first moment.

[0060] By determining the word token T k_B of the first track according to the first cross-attention prediction result, more information in the audio mixed by the two tracks can be understood when generating the audio of the first track, thereby improving the overall coordination of the generated audio.

[0061] By determining the word token T k_B of the first track according to the second cross-attention prediction result, more audio features of the other track can be understood when generating the audio of the first track, thereby improving the coordination between different track audios.

[0062] In step S207, the word token of the second track at the first moment is generated according to the feature vector of the second track.

[0063] In some embodiments, the word token T k_V of the second track at the first moment can be directly obtained by autoregressive decoding on the feature vector of the second track. In such embodiments, the mixed audio generated in the first stage and the audio information of the first track are not referred to when predicting the audio of the second track.

[0064] In other embodiments, the generation of the word token of the second track at the first moment can also refer to the mixed audio generated in the first stage and / or the audio information of the first track. In an example, the feature vector H k_V of the second track at the first moment and at least one of the mixed feature t k at the first moment and the feature vector H k_B of the first track at the first moment are fused to obtain the word token T k_B of the second track at the first moment.

[0065] Fusing the feature vector of the second track at the first time instant and the mix feature at the first time instant can include determining a third cross-attention prediction result of the feature vector of the second track H k_V and the mix feature t k at the first time instant. For example, the feature vector of the second track H k_V may be taken as a query (Q) of a cross-attention mechanism, the mix feature t k may be taken as a key (K) and a value (V) of the cross-attention mechanism, to obtain the third cross-attention prediction result between the feature vector of the second track H k_V and the mix feature t k .

[0066] Fusing the feature vector of the second track at the first time instant and the feature vector of the first track at the first time instant can include determining a fourth cross-attention prediction result of the feature vector of the second track H k_V and the feature vector of the first track H k_B at the first time instant. For example, the feature vector of the second track H k_V may be taken as a query (Q) of a cross-attention mechanism, the feature vector of the first track at the first time instant H k_B may be taken as a key (K) and a value (V) of the cross-attention mechanism, to obtain the fourth cross-attention prediction result between the feature vector of the second track H k_V and the feature vector of the first track H k_B .

[0067] Step S207 can include determining the wordpiece T k_V of the second track at the first time instant according to at least one of the third cross-attention prediction result and the fourth cross-attention prediction result. For example, the feature vector of the second track at the first time instant can be transformed by using at least one of the third cross-attention prediction result and the fourth cross-attention prediction result, to obtain the wordpiece of the second track at the first time instant. The transformation of the feature vector of the second track can be implemented based on a multi-head attention mechanism. The transformation of the feature vector of the second track in step S207 can be implemented in the same way as the transformation of the feature vector of the first track based on the multi-head attention mechanism described in step S206, which will not be repeated here.

[0068] In step S208, the audio of the first sub-track and the audio of the second sub-track are generated based on the token of the first sub-track at the first time instant and the token of the second sub-track at the first time instant respectively. The aforementioned steps can be used to sequentially generate the token sequence of the first sub-track containing the audio tokens of the first sub-track at different time instants and the token sequence of the second sub-track containing the audio tokens of the second sub-track at different time instants. In an example, the token sequence of the first sub-track including the token of the first sub-track at the first time instant and the token sequence of the second sub-track including the token of the second sub-track at the first time instant can be processed by using the inverse operation of the vectorization of the audio, so as to obtain the audio of the first sub-track and the audio of the second sub-track.

[0069] In some embodiments, the method 200 can further include synthesizing the audio of the first sub-track and the audio of the second sub-track. By synthesizing the audio of different sub-tracks, the synthesized audio can be obtained, which can be used for subsequent processing, such as further processing for mixing, style transfer, etc.

[0070] The principle of the present disclosure is described above by taking an example that the mixed audio includes both the first sub-track and the second sub-track. In other embodiments, the mixed audio can further include more sub-track audios, such as a third sub-track. For example, the accompaniment sub-track can include more dimensional instrument sub-tracks, such as percussion, string, wind, etc.

[0071] In the case that the mixed audio further mixes the third sub-track, the method 200 can further include third decoding the shared feature at the first time instant to obtain the feature vector of the third sub-track at the first time instant. The third decoding can be implemented by using various feature transformation manners, such as using the Transformer structure or any other suitable neural network structure.

[0072] In the case that the third sub-track exists, the step S206 can further include fusing the feature vector of the first sub-track at the first time instant and the feature vector of the third sub-track at the first time instant. The fifth cross-attention prediction result of the feature vector of the first sub-track at the first time instant and the feature vector of the third sub-track at the first time instant can be determined, and the token of the first sub-track at the first time instant is determined according to the aforementioned first cross-attention prediction result, the second cross-attention prediction result and the fifth cross-attention prediction result. For example, the feature vector of the first sub-track at the first time instant can be transformed by using at least one of the first cross-attention prediction result, the second cross-attention prediction result and the fifth cross-attention prediction result, so as to obtain the token of the first sub-track at the first time instant. The transformation of the feature vector of the first sub-track containing the cross-attention prediction result of the third sub-track can be implemented by using the multi-head attention mechanism based transformation manner of the feature vector of the first sub-track based on the multi-head attention mechanism described in step S206, which will not be described here again.

[0073] It can be understood that in the case of audio including more than two sub-tracks in the mix audio, the person skilled in the art can select part or all of the sub-tracks to be fused with the sub-track currently being predicted according to actual factors such as the style, content, etc. of the audio. In some examples, a deep learning model can also be used to predict the input generation condition to determine which sub-tracks will be fused with the first sub-track. Using the above method, the information of different sub-tracks can be flexibly fused to meet the needs of different application scenarios for audio generation effect.

[0074] Figure 3 An exemplary process of the audio generation method according to the present disclosure is shown.

[0075] In step 301, a condition C, such as a text, audio, image form of a generation condition, can be obtained.

[0076] The condition C can be processed using various vectorization methods to obtain a hidden state 302.

[0077] In step 303, the condition hidden state 302 can be encoded to generate a mix feature 304.

[0078] In step 305, the mix feature 304 can be decoded to obtain a mix token 306.

[0079] In step 307, shared features 308 of the vocal sub-track and the accompaniment sub-track can be generated according to the condition hidden state 302 and the mix feature 304. Step 307 can be implemented using any suitable deep learning backbone network for feature transformation.

[0080] In step 309, the shared features 308 can be feature-transformed to obtain vocal features 310.

[0081] In step 312, the shared features 308 can be feature-transformed to generate accompaniment feature(s) 313.

[0082] In step 311, the vocal features 310 can be fused with the mix feature 304 and the accompaniment feature 313 to obtain vocal tokens 315.

[0083] In step 314, the accompaniment feature 313 can be fused with the mix feature 304 and the vocal feature 310 to obtain accompaniment tokens 316. The fusion steps 311, 314 can be implemented using the method described based on the cross-attention mechanism, which will not be repeated here. Figure 2 The fusion steps 311, 314 can be implemented using the method described based on the cross-attention mechanism, which will not be repeated here.

[0084] In step 317, the vocal tokens 315 can be decoded to obtain vocal audio 319. In step 318, the accompaniment tokens 316 can be decoded to obtain accompaniment audio 320.

[0085] In step 321, the human voice audio 319 and the accompaniment audio 320 can be processed to obtain the audio 322 to be generated.

[0086] Figure 4 An exemplary block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown. The method 200 described in connection with FIG. 400 can be implemented with the data processing apparatus. Figure 2

[0087] As shown, the data processing apparatus 400 can include a condition obtaining unit 401, a mix generating unit 402, a shared feature encoding unit 403, a first stem decoding unit 404, a second stem decoding unit 405, a first stem generating unit 406, a second stem generating unit 407, and an audio generating unit 408. Figure 4 The condition obtaining unit 401 can be configured to obtain a generation condition for music generation.

[0088] The mix generating unit 402 can be configured to perform auto-regressive coding on a condition vector corresponding to the generation condition to obtain a reference mix feature of the audio to be generated, wherein the audio is formed by mixing at least a first stem and a second stem.

[0089] The shared feature encoding unit 403 can be configured to perform auto-regressive coding on the condition vector and the reference mix feature to obtain a shared feature for at least the first stem and the second stem at a first time instance, wherein the reference mix feature comprises at least a mix feature at the first time instance.

[0090] The first stem decoding unit 404 can be configured to perform a first decoding on the shared feature at the first time instance to obtain a feature vector of the first stem at the first time instance.

[0091] The second stem decoding unit 405 can be configured to perform a second decoding on the shared feature at the first time instance to obtain a feature vector of the second stem at the first time instance.

[0092] The first stem generating unit 406 can be configured to fuse at least one of the feature vector of the first stem at the first time instance and the mix feature at the first time instance and the feature vector of the second stem at the first time instance to obtain a wordpiece of the first stem at the first time instance.

[0093] The second stem generating unit 407 can be configured to generate a wordpiece of the second stem at the first time instance according to the feature vector of the second stem.

[0094]

[0095] ​​The audio generation unit 408 can be configured to generate the audio of the first stem and the audio of the second stem based on the token of the first stem at the first time instant and the token of the second stem at the first time instant, respectively.

[0096] In some embodiments, the first stem generation unit 406 is configured to determine the token of the first stem at the first time instant according to at least one of: a first cross-attention prediction result of the feature vector of the first stem at the first time instant and the mix feature at the first time instant, and a second cross-attention prediction result of the feature vector of the first stem at the first time instant and the feature vector of the second stem at the first time instant.

[0097] In some embodiments, determining the token of the first stem at the first time instant according to at least one of the first cross-attention prediction result and the second cross-attention prediction result comprises transforming the feature vector of the first stem at the first time instant with at least one of the first cross-attention prediction result and the second cross-attention prediction result to obtain the token of the first stem at the first time instant.

[0098] In some embodiments, generating the token of the second stem at the first time instant according to the feature vector of the second stem comprises fusing at least one of the feature vector of the second stem at the first time instant and the mix feature at the first time instant and the feature vector of the first stem at the first time instant to obtain the token of the second stem at the first time instant.

[0099] In some embodiments, fusing at least one of the feature vector of the second stem at the first time instant and the mix feature at the first time instant and the feature vector of the first stem at the first time instant to obtain the token of the second stem at the first time instant comprises determining the token of the second stem at the first time instant according to at least one of: a third cross-attention prediction result of the feature vector of the second stem at the first time instant and the mix feature at the first time instant, and a fourth cross-attention prediction result of the feature vector of the second stem at the first time instant and the feature vector of the second stem at the first time instant.

[0100] In some embodiments, determining the token of the second stem at the first time instant according to at least one of the third cross-attention prediction result and the fourth cross-attention prediction result comprises transforming the feature vector of the second stem at the first time instant with at least one of the third cross-attention prediction result and the fourth cross-attention prediction result to obtain the token of the second stem at the first time instant.

[0101] In some embodiments, the reference mix feature further comprises a mix feature at a second time instant, wherein the second time instant is chronologically subsequent to the first time instant.

[0102] In some embodiments, the data processing apparatus 400 further comprises a synthesizing unit configured to synthesize the audio of the first sub-track and the audio of the second sub-track.

[0103] In some embodiments, the generation condition comprises at least one of text, image, audio, video related to the audio to be generated.

[0104] In some embodiments, the first sub-track is an accompaniment sub-track, and the second sub-track is a vocal sub-track.

[0105] In some embodiments, the audio is further mixed with a third sub-track, and the data processing apparatus 400 further comprises a third sub-track decoding unit configured to perform third decoding on the shared feature at the first time instant to obtain a feature vector of the third sub-track at the first time instant.

[0106] In some embodiments, the first sub-track generating unit is configured to determine the wordpiece of the first sub-track at the first time instant according to at least one of: a first cross-attention prediction result of the feature vector of the first sub-track at the first time instant and the mix feature at the first time instant, a second cross-attention prediction result of the feature vector of the first sub-track at the first time instant and the feature vector of the second sub-track at the first time instant, and a fifth cross-attention prediction result of the feature vector of the first sub-track at the first time instant and the feature vector of the third sub-track at the first time instant.

[0107] It should be understood that, Figure 4 The various modules or units of the apparatus 400 shown in FIG. 4 can correspond to the various steps in the method 200 described with reference to FIG. 2. Thus, the operations, features and advantages described above for the method 200 apply equally to the apparatus 400 and the modules and units included therein. For the sake of brevity, certain operations, features and advantages are not described again here. Figure 2 The various modules or units of the apparatus 400 shown in FIG. 4 can correspond to the various steps in the method 200 described with reference to FIG. 2. Thus, the operations, features and advantages described above for the method 200 apply equally to the apparatus 400 and the modules and units included therein. For the sake of brevity, certain operations, features and advantages are not described again here.

[0108] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various units discussed herein can be divided among more units, and / or at least some of the functions of more units can be combined into a single unit.

[0109] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with the relevant legal regulations and do not violate public order and good customs.

[0110] According to embodiments of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the embodiments of the present disclosure.

[0111] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0112] According to an embodiment of the present disclosure, a computer program product including a computer program is also provided, wherein the computer program, when executed by a processor, implements the method according to the embodiments of the present disclosure.

[0113] Reference Figure 5 A block diagram of the structure of an electronic device 500 that can be a server or a client of the present disclosure, which is an example of a hardware device that can be applied to aspects of the present disclosure, will now be described. The electronic device is intended to represent a wide variety of digital electronic computer devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent a variety of mobile devices such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0114] As shown in Figure 5 The electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the electronic device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0115] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0116] The computing unit 501 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the method 200. For example, in some embodiments, the method 200 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the method 200 by any other appropriate means, such as by means of firmware.

[0117] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0118] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowchart and / or block diagram block or blocks. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0121] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0122] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0123] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0124] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A data processing method, comprising: Obtain the generation conditions used for music generation; The condition vector corresponding to the generation conditions is subjected to autoregressive encoding and decoding to obtain the reference mixing features of the audio to be generated, wherein the audio is formed by mixing at least a first track and a second track. The conditional vector and the reference mix feature are subjected to autoregressive encoding to obtain shared features at least for the first track and the second track at a first time point, wherein the reference mix feature includes at least the mix feature at the first time point. The shared features at the first time point are first decoded to obtain the feature vector of the first track split at the first time point; The shared features at the first time point are decoded a second time to obtain the feature vector of the second track split at the first time point; At least one of the feature vector of the first track at the first time point, the mixing feature at the first time point, and the feature vector of the second track at the first time point is fused to obtain the word unit of the first track at the first time point. Generate the word units of the second track at the first time point based on the feature vector of the second track. The audio of the first track and the audio of the second track are generated based on the words of the first track at the first time point and the words of the second track at the first time point, respectively.

2. The method of claim 1, wherein, At least one of the feature vector of the first track at the first time point, the mixing feature at the first time point, and the feature vector of the second track at the first time point is fused to obtain the lexical units of the first track at the first time point, including determining the lexical units of the first track at the first time point based on at least one of the following: The feature vector of the first track at the first time point and the first cross-attention prediction result of the mixing features at the first time point, and The second cross-attention prediction result of the feature vector of the first track at the first time point and the feature vector of the second track at the first time point.

3. The method as described in claim 2, wherein, The terms used to determine the first track segment at the first time step based on at least one of the first cross-attention prediction result and the second cross-attention prediction result include: The feature vector of the first track at the first time point is transformed using at least one of the first cross-attention prediction results and the second cross-attention prediction results to obtain the word units of the first track at the first time point.

4. The method of claim 1, wherein, Generating the lexical units of the second track at the first time step based on the feature vector of the second track includes: The feature vector of the second track at the first time point and the mixing feature at the first time point are fused together with at least one of the feature vector of the first track at the first time point to obtain the word unit of the second track at the first time point.

5. The method of claim 4, wherein, The feature vector of the second track at the first time point and the mixing feature at the first time point are fused with at least one of the feature vectors of the first track at the first time point to obtain the lexical units of the second track at the first time point, including determining the lexical units of the second track at the first time point according to at least one of the following: The third cross-attention prediction result of the feature vector of the second track at the first time point and the mixing feature at the first time point, and The fourth cross-attention prediction result of the feature vector of the second track at the first time point and the feature vector of the second track at the first time point.

6. The method of claim 5, wherein, The terms used to determine the second track segment at the first time step based on at least one of the third cross-attention prediction result and the fourth cross-attention prediction result include: The feature vector of the second track at the first time point is transformed using at least one of the third cross-attention prediction results and the fourth cross-attention prediction results to obtain the word units of the second track at the first time point.

7. The method of claim 1, wherein, The reference mixing feature also includes a mixing feature at a second time point, wherein the second time point is sequentially located after the first time point.

8. The method according to any one of claims 1-7, further comprising: The audio from the first track and the audio from the second track are combined.

9. The method according to any one of claims 1-7, wherein, The generation conditions include at least one of the following: text, image, audio, and video related to the audio to be generated.

10. The method according to any one of claims 1-7, wherein, The first track is the accompaniment track, and the second track is the vocal track.

11. The method according to any one of claims 1-7, wherein, The audio also includes a third track, and the method further includes: The shared features at the first time point are decoded in the third time point to obtain the feature vector of the third track at the first time point.

12. The method of claim 11, wherein, At least one of the feature vectors of the first track at the first time point, the mixing features at the first time point, and the feature vectors of the second track at the first time point is fused to obtain the lexical units of the first track at the first time point. The lexical units of the first track at the first time point are determined according to at least one of the following: The first cross-attention prediction result of the feature vector of the first track at the first time point and the mixing feature at the first time point. The second cross-attention prediction results of the feature vectors of the first track at the first time point and the feature vectors of the second track at the first time point, and The fifth cross-attention prediction result of the feature vector of the first track at the first time point and the feature vector of the third track at the first time point.

13. A data processing apparatus, comprising: The condition acquisition unit is configured to acquire the generation conditions used for music generation; The mixing generation unit is configured to perform autoregressive encoding and decoding on the condition vector corresponding to the generation conditions to obtain reference mixing features of the audio to be generated, wherein the audio is formed by mixing at least a first track and a second track. A shared feature coding unit is configured to perform autoregressive coding on the condition vector and the reference mix features to obtain shared features at least for the first track and the second track at a first time step, wherein the reference mix features include at least the mix features at the first time step. The first track-splitting decoding unit is configured to perform a first decoding on the shared features at the first time point to obtain the feature vector of the first track at the first time point. The second track-splitting decoding unit is configured to perform a second decoding on the shared features at the first time point to obtain the feature vector of the second track at the first time point. The first track generation unit is configured to fuse at least one of the feature vector of the first track at the first time point, the mixing feature at the first time point, and the feature vector of the second track at the first time point to obtain the word of the first track at the first time point. The second track generation unit is configured to generate the word of the second track at the first time step based on the feature vector of the second track. The audio generation unit is configured to generate audio for the first track and audio for the second track based on the word units of the first track at the first time point and the word units of the second track at the first time point, respectively.

14. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-12.

Images