A method and apparatus for processing audio data

Abstract

The application discloses a processing method and device of audio data, and relates to the technical field of multimedia. The method comprises the following steps: acquiring m (m is an integer greater than or equal to 2) audio segments; determining m-1 transition audio information according to the m audio segments; and generating target mashup audio according to the m audio segments and the m-1 transition audio information. The m-1 transition audio information is used to connect the m audio segments. For the first transition audio information in the m-1 transition audio information, the first transition audio information is used to connect a first audio segment and a second audio segment which are sequentially arranged in the m audio segments. The arrangement of the m audio segments refers to the mashup order of the m audio segments.

Description

TECHNICAL FIELD

[0001] The present application relates to the multimedia technical field, and in particular to an audio data processing method and device. BACKGROUND

[0002] With the wide spread and application of digital streaming media music, and with the popularity and development of wireless terminal devices such as mobile phones, tablets, earphones, etc., listening to music has become a necessity for most people in different environments, and people's demand for music diversification is also growing. For example, in addition to the listening experience of listening to the entire audio from the beginning to the end, people's demand for audio composed of multiple audio segments, i.e., mix audio, is also increasing.

[0003] Currently, when implementing audio mixing, only at least two audio segments with high similarity can usually be spliced and mixed. However, the style of the mixed audio obtained in this way is often relatively single.

[0004] Therefore, how to obtain more rich and diverse mix audio is a technical problem to be solved in the prior art. SUMMARY

[0005] The present application provides an audio data processing method and device, which can obtain more rich and diverse mix audio.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an audio data processing method, which comprises: obtaining m audio segments, m being an integer greater than or equal to 2. Determining m-1 transition audio information according to the m audio segments. Generating a target mix audio according to the m audio segments and the m-1 transition audio information. Wherein the m-1 transition audio information is used to link the m audio segments. Wherein, for the first transition audio information in the m-1 transition audio information, the first transition audio information is used to link a first audio segment and a second audio segment that are sequentially continuous in the m audio segments. Here, the order of the m audio segments is the mixing order of the m audio segments.

[0008] As can be seen, when multiple audio segments are mixed based on the method provided by the present application, new transition audio information for linking the multiple audio segments can be generated. Therefore, the method provided by the present application does not need to consider the similarity of the multiple audio segments used to mix the target mix audio. That is, the method provided by the present application can obtain more rich and diverse mix audio.

[0009] In a possible design, the determining the m-1 pieces of transition audio information according to the m pieces of audio segments includes: determining a first piece of transition audio information according to first information of the first piece of audio segment and second information of the second piece of audio segment. The first information includes musical instrument digital interface (MIDI) information and audio feature information of the first piece of audio segment, the second information includes MIDI information and audio feature information of the second piece of audio segment, and the first piece of transition audio information includes MIDI information of a first transition audio corresponding to the first piece of transition audio information.

[0010] In another possible design, the audio feature information includes at least one of main melody track position information, a style label, an emotion label, rhythm information, beat information, or key information of the audio segment.

[0011] Based on the two possible designs, the transition audio information for splicing the multiple pieces of audio segments generated by the method provided in this application is generated in the MIDI domain. Since the MIDI information of the audio is the most original form of the audio, the MIDI information records information such as note pitch, note intensity, and note duration of the audio. Therefore, compared with directly splicing the multiple pieces of audio in the time domain, the transition audio information for splicing the two pieces of audio generated by the method provided in this application after processing the MIDI information of the audio segment in the MIDI domain is generated based on audio theory. In this way, the spliced audio based on the transition audio information is more smooth and natural in hearing. Moreover, processing data in the MIDI domain is also more conducive to flexibility and consistency of the spliced audio in later rendering.

[0012] In another possible design, the determining the first piece of transition audio information according to the first information of the first piece of audio segment and the second information of the second piece of audio segment includes: determining the first piece of transition audio information according to the first information of the first piece of audio segment, the second information of the second piece of audio segment, and a preset neural network model.

[0013] In another possible design, when the first piece of audio segment is located before the second piece of audio segment in the target spliced audio, the first piece of transition audio information is determined based on a feature vector used to represent the first piece of transition audio information, and the feature vector of the first piece of transition audio information is determined based on a first vector and a second vector. The first vector is a feature vector generated according to the first information at a time sequence end of the first piece of audio segment, and the second vector is a feature vector generated according to the second information at a time sequence start of the second piece of audio segment.

[0014] Based on the two possible designs, the method provided in this application processes the MIDI information of the plurality of audio segments in the MIDI domain through a neural network model, thereby obtaining the MIDI information of the transition audio for linking the plurality of audio segments. In this way, based on the strong learning ability of the neural network, the transition audio information for linking the plurality of audio segments obtained by the application based on the learning of audio music theory can more naturally and smoothly link the plurality of audio segments.

[0015] In another possible design, the obtaining of the m audio segments includes: in response to a first operation of a user, determining k target audios. The m audio segments are extracted from the k target audios. Here, 2≤k≤m, and k is an integer.

[0016] Based on this possible design, the application can string the audio segments in the plurality of target audios selected by the user based on the user's intention, thereby improving the user experience.

[0017] In another possible design, before the determining of the m-1 transition audio information according to the m audio segments, the method further includes: determining a stringing order of the m audio segments.

[0018] In another possible design, the method further includes: in response to a second operation of a user, re-determining the stringing order of the m audio segments. The m-1 transition audio information is re-determined according to the re-determined stringing order and the m audio segments. The target stringing audio is re-generated according to the re-determined m-1 transition audio information and the m audio segments.

[0019] According to this possible design, after the target stringing audio is generated by the method provided in this application, when the user is not satisfied with the target stringing audio, a second operation can be input to the terminal device, so that the terminal device responds to the second operation, adjusts the stringing order of the m audio segments for generating the target stringing audio, and re-generates a new target stringing audio. In this way, through repeated interaction between the device and the user, the user can obtain a satisfactory target stringing audio, thereby improving the user experience.

[0020] In another possible design, the method further includes: in response to a third operation of a user, rendering the target stringing audio.

[0021] In another possible design, the method further includes: outputting the target stringing audio.

[0022] In a second aspect, the application provides an audio data processing apparatus.

[0023] In a possible design, the processing apparatus is configured to perform any of the methods in the first aspect. The processing apparatus can be divided into functional modules according to any of the methods in the first aspect. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. For example, the processing apparatus can be divided into an obtaining unit, a determining unit, a generating unit, and the like according to functions. The possible technical solutions and beneficial effects of each functional module are described in the first aspect or the corresponding possible design, which will not be repeated here.

[0024] In another possible design, the processing apparatus includes one or more processors and a transmission interface, the one or more processors receive or send data through the transmission interface, and the one or more processors are configured to invoke program instructions stored in a memory, so that the processing apparatus performs any of the methods in the first aspect or any of the possible designs.

[0025] In a third aspect, the present application provides a computer-readable storage medium, which includes program instructions, when the program instructions are executed on a computer or a processor, the computer or the processor performs any of the methods provided in any of the possible implementation manners of the first aspect.

[0026] In a fourth aspect, the present application provides a computer program product, when the computer program product is executed on a processing apparatus of audio data, any of the methods provided in any of the possible implementation manners of the first aspect is executed.

[0027] In a fifth aspect, the present application provides a processing system of audio data, which includes a terminal device and a server. The terminal device is configured to perform a method part of any of the methods in any of the possible implementation manners of the first aspect, which interacts with a user, and the server is configured to perform a method part of any of the methods in any of the possible implementation manners of the first aspect, which generates a target mashup audio.

[0028] It can be understood that any of the apparatuses, computer storage media, computer program products, or systems provided in the present application can be applied to the corresponding methods provided in the present application, and thus the beneficial effects achieved by the apparatuses, computer storage media, computer program products, or systems can refer to the beneficial effects of the corresponding methods, which will not be repeated here.

[0029] In the present application, the name of the processing apparatus of audio data does not constitute a limitation on the device or the functional module itself, and in actual implementation, these devices or functional modules can appear in other names. As long as the functions of each device or functional module are similar to those in the present application, they belong to the scope of the claims of the present application and equivalent technologies. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A schematic diagram of a hardware structure of a mobile phone is provided for an embodiment of the present application;

[0031] Figure 2 A schematic diagram of a processing system of audio data is provided for an embodiment of the present application;

[0032] Figure 3 A flowchart of a processing method of audio data is provided for an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a first operation input by a user on an audio editing interface of an audio editing application is provided for an embodiment of the present application;

[0034] Figure 5 A schematic diagram of another first operation input by a user on an audio editing interface of an audio editing application is provided for an embodiment of the present application;

[0035] Figure 6 A schematic diagram of still another first operation input by a user on an audio editing interface of an audio editing application is provided for an embodiment of the present application;

[0036] Figure 7 A schematic diagram of still another first operation input by a user on an audio editing interface of an audio editing application is provided for an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a structure of a preset neural network model is provided for an embodiment of the present application;

[0038] Figure 9 A schematic diagram of another structure of a preset neural network model is provided for an embodiment of the present application;

[0039] Figure 10 A schematic diagram of a second operation is provided for an embodiment of the present application;

[0040] Figure 11 A schematic diagram of rendering and outputting MIDI information of a target jam audio is provided for an embodiment of the present application;

[0041] Figure 12 A schematic diagram of a structure of a processing device of audio data is provided for an embodiment of the present application;

[0042] Figure 13 A schematic diagram of a signal bearing medium for bearing a computer program product is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to more clearly understand the embodiments of the present application, part of the terms or technologies involved in the embodiments of the present application are explained as follows:

[0044] 1) Musical instrument digital interface (MIDI)

[0045] MIDI is the most widely used music standard format in the music composition field, which can be called "music score that computers can understand".

[0046] MIDI records music by using digital control signals of notes. That is, MIDI transmits not sound signals themselves, but instructions such as notes and control parameters. These instructions can instruct MIDI devices to play music, for example, instruct MIDI devices to play a certain note with the volume size indicated in the instruction, etc. The instructions transmitted by MIDI can be uniformly represented as MIDI messages or MIDI information.

[0047] Generally, MIDI information can be presented in the form of a score or in the form of a data stream. When the MIDI information is presented in the form of a score, it can be simply referred to as MIDI score.

[0048] For a music waveform signal in time domain stored in a waveform audio file format (WAV), when the music waveform signal is transcribed into MIDI information, the MIDI information can be understood as an expression form of the music waveform signal in the MIDI domain. Wherein, the time domain refers to the time domain.

[0049] It can be understood that the MIDI information can generally include multiple sound tracks, and each sound track is marked with the starting position of the note, the ending position of the note, the pitch of the note, and the dynamic information of the note, etc. Wherein, a sound track is used to represent a kind of instrument sound / vocal sound. It can be understood that the size of a complete music expressed by MIDI information is often only a few tens of kilobytes (KB), but it can include dozens of sound tracks.

[0050] Currently, almost all modern music is synthesized by using MIDI information plus timbre library. Wherein, the timbre library (or called sampling library) includes various sounds that can be heard and created by humans, for example, including performances of various musical instruments, performances of various human voices, voice-overs, and recordings of various natural and artificial sounds, etc.

[0051] 2) Hidden space

[0052] For a feature of an intermediate layer output of a neural network, the space in which the original data represented by the feature is transformed after passing through several neural network layers can be referred to as a latent space. Generally, the dimension of the latent space is generally less than the spatial dimension of the original data.

[0053] The latent space can also be understood as an abstract extraction and representation of the features of the original data.

[0054] 3) Sequence model network, bidirectional sequence model network

[0055] Generally, a model that contains sequence data in the input or output can be called a sequence model. Sequence models are usually used to process data with certain sequential relationships. The neural network used to build a sequence model can be referred to as a sequence model network.

[0056] Among them, common sequence model networks include recurrent neural networks (RNN), long short-term memory (LSTM), gated recurrent units (GRU), transformers, etc.

[0057] It should be understood that the prediction result predicted by the sequence model network at time t is usually based on learning data before time t.

[0058] In some cases, when the prediction result predicted by the sequence model network at time t is based on learning data before time t and learning data after time t, such a sequence network model is called a bidirectional sequence model network. As can be seen, the bidirectional sequence network model combines the context information of time t in the input data when predicting the input data.

[0059] It should be understood that the bidirectional sequence model can predict the prediction result at any time of the input data.

[0060] Among them, common bidirectional sequence model networks include (bidirectional) recurrent neural networks ((Bi-)RNN), (bidirectional) long short-term memory ((Bi-)LSTM), (bidirectional) gate recurrent units ((Bi-)GRU), transformers, etc.

[0061] 4), Other terms

[0062] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean "an example of" or "an example, only. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the exemplary or for example embodiments are presented for purposes of clarity and illustration only.

[0063] In the embodiments of the present application, the terms "first", "second", "third", etc. are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. The term "at least one" in the present application means one or more. In the description of the present application, the meaning of the term "a plurality of" is two or more, unless otherwise specified.

[0064] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or", is a description of the associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects.

[0065] It should also be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] It should be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0067] It should also be understood that the term "comprise" (also referred to as "includes", "including", "comprises" and / or "comprising") when used in the present specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0068] It should be understood that the stringing described below in the embodiments of the present application refers to a process of extracting a plurality of audio segments from different audios and combining the plurality of audio segments in a preset order, i.e., a stringing order of the plurality of audio segments.

[0069] The embodiments of the present application provide a processing method of audio data. The method determines m-1 transition audio information according to m audio segments obtained in advance, and then connects the m audio segments based on the m-1 transition audio information, so as to generate a target stringing audio after the m audio segments are stringed. One transition audio information is used to connect two audio segments adjacent in a stringing order.

[0070] Since the method does not need to consider the feature similarity between the plurality of audio segments when stringing the plurality of audio segments, the method of the embodiments of the present application can obtain rich and various stringing audios.

[0071] The embodiments of the present application also provide a processing apparatus of audio data. The processing apparatus can be a terminal device. The terminal device can be a mobile phone, a tablet computer, a notebook computer, a personal digital assistant (PDA), a netbook, a wearable electronic device (such as a smart watch, smart glasses), and the like. The terminal device can also be a desktop computer, a smart television, a vehicle-mounted device, and the like. The terminal device can also be any other terminal device capable of implementing the embodiments of the present application, which is not limited in the present application.

[0072] The above processing apparatus is taken as a mobile phone as an example, and the mobile phone is taken as an example. Figure 1 Figure 1 A hardware structure schematic diagram of a mobile phone 10 is shown. As shown in Figure 1 The mobile phone 10 can include a processor 110, an internal memory 120, an external memory interface 130, a camera 140, a touch screen 150, an audio module 160, a communication module 170, and the like.

[0073] ​The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0074] The controller can be the nerve center and command center of the mobile phone 10. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and instruction execution.

[0075] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also constantly self-learn. Through the NPU, intelligent cognitive applications of the mobile phone 10 can be realized, for example: text recognition, image recognition, face recognition, etc.

[0076] The memory can also be provided in the processor 110, for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or are repeatedly used by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0077] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0078] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). The I2S interface can be used for audio communication. The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. The MIPI interface can be used to connect the processor 110 and peripheral devices such as the camera 140, the touch screen 150, etc. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal.

[0079] The internal memory 120 can be used to store computer executable program codes including instructions. The processor 110 performs various functional applications and data processing of the mobile phone 10 by running the instructions stored in the internal memory 120, such as the audio data processing method provided by the embodiments of the present application.

[0080] The external memory interface 130 can be used to connect an external memory card, such as a Micro SD card, to extend the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface 130 to implement a data storage function. For example, files such as music, videos, and pictures are saved in the external memory card.

[0081] The camera 140 is used to acquire a still image or a video. An object generates an optical image through a lens and projects the optical image onto a photosensitive element. A digital signal processor is used to process a digital signal, which can be a digital image signal or another digital signal. It should be understood that the mobile phone 10 can include n cameras 140, where n is a positive integer.

[0082] The touch screen 150 is used for interaction between the mobile phone 10 and a user. The touch screen 150 includes a display panel 151 and a touch panel 152. The display panel 151 is used to display text, images, videos, and the like. The touch panel 152 is used to input instructions of the user.

[0083] The audio module 160 is used to convert digital audio information into an analog audio signal output and to convert an analog audio input into a digital audio signal. The audio module 160 can include at least one of a speaker 161, a receiver 162, a microphone 163, and a headset interface 164.

[0084] The speaker 161, also referred to as a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The receiver 162, also referred to as a “earpiece”, is used to convert an audio electrical signal into a sound signal. The microphone 163, also referred to as a “microphone” and a “sound transducer”, is used to convert a sound signal into an electrical signal. The headset interface 164 is used to connect a wired headset. The headset interface 164 can be a USB interface, an open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0085] In this way, the mobile phone 10 can implement an audio function through the speaker 161, the receiver 162, the microphone 163, the headset interface 164, and an application processor in the audio module 160. For example, the user’s voice input, voice / music playback, and the like.

[0086] The communication module 170 is used to implement a communication function of the mobile phone 10. Specifically, the communication module 170 can be implemented through an antenna, a mobile communication module, a wireless communication module, a modem processor, and a baseband processor.

[0087] Antennas are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 10 can be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 used for the mobile communication module can be multiplexed as a diversity antenna for the wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0088] The mobile communication module can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. on the mobile phone 10. The mobile communication module can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves from the antennas, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed signals to the modem processor for demodulation. The mobile communication module can also amplify the signals modulated by the modem processor, and convert the signals into electromagnetic waves for radiation via the antennas. In some embodiments, at least part of the functional modules of the mobile communication module can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module can be disposed in the same device as at least part of the modules of the processor 110. The modem processor can include a modulator and a demodulator.

[0089] The wireless communication module can provide a solution for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), GNSS, frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. on the mobile phone 10. The wireless communication module can be one or more devices integrated with at least one communication processing module. The wireless communication module receives electromagnetic waves via the antennas, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module can also receive signals to be transmitted from the processor 110, perform frequency modulation and amplification on the signals, and convert the signals into electromagnetic waves for radiation via the antennas.

[0090] For example, the GNSS in the embodiments of this application may include: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation System (SBAS), etc.

[0091] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0092] It should be noted that when the aforementioned processing device is a terminal device, the above-mentioned audio data processing method can be implemented through an application (App) installed on the terminal device. This App has audio editing capabilities. For example, the App could be a music editing app, etc.

[0093] The app can be an app with human intervention capabilities. Here, human intervention means that the app can receive and respond to user input commands. In other words, the app can interact with the user. The app can include an interactive interface for interacting with the user, which is displayed on the terminal device's screen (e.g., ...). Figure 1 The display panel 151 shown is displayed.

[0094] It should be understood that if the terminal device includes a touchscreen (e.g.) Figure 1 The touchscreen 150 shown can be used by the user to operate the terminal device's touchscreen (e.g., operate...). Figure 1 The touchpad 152 shown enables interaction with the App. If the terminal device does not include a touchscreen (e.g., the terminal device is a regular desktop computer), the user can interact with the App through input / output devices such as a mouse and keyboard on the terminal device.

[0095] It should also be understood that the aforementioned App can be an embedded application installed on the terminal device (i.e., a system application of the terminal device) or a downloadable application.

[0096] The embedded application is an application provided by an operating system of a device (such as a mobile phone). For example, the embedded application can be a music application provided by the mobile phone when the mobile phone is manufactured. The downloadable application is an application that can provide its own communication connection. The downloadable application is an App that can be pre-installed in the device, or a third-party App downloaded and installed in the device by the user. For example, the downloadable application can be a music clip App, which is not limited in the embodiments of the present application.

[0097] It should be further noted that the processing apparatus described above can also be a server. In this case, the embodiments of the present application further provide a processing system of audio data. The processing system includes a server and a terminal device, and the server and the terminal device can be connected and communicated through wired or wireless means.

[0098] As shown in Figure 2 , Figure 2 A schematic diagram of a processing system 20 provided by the embodiments of the present application is shown. The processing system 20 includes a terminal device 21 and a server 22. The terminal device 21 can interact with the user through a client App (such as a client App of audio clip), for example, receive the instruction input by the user, and transmit the received instruction to the server 22. Then, the server 22 is used to execute the processing method of audio data provided by the embodiments of the present application based on the instruction received from the terminal device 21, and transmit the generated MIDI information of the target mashup audio and / or the target mashup audio to the terminal device 21. In this way, the terminal device 21 can receive the MIDI information of the target mashup audio and / or the target mashup audio sent by the server 22, and play the target mashup audio to the user through an audio module, and / or display the MIDI spectrum of the target mashup audio to the user through a display screen. This is not limited.

[0099] The processing method of audio data provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0100] Referring to Figure 3 , Figure 3 A flowchart of a processing method of audio data provided by the embodiments of the present application is shown. The method is executed by the audio data processing apparatus described above. In the following, the audio data processing apparatus described above is taken as an example of a terminal device, and the method can include the following steps:

[0101] S101, m audio segments are acquired, and m is an integer greater than or equal to 2.

[0102] The m audio segments include segments in different audios.

[0103] Specifically, the terminal device can first determine the k target audios, and then extract m audio clips from the k target audios. Wherein, 2≤k≤m, and k is an integer.

[0104] That is, the terminal device can extract at least one audio clip from a target audio.

[0105] S102, determining m-1 transition audio information according to the m audio clips obtained above.

[0106] Wherein, the m-1 transition audio information is used to link the m audio clips. For the first transition audio information in the m-1 transition audio information, the first transition audio information is the transition audio information used to link the first audio clip and the second audio clip in the m audio clips obtained by the terminal device in order. It should be understood that the order here is the order of the stringing sequence of the m audio clips. It should be understood that the stringing sequence is determined by the terminal device in advance.

[0107] Specifically, the terminal device can determine the m-1 transition audio information according to the preset neural network model and the information of the m audio clips obtained in S101. Here, the transition audio information is the MIDI information of the transition audio segment.

[0108] S103, generating the MIDI information of the target stringing audio according to the m audio clips and the m-1 transition audio information.

[0109] The terminal device connects the MIDI information of the m audio clips through the m-1 transition audio information (i.e. the MIDI information of the transition audio segment), that is, generates the MIDI information of the target stringing audio.

[0110] S104, determining the target stringing audio according to the MIDI information of the target stringing audio determined above.

[0111] Optionally, the terminal device can also output the target stringing audio after determining the target stringing audio based on the MIDI information of the target stringing audio.

[0112] By the method provided in the embodiment of the present application, the terminal device can generate m-1 pieces of transition audio information for linking the m pieces of audio segments based on the m pieces of audio segments. In this way, the m pieces of audio segments can be linked by the m-1 pieces of transition audio information, so that the MIDI information of the target mashup audio after the m pieces of audio segments are mashed up is obtained. In this way, when the terminal device converts the MIDI information of the target mashup audio into an audio format, the target mashup audio is obtained. It can be seen that when the plurality of audio segments are mashed up in this way, the terminal device can generate a brand-new transition audio segment for linking the plurality of audio segments, and therefore the method provided in the embodiment of the present application does not need to consider the similarity of the audio segments used to obtain the target mashup audio. That is, the method provided in the embodiment of the present application can obtain more abundant and diversified mashup audios.

[0113] In addition, the method provided in the embodiment of the present application generates the transition audio information in the MIDI domain. Since the MIDI information of the audio is the most original form of the audio, the MIDI information records the pitch, intensity and duration of the notes of the audio. Therefore, compared with directly mashing up the plurality of audio segments in the time domain, the method provided in the embodiment of the present application can generate the transition audio information for linking the two audio segments based on the audio theory, and the mashup audio obtained based on the transition audio information is more smooth and natural in hearing. In addition, processing the data in the MIDI domain is more conducive to the flexibility and consistency of the mashup audio in the later rendering.

[0114] The S101-S104 are described in detail as follows:

[0115] In S101, it can be understood that the audio in S101 can be a complete / incomplete song / music, and the audio segment is a segment of audio cut from the audio. It should also be understood that the audio or the audio segment has a time sequence.

[0116] Optionally, the terminal device can randomly determine k audios in the media database or the locally stored music database as the k target audios.

[0117] Optionally, the terminal device can first receive a first operation input by a user and determine the k target audios in response to the first operation. It can be understood that the terminal device is installed with an application program having an audio editing function, and the first operation is an operation of the user on an audio editing interface of the application program.

[0118] In a possible implementation, the first operation can include a selection operation of the target audio on the audio editing interface. Optionally, the selection operation can include a selection operation of the music database and an operation of selecting the target audio in the selected music database.

[0119] The music database can be a locally stored music database, a music database categorized by the system based on audio scene tags, emotion tags, style tags, etc., a music database automatically recommended by the system, or a custom music database configured by the user by deleting or adding audio from the system-recommended or categorized music database. This application embodiment does not limit the scope of the database. The system can be any media system that connects and communicates with an application with audio editing capabilities installed on the terminal device; this application embodiment does not limit the scope of the system either.

[0120] As an example, the system's recommended music database could be one based on the user's current scene / state detected by the terminal device's sensors. For instance, if the terminal device's sensors detect that the user is running, the system could recommend a music database including upbeat music. Alternatively, the system's recommended music database could be a randomly displayed streaming music database, such as various music charts, or a database including popular classical music. And so on.

[0121] It's important to note that each audio file can be tagged with elements such as scene, style, and emotion during production. Scene refers to the appropriate setting for listening to the audio, such as a work setting, a study setting, or a running setting. Style refers to the audio's musical style, such as rock, electronic, or light music. Emotion refers to the emotions expressed in the audio, such as sadness, longing, or loneliness. Further details on these will not be provided.

[0122] Taking a mobile phone (phone 10) as an example, and the phone 10 has an audio editing application installed, in one example, refer to... Figure 4 , Figure 4 This illustration shows a schematic diagram of a first operation input by a user on the audio editing interface of an audio editing application, according to an embodiment of this application.

[0123] like Figure 4 As shown in (a), the touchscreen of the mobile phone 10 displays an audio editing interface 401 of an audio editing application. It can be seen that the audio editing interface 401 is a music database selection interface under the "Categorized Music Library" tab. This allows the user to select music from the music database on the audio editing interface 401.

[0124] like Figure 4As shown in (a) of FIG. 4, on the audio clip interface 401, there are displayed type labels of the audio classified by the system based on different classification criteria of the audio. As shown in the figure, on the audio clip interface 401, there are displayed type labels of the audio classified based on the scene suitable for listening to the audio, such as a "work" label, a "running" label, and the like. On the audio clip interface 401, there are also displayed type labels of the audio classified based on the emotion expressed by the audio, such as a "happy" label, an "excited" label, and the like. On the audio clip interface 401, there are also displayed type labels of the audio classified based on the music style of the audio, such as a "pop" label, a "blues" label, and the like. It is easily understood that, Figure 4 The type labels of the audio and the display format thereof shown in (a) of FIG. 4 are only exemplary and do not limit the protection scope of the embodiments of the present application.

[0125] In this way, the user can operate (for example, click with a finger / touch pen) the type labels displayed on the audio clip interface 401 based on the user's own needs / preferences, for example, the user can click the "running" label, the "happy" label, the "excited" label, and the "blues" label with a finger, respectively, and after operating (for example, clicking with a finger / touch pen) the "OK" button on the audio clip interface 401, the mobile phone 10 displays, in response, an interface of all the audio with the "running" label, the "happy" label, the "excited" label, and the "blues" label recommended by the system based on the type labels selected by the user, for example, the target audio selection interface 402 shown in (b) of FIG. 4. It is understood that all the audio displayed on the target audio selection interface 402 constitutes the music database selected by the user. Figure 4

[0126] It is understood that when the type label selected by the user on the audio clip interface 401 is the "automatic" label, the audio displayed on the target audio selection interface 402 by the mobile phone 10 is the audio automatically recommended by the mobile phone 10 for the user to listen to in the current environment according to the current environment / state of the user operating the mobile phone 10 detected by the sensors (for example, a gyroscope sensor, a noise sensor, and the like) configured in the mobile phone 10. Details are not described herein.

[0127] Further, the user can perform a selection operation of the target audio on the target audio selection interface 402, for example, the user can select k target audios on the target audio selection interface 402 based on the user's own needs / preferences. In response, the mobile phone 10 determines the k target audios.

[0128] In another example, referring to Figure 5 , Figure 5 FIG. 5 shows another first operation input by a user on an audio clip interface of an audio clip application according to an embodiment of the present application.​

[0129] like Figure 5 As shown in (a), the touchscreen of mobile phone 10 displays an audio editing interface 501 of an audio editing application. It can be seen that the audio editing interface 501 is the music database selection interface displayed after the user selects the "Recommended Music Library" tab on the audio editing interface 401. In this way, the user can perform music database selection operations on the audio editing interface 501.

[0130] like Figure 5 As shown in (a), the audio editing interface 501 displays icons for multiple music databases showcased by the system. Examples include the "Popular Classics" music database icon, the "Internet Sweet Songs" music database icon, the "Light Music Collection" music database icon, and the "Golden Hits Chart" music database icon. This is easy to understand. Figure 5 The display format of the music database and its identifier shown in (a) is merely illustrative and is not intended to limit the scope of protection of the embodiments of this application. For example, the mobile phone 10 may also display the identifiers of different types of music databases on multiple interfaces, which is not a limitation.

[0131] In this way, users can operate (e.g., tap with a finger / stylus) a music database (e.g., a "Popular Classics" music library) displayed on the audio editing interface 501 according to their own needs or preferences. After operating (e.g., tapping with a finger / stylus) the "OK" button on the audio editing interface 501, the mobile phone 10 will display an interface for the audio in the "Popular Classics" music library, for example... Figure 5 The target audio selection interface 502 is shown in (b) of the diagram.

[0132] Furthermore, users can select target audio on the target audio selection interface 502. For example, users can select k target audios based on their needs / preferences on the target audio selection interface 502. In response, the mobile phone 10 will determine the k target audios.

[0133] In yet another example, refer to Figure 6 , Figure 6 This illustration shows a schematic diagram of a first operation input by a user on the audio editing interface of an audio editing application, as provided in another embodiment of this application.

[0134] like Figure 6 As shown, the touchscreen of mobile phone 10 displays an audio editing interface 601 of an audio editing application. It can be seen that audio editing interface 601 is the target audio selection interface displayed after the user selects the "Local Music Library" tab on audio editing interface 401 or audio editing interface 501. In this way, the user can select target audio on audio editing interface 501.

[0135] It can be understood that the operation of displaying the audio clip interface 601 after the user selects the "local music library" tab on the audio clip interface 401 or the audio clip interface 501 is equivalent to the operation of the user selecting the local music database on the audio clip interface 401 or the audio clip interface 501.

[0136] As shown in Figure 6 , the audio clip interface 601 displays a plurality of locally stored audios, for example, in the form of a list. In this way, the user can perform a selection operation of a target audio on the audio clip interface 601, for example, the user can select k target audios on the audio clip interface 601 based on the user's own needs / preferences. In response, the mobile phone 10 determines the k target audios.

[0137] It can be easily understood that the format of the display of the plurality of locally stored audios shown in Figure 6 is only exemplary and does not limit the embodiments of the present application. For example, the mobile phone 10 can also divide the plurality of locally stored audios into a plurality of groups and display the audio list in different groups in a plurality of hierarchical interfaces, which is not limited.

[0138] In another possible implementation, the first operation described above can include an input operation of the user inputting the number of target audios on the audio clip interface and a selection operation of the music database.

[0139] As an example, reference is made to Figure 7 , Figure 7 , which shows another first operation of the user inputting on the audio clip interface of the audio clip application according to an embodiment of the present application.

[0140] As shown in Figure 7 , Figure 7 , the audio clip interface 701 shown includes an interface for inputting the number of mashup audios (i.e., the input box 702), so that the user can input the number of mashup audios through the input box 702. For example, the value of k is 3, i.e., the user can input the value "3" in the input box 702.

[0141] In addition, the user can select a music database by operating the "music library" button on the audio clip interface 701 according to the user's own needs / preferences. Here, the process of displaying the selection of the music database after the user operates the "music library" button on the audio clip interface 701 can refer to the description of the selection of the music database in (a) of Figure 4 , (b) of Figure 5 , and Figure 6 , which will not be repeated here.

[0142] When the user selects a music database by operating the "music library" button on the audio clip interface 701, the mobile phone 10 can select k audio files in the music database selected by the user as target audio files according to the k value input by the user in the input box 702.

[0143] Alternatively, the mobile phone 10 can select k audio files in the music database selected by the user as target audio files according to the k value input by the user in the input box 702 based on preset rules. For example, the mobile phone 10 can randomly select k audio files in the music database as target audio files, or the mobile phone 10 can select the first k audio files in the music database as target audio files, etc. The embodiments of the present application do not limit this.

[0144] In this way, after the terminal device determines the k target audio files, the terminal device can extract m audio clips from the k target audio files by using a preset algorithm. As an example, the preset algorithm can be an algorithm for extracting the chorus / climax part of a song. The embodiments of the present application do not limit this.

[0145] Alternatively, the terminal device can be preset with the order of the m audio clips in the mashup, or the terminal device can further interact with the user to determine the order of the m audio clips in the mashup.

[0146] As an example, in combination with Figure 4 , Figure 5 and Figure 6 , when the user selects k target audio files on the target audio selection interface 402 shown as (b) in Figure 4 and operates the "OK" button on the target audio selection interface 402 (for example, clicks with a finger / stylus), or when the user selects k target audio files on the target audio selection interface 502 shown as (b) in Figure 5 and operates the "OK" button on the target audio selection interface 502 (for example, clicks with a finger / stylus), or when the user selects k target audio files on the audio clip interface 601 shown as (b) in Figure 6 and operates the "OK" button on the audio clip interface 601 (for example, clicks with a finger / stylus), the terminal device (i.e., the mobile phone 10) can display the mashup order selection interface 403 shown as (c) in Figure 4 . As shown as (c) in Figure 4 , the mashup order selection interface 403 can include "order", "random", and "custom" options.

[0147] When the user selects the "sequence" option, the mobile phone 10 can, in response, mix the m audio clips extracted from the k target audios in the order of the k target audios in the music database.

[0148] When the user selects the "random" option, the mobile phone 10 can, in response, mix the m audio clips extracted from the k target audios randomly.

[0149] When the user selects the "custom" option, the user can further input the identification (e.g., the number) of the k target audios in a preset order in the "custom" option box 4031. In this way, the mobile phone 10 can, in response, mix the m audio clips extracted from the k target audios in the preset order. The preset order is the order customized by the user.

[0150] Another example is shown in FIG. 7, where the audio clip interface 701 includes three options for inputting the order of the mixed songs. Figure 7 The audio clip interface 701 shown in FIG. 7 also includes three options for inputting the order of the mixed songs, which are described above in the description of (c) of FIG. 6, and thus are not described again here. Figure 4

[0151] At S102, after determining the m audio clips, the terminal device can determine the audio feature information of the m audio clips.

[0152] The audio feature information of the audio clips can include at least one of the main melody track position information, the style label, the emotion label, the rhythm information, the beat information, or the key information of the audio clips. It can be understood that the beat is the beat of the music, and the key is the key of the music. The specific implementation of the terminal device obtaining the audio feature information of the audio clips is not limited in the embodiments of the present application. The process of the terminal device determining the audio feature information of the m audio clips is not described in detail.

[0153] For each of the m audio clips determined by the terminal device, the terminal device can further separate the music and the vocals of each audio clip by using the music-vocal detection technology.

[0154] Taking a first audio clip of the m audio clips as an example, the terminal device can separate the vocals and the sounds of various musical instruments (e.g., piano, bass, drum, violin, etc.) in the first audio clip by using the music-vocal detection technology, and convert the separated multi-track musical instruments and vocals into MIDI format data, which is the MIDI information of the first audio clip. The music-vocal detection technology is not described in detail in the embodiments of the present application.​

[0155] It should be noted that the audio segment can also not include human voice. In this case, the terminal device can separate the multi-track instrument sound in the first audio segment through a music voice detection technology, and convert the multi-track instrument sound into MIDI format data.

[0156] Further, the terminal device can determine m-1 transition audio information according to the audio feature information and the MIDI information of the m audio segments and a preset neural network model. One of the m-1 transition audio information is transition audio information used to link two audio segments in the m audio segments obtained by the terminal device in sequence, where the sequence of the m audio segments refers to the sequence of the m audio segments, and the transition audio information is the MIDI information of the transition audio segment. Here, the detailed description of the terminal device determining the sequence of the m audio segments can refer to the related description in S101, which will not be repeated here.

[0157] The preset neural network model can be pre-stored in the terminal device or pre-stored on a server in communication connection with the terminal device, and the embodiments of the present application do not limit this.

[0158] The preset neural network model includes an encoder, an information extraction module, an information generation module, and a decoder. Here, the encoder, the information extraction module, the information generation module, and the decoder are all sequence model network structures, and the information extraction module is a bidirectional sequence model network structure.

[0159] As an example, the encoder, the information generation module, and the decoder can be RNN, LSTM, GRU, Transformer, etc. The information extraction module can be Bi-RNN, Bi-LSTM, Bi-GRU, Transformer, etc.

[0160] It should be understood that the preset neural network model includes at least two encoders, at least two information extraction modules, at least one information generation module, and at least one decoder. The network structures of the at least two encoders are the same, the network structures of the at least two information extraction modules are the same, the network structures of the at least one information generation module are the same, and the network structures of the at least one decoder are the same. In addition, in one preset neural network model, the network structures of the encoder and the decoder can also be the same. It should be noted that the data flow directions of the encoder and the decoder are opposite. As an example, the input of the encoder can be used as the output of the decoder, and the output of the encoder can be used as the input of the decoder.

[0161] It should be understood that the network parameters of the at least two encoders, at least two information extraction modules, at least one information generation module, and at least one decoder are all determined during the training of the preset neural network model. For a detailed explanation of training the preset neural network model, please refer to the training section below. Figure 8 The description of the preset neural network model 80 shown is not repeated here.

[0162] In one possible implementation, when the number of audio segments that the preset neural network model needs to process is m, the preset neural network model includes m inputs and m-1 outputs. In this case, the preset neural network model has m encoders, 2×(m-1) information extraction modules, and m-1 information generation modules and decoders.

[0163] In this case, the preset neural network model can process the information of the input m audio segments simultaneously and output the MIDI information of m-1 transition audio segments.

[0164] For example, let's take the value of m as 2. (See reference) Figure 8 , Figure 8 A schematic diagram of the structure of a preset neural network model provided in an embodiment of this application is shown.

[0165] like Figure 8 As shown, the preset neural network model 80 includes two encoders for receiving two inputs, namely encoder 811 and encoder 812. The preset neural network model 80 includes two (i.e., 2 × (2-1)) information extraction modules, namely information extraction module 821 and information extraction module 822. The preset neural network model 80 also includes one (i.e., (2-1)) information generation module (i.e., information generation module 83) and one (i.e., (2-1)) decoder (i.e., decoder 84).

[0166] Another example, taking the value of m as 4, see reference. Figure 9 , Figure 9 A schematic diagram of another preset neural network model provided in an embodiment of this application is shown.

[0167] like Figure 9As shown, the preset neural network model 90 includes 4 encoders for receiving 4 inputs, i.e., the encoder 911, the encoder 912, the encoder 913, and the encoder 914. The preset neural network model 90 includes 6 (i.e., 2 x (4-1)) information extraction modules, i.e., the information extraction module 921, the information extraction module 922, the information extraction module 923, the information extraction module 924, the information extraction module 925, and the information extraction module 926. The preset neural network model 90 includes 3 (i.e., (4-1)) information generation modules, i.e., the information generation module 931, the information generation module 932, and the information generation module 933. The preset neural network model 90 further includes 3 (i.e., (4-1)) decoders, i.e., the decoder 941, the decoder 942, and the decoder 943.

[0168] In another possible implementation, the number of encoders and information extraction modules in the preset neural network model are both 2, and the number of information generation modules and decoders are both 1 (e.g., as shown in the preset neural network model). That is, the preset neural network model includes 2 inputs and 1 output. Figure 8

[0169] In this case, the preset neural network model can process 2 inputs (i.e., information of 2 audio clips) at a time and output 1 transition audio information. When the number of audio clips that need to be processed by the preset neural network model is m, and m is greater than 2, the preset neural network model can process the information of m audio clips serially for m-1 times, i.e., m-1 transition audio information can be obtained. It should be understood that the two audio clips processed by the preset neural network model each time are two audio clips that are adjacent in the serial order.

[0170] As an example, when the audio clips that need to be processed by the preset neural network model include audio clip 1, audio clip 2, audio clip 3, and audio clip 4, i.e., the value of m is 4, and the serial order of the 4 audio clips is: audio clip 1→audio clip 4→audio clip 3→audio clip 2, the terminal device can process the information of the 4 audio clips serially for 3 (i.e., (4-1)) times, i.e., 3 transition audio information can be obtained. Specifically, the terminal device can take the information of audio clip 1 and audio clip 4 as the two inputs of the preset neural network, so that transition audio information 1 for connecting audio clip 1 and audio clip 4 can be obtained. The terminal device can take the information of audio clip 4 and audio clip 3 as the two inputs of the preset neural network, so that transition audio information 2 for connecting audio clip 4 and audio clip 3 can be obtained. The terminal device can further take the information of audio clip 3 and audio clip 2 as the two inputs of the preset neural network, so that transition audio information 3 for connecting audio clip 3 and audio clip 2 can be obtained.​

[0171] The following takes the value of m as 2, that is, the terminal device obtains two audio segments (for example, the two audio segments include the first audio segment and the second audio segment), and the string burning order is the first audio segment -> the second audio segment as an example, in combination with Figure 8 The process of processing the information of each module in the preset neural network model and the audio segment provided in the embodiments of the present application is described in detail. As can be seen from the string burning order, the first audio segment is the preceding musical section (referred to as the preceding musical section) in the target string burning audio, and the second audio segment is the subsequent musical section (referred to as the subsequent musical section) in the target string burning audio.

[0172] Here, the first audio segment as the preceding musical section, its audio feature information and MIDI information can be referred to as the first information. The second audio segment as the subsequent musical section, its audio feature information and MIDI information can be referred to as the second information. Among them, the first information can be used as an input of the preset neural network model 80, and the second information can be used as another input of the preset neural network model 80.

[0173] It should be noted that the multiple sound tracks in the MIDI information (i.e., the MIDI information of the first audio segment) in the first information and the multiple sound tracks in the MIDI information (i.e., the MIDI information of the second audio segment) in the second information are the same. Specifically, the number of sound tracks and their types included in the MIDI information in the first information are the same as the number of sound tracks and their types included in the MIDI information in the second information.

[0174] For example, the MIDI information in the first information includes 3 sound tracks, which are a vocal sound track, a piano sound track, and a violin sound track. Then, the MIDI information in the second information also includes these 3 sound tracks.

[0175] It should be understood that assuming that the number of sound tracks included in the MIDI information in the first information and the number of sound tracks included in the MIDI information in the second information are different, for example, the number of sound tracks included in the MIDI information in the first information is greater than the number of sound tracks included in the MIDI information in the second information, and the MIDI information in the first information includes all sound track types included in the MIDI information in the second information, the terminal device can add empty tracks to the MIDI information in the second information to make the number of sound tracks included in the MIDI information in the first information and the number of sound tracks included in the MIDI information in the second information the same.

[0176] In this way, the terminal device can input the first information as one input (e.g., input 1) of the preset neural network model 80 into the encoder 811. The encoder 811 can output a first sequence corresponding to the first audio segment after processing the received first information. Here, the first sequence is a sequence obtained by the encoder 811 performing feature extraction on the MIDI information and the audio feature information in the first information. Here, the first sequence can be understood as a sequence obtained by the encoder 811 reducing the dimension of the first information, or the first sequence can be understood as a sequence obtained by the encoder 811 compressing the first information to a hidden space.

[0177] It should be understood that the first sequence is a one-dimensional sequence in time sequence, and the length of the first sequence is determined by the length of the first audio segment. It can be understood that the longer the first audio segment, the longer the first sequence; the shorter the first audio segment, the shorter the first sequence.

[0178] As an example, the first sequence can be represented as “{P1, P2, …, P s}, where P represents a feature vector, and s represents the number of feature vectors. It should be understood that since the audio segment itself has a time sequence, the first sequence also has a time sequence. In this way, P1 can be a feature vector corresponding to the beginning time of the audio segment, and P s may be a feature vector corresponding to the end time of the audio segment.

[0179] Similarly, the terminal device can input the second information as another input (e.g., input 2) of the preset neural network model 80 into the encoder 812, so that the encoder 812 can output a second sequence corresponding to the second audio segment after processing the received second information. The description of the second sequence can be referred to the first sequence, and will not be repeated here.

[0180] As an example, the second sequence can be represented as “{F1, F2, …, F t}, where F represents a feature vector, and t represents the number of feature vectors. Wherein, F1 can be a feature vector corresponding to the beginning time of the audio segment, and F t may be a feature vector corresponding to the end time of the audio segment.

[0181] Then, the information extraction module 821 receives the first sequence output by the encoder 811. Since the audio segment itself has a time sequence, and the first audio segment is a prelude, the information extraction module 821 can output a first vector corresponding to the end time of the first audio segment after learning the first sequence. This process can also be understood as the information extraction module 821 further reducing the dimension of the first sequence. It should be noted that the first vector carries the features of the first sequence and corresponds to the end time of the first sequence.

[0182] Similarly, the information extraction module 822 receives the second sequence output by the encoder 812. After the information extraction module 821 learns the second sequence, a second vector corresponding to the start time of the second audio segment can be output. This process can also be understood as the information extraction module 822 further reducing the dimension of the second sequence. It should be noted that the second vector carries the characteristics of the second sequence and corresponds to the start time of the second sequence.

[0183] Then, the preset neural network model 80 sums the first vector and the second vector to obtain a third vector. That is, the third vector = the first vector + the second vector. Here, the sum of the hidden space vector corresponding to the end time of the first audio segment (i.e., the first vector) and the hidden space vector corresponding to the start time of the second audio segment (i.e., the second vector) (i.e., the third vector) can be used as the hidden space vector corresponding to the transition audio segment that connects the first audio segment and the second audio segment.

[0184] In this way, based on the transition vector, the preset neural network model 80 can determine the transition audio segment used to connect the first audio segment and the second audio segment. Specifically, the preset neural network model 80 inputs the third vector into the information generation network module 83, and the information generation network module 83 can learn the received third vector and output a third sequence. It should be understood that the third sequence is a sequence composed of feature vectors of the transition audio segment used to connect the first audio segment and the second audio segment.

[0185] As an example, the third sequence can be represented as “{M1, M2, …M j}”, where M represents a feature vector, and j represents the number of feature vectors. M1 can be a feature vector corresponding to the start time of the transition audio segment used to connect the first audio segment and the second audio segment, and M j may be a feature vector corresponding to the end time of the transition audio segment.

[0186] Then, the decoder 84 receives the third sequence output by the information generation module 83, learns the third sequence, and outputs the transition audio information used to connect the first audio segment and the second audio segment, i.e., the MIDI information of the transition audio segment.

[0187] It should be understood that the preset neural network model 80 shown in the above Figure 8 may be a neural network model trained in advance based on a plurality of training samples. One training sample includes MIDI information and audio feature information of two audio segments, and the label value of the training sample is the MIDI information of the transition audio segment constructed by the domain expert according to the two audio segments. In this way, the neural network is repeatedly iteratively trained based on a plurality of training samples, and the preset neural network model shown in the above Figure 8 is obtained.

[0188] At S103, the terminal device inserts the m-1 transition audio information (i.e., the MIDI information of the transition audio segment) generated at S102 into the MIDI information of the m audio segments, respectively, to achieve the purpose of connecting the MIDI information of the m audio segments by the m-1 transition audio information, i.e., to generate the target mashup audio obtained after the m target audio segments are mashed up.

[0189] As an example, assuming that m is 3, the m audio segments include audio segment 1, audio segment 2, and audio segment 3, and the m-1 transition audio information includes transition audio information 1 and transition audio information 2. When the mashup order of the 3 audio segments is audio segment 1→ audio segment 3→ audio segment 2, and the transition audio information 1 is the transition audio information for connecting audio segment 1 and audio segment 3, and the transition audio information 2 is the transition audio information for connecting audio segment 3 and audio segment 2, the terminal device can insert the transition audio information 1 between the MIDI information of audio segment 1 and audio segment 3, and can insert the transition audio information 2 between the MIDI information of audio segment 3 and audio segment 2. In this way, the terminal device generates the MIDI information of the target mashup audio obtained after audio segment 1, audio segment 2, and audio segment 3 are mashed up.

[0190] In actual application, after the terminal device generates the MIDI information of the target mashup audio, the terminal device can play the target mashup audio to the user. When the user thinks that the target mashup audio is not the mashup audio that the user wants, the user can input a second operation to the terminal device. In this way, the terminal device responds to the second operation, adjusts the mashup order of the m audio segments used to mash up the target mashup audio, and re-generates the m-1 transition audio information and the MIDI information of the target mashup audio based on the adjusted mashup order and the m audio segments.

[0191] Then, the terminal device can play the re-generated target mashup audio to the user. When the user is satisfied with the target mashup audio, S104 can be executed; when the user is not satisfied with the target mashup audio, the user can input the second operation to the terminal device again, so that the terminal device adjusts the mashup order of the m audio segments again and re-generates the MIDI information of the target mashup audio again. It can be seen that through repeated interaction between the terminal device and the user, the target mashup audio that satisfies the user can be obtained, and the user experience is improved.

[0192] In a possible implementation, after the terminal device generates the MIDI information of the target mashup audio, the terminal device displays the MIDI score of the target mashup audio in the form of a graph on the display panel. In this way, the second operation of the user can be a drag operation on the MIDI score. In this way, after receiving the second operation of the user, the terminal device can respond to the second operation to redetermine the mashup order of the m audio clips used to mash up the aforementioned target mashup audio.

[0193] In this way, based on the redetermined mashup order of the m audio clips and the m audio clips, the terminal device can redetermine the m-1 transition audio information by performing S102. Further, based on the redetermined m-1 transition audio information and the m audio clips, the terminal device can regenerate the target mashup audio. Here, the process in which the terminal device regenerates the target mashup audio can refer to the detailed description of the process in which the terminal device generates the target mashup audio, which will not be described herein.

[0194] As an example, taking the terminal device as the mobile phone 10, and taking the MIDI information of the target mashup audio generated by the mobile phone 10 as including three sound tracks, and taking the target mashup audio as a mashup audio obtained by mashup of the audio clip 1, the audio clip 2, and the audio clip 3 in the mashup order of audio clip 1->audio clip 3->audio clip 2, referring to Figure 10 , (a), Figure 10 , (a) shows a schematic diagram of a second operation provided in an embodiment of the present application.

[0195] As shown in Figure 10 , (a), when the mobile phone 10 generates the MIDI information of the target mashup audio for the first time, the mobile phone 10 can display the MIDI score of the target mashup audio on the display panel, for example, the MIDI score displayed on the mashup audio editing interface 1001 shown in Figure 10 , (a).

[0196] The MIDI score displayed on the mashup audio editing interface 1001 includes three sound tracks, which are a sound track 1 shown by a black band, a sound track 2 shown by a white band, and a sound track 3 shown by a striped band.

[0197] The starting line on the MIDI score displayed on the mashup audio editing interface 1001 is used to mark the start of the target mashup audio. The MIDI score displayed on the mashup audio editing interface 1001 further includes a plurality of division lines, which are used to distinguish different audio clips and transition audio segments in the target mashup audio.

[0198] For example, based on the medley order of audio segments in the target medley audio, the audio segment between the starting line and split line 1 is audio segment 1; the audio segment between split line 1 and split line 2 is transitional audio segment 1 connecting audio segment 1 and audio segment 3; the audio segment between split line 2 and split line 3 is audio segment 3; the audio segment between split line 3 and split line 4 is transitional audio segment 2 connecting audio segment 3 and audio segment 2; and the audio segment to the right of split line 4 is audio segment 2. Figure 10 (In (a) of the image, the MIDI score does not show the termination line used to mark the end of the target medley audio). It is understood that the names of each audio segment can also be displayed on the MIDI score displayed on the medley audio editing interface 1001, and this embodiment of the application does not limit this.

[0199] When a user interacts with the play icon 1002 on the medley audio editing interface 1001 (e.g., by tapping it with a finger or stylus), the mobile phone 10 plays the target medley audio to the user in response. If the user is not satisfied with the target medley audio, they can input a second operation into the mobile phone 10. This second operation could be a dragging operation of the MIDI score displayed on the medley audio editing interface 1001 (e.g., sliding a finger or stylus on the display panel), for example, the user pressing and holding the MIDI score of audio segment 1 (i.e.,...). Figure 10 (a) shows the area between the start line and the dividing line 1 in the MIDI score, and along... Figure 10 Slide the arrow in (a) to the MIDI score of audio clip 2 (i.e., Figure 10 The area to the right of the dividing line 4 in the MIDI score shown in (a) is located at this position. In response, the mobile phone 10 swaps the medley order of audio clip 1 and audio clip 2, that is, the mobile phone 10 redetermines the medley order of audio clip 1, audio clip 2, and audio clip 3 as audio clip 2 → audio clip 3 → audio clip 1.

[0200] Furthermore, the mobile phone 10 can regenerate the target medley audio based on audio segment 1, audio segment 2, audio segment 3, and the redefined medley order.

[0201] In another implementation, the second operation can be the user inputting the target medley order on the audio editing interface displayed after the terminal device generates the MIDI information for the target medley audio. In response, the terminal device receives the user-input target medley order. That is, the terminal device redetermines the target medley order of the m audio segments.

[0202] In this way, the terminal device can determine the target crossfading order of the m audio segments based on the received target crossfading order of the m audio segments, and the m audio segments, and re-determine m-1 pieces of transition audio information by performing S102. Further, the terminal device can re-generate the target crossfaded audio based on the m-1 pieces of transition audio information and the m audio segments.

[0203] As an example, taking the terminal device as the mobile phone 10, and the target crossfaded audio as the crossfaded audio obtained by crossfading the audio segment 1, the audio segment 2 and the audio segment 3 in the crossfading order of audio segment 1→audio segment 3→audio segment 2, reference is made to Figure 10 (b), and Figure 10 (b) shows another schematic diagram of the second operation provided by the embodiments of the present application.

[0204] As shown in Figure 10 (b), when the mobile phone 10 generates the MIDI information of the target crossfaded audio for the first time, the mobile phone 10 can display a crossfaded audio editing interface 1001 as shown in Figure 10 (b) on the display panel. When the user can operate (for example, click with a finger or a touch pen) the play icon 1002 on the crossfaded audio editing interface 1001, the mobile phone 10 can play the target crossfaded audio in response. When the user is not satisfied with the target crossfaded audio, the user can input a second operation to the mobile phone 10.

[0205] Specifically, the user can input the desired target crossfading order in the input box 1003 of the target crossfading order on the crossfaded audio editing interface 1001, for example, the user inputs “2, 3, 1” in the input box 1003, where “2” can be used to represent the identifier of the audio segment 2, “3” can be used to represent the identifier of the audio segment 3, and “1” can be used to represent the identifier of the audio segment 1, and “2, 3, 1” can be used to represent the crossfading order of the audio segment 1, the audio segment 2 and the audio segment 3 as audio segment 2→audio segment 3→audio segment 1. In response, the mobile phone 10 receives the target crossfading order input by the user. In this way, the mobile phone 10 determines the target crossfading order of the audio segment 1, the audio segment 2 and the audio segment 3.

[0206] Then, the mobile phone 10 can re-generate the target crossfaded audio based on the audio segment 1, the audio segment 2, the audio segment 3, and the received target crossfading order.

[0207] Optionally, as shown in Figure 10 (b), the crossfaded audio editing interface 1001 can also display the current crossfading order, for example, “Current crossfading order: 1, 3, 2”. It should be understood that the current crossfading order can be used as a reference when the user provides the input target crossfading order.

[0208] It should be understood that the above possible ways of implementing the second operation are only exemplary and do not constitute a limitation on the protection scope of the embodiments of the present application.

[0209] At S104, optionally, the terminal device can directly save and output the latest MIDI information of the target jam audio generated at S103.

[0210] Optionally, the terminal device can also synthesize the time-domain waveform of the target jam audio according to the latest MIDI information of the target jam audio generated at S103, thereby obtaining the target jam audio. Optionally, the terminal device can also save / output the target jam audio.

[0211] Wherein, the embodiments of the present application do not limit the specific way in which the terminal device synthesizes the time-domain waveform of the target jam audio according to the MIDI information of the target jam audio. As an example, the terminal device can synthesize the time-domain waveform of the target jam audio by loading a timbre library for the MIDI information of the target jam audio, or can synthesize the time-domain waveform of the target jam audio according to the MIDI information of the target jam audio and a wave table (a wave table is a file pre-recorded with all the sounds (including various octaves, tones, etc.) that can be emitted by various real musical instruments), or can also synthesize the time-domain waveform of the target jam audio according to the MIDI information of the target jam audio and a physical model / neural network model, of course, not limited to this. Here, the physical model / neural network model is a model pre-constructed for synthesizing audio waveforms, which is not described in detail in the embodiments of the present application.

[0212] It should be understood that when the terminal device synthesizes the time-domain waveform of the target jam audio according to the MIDI information of the target jam audio, it only synthesizes the time-domain waveform of the target jam audio according to all the instrument tracks in the MIDI information of the target jam audio except the vocal track.

[0213] Optionally, the terminal device can receive a third operation of the user and, in response to the third operation, render the latest MIDI information of the target jam audio generated at S103, and synthesize the time-domain waveform of the target jam audio according to the rendered MIDI information. Then, the terminal device further renders the synthesized time-domain waveform of the target jam audio to obtain the rendered target jam audio. Optionally, the terminal device can also save / output the rendered target jam audio.

[0214] Wherein, the detailed description of the terminal device synthesizing the time-domain waveform of the target jam audio according to the rendered MIDI information can refer to the description of the terminal device synthesizing the time-domain waveform of the target jam audio according to the MIDI information of the target jam audio above, which is not described here.

[0215] The third user operation may include selecting an audio rendering processing method input by the user in the audio rendering interface, and selecting a processing method for synthesizing the time-domain waveform of the target medley audio, also input by the user in the audio rendering interface. Rendering the MIDI information of the target medley audio may include, for example, source separation. The processing method for synthesizing the time-domain waveform of the target medley audio may include loading a timbre library, wavetable synthesis, or physical model synthesis. Rendering the time-domain waveform of the target medley audio may include mixing, vocal style transfer, etc., which are not limited in this embodiment.

[0216] Optionally, the terminal device can save the time-domain waveform of the rendered target cascaded audio as an audio file of any audio format; this embodiment of the application does not specifically limit this. As an example, the terminal device can save the rendered target cascaded audio waveform as an audio file in WAV format, an audio file in free lossless audio codec (FLAC) format, an audio file in moving picture expertsgroup audio layer III (MP3) format, an audio compression format (OGGVobis, ogg), etc., and is not limited to these.

[0217] Optionally, the terminal device can also save the project that generated the target audio montage. In this way, the terminal can reset the montage order of the m audio segments used to montage the target audio montage based on the saved project file, and then re-process the montage, which can improve the efficiency when montaging the same m audio segments again in the future.

[0218] As an example, taking the mobile phone 10 as the terminal device, refer to... Figure 11 , Figure 11 This illustration shows a schematic diagram of rendering and outputting MIDI information of a target medley audio file according to an embodiment of this application.

[0219] like Figure 11 As shown in (a), after the mobile phone 10 finally determines the MIDI information of the target medley audio, the mobile phone 10 can display the audio rendering interface 1101. The user can input a third operation on the audio rendering interface 1101, which may include: the user selecting to enable "Remove Vocals" under the "Source Separation" option. Figure 11The options include (a) the selection operation of "Remove Vocals" (black square indicates on, white square indicates off), the selection operation of "Load Sound Library" under "Audio Waveform Synthesis", the selection operation of "Record Vocals" under "Mix", and the selection operation of "Singer A" as the migration target under "Vocal Style Migration".

[0220] In response, after receiving the user's third operation and the user's operation on the "OK" button on the audio rendering interface 1101 (e.g., clicking), the mobile phone 10 can delete or invalidate the vocal track in the MIDI information of the target medley audio, load the timbre library into the MIDI information of the target medley audio to synthesize the time-domain waveform of the target medley audio, then open the recording interface to record the vocals of the target medley audio, and transfer the vocals in the target medley audio to the voice of singer A.

[0221] Furthermore, after receiving a third user action and the user's action on the "OK" button on the audio rendering interface 1101 (e.g., clicking), the phone 10 can display the following: Figure 10 The audio publishing interface 1102 is shown in (b) above. In this way, the terminal device 10 can interact with the user through the audio publishing interface 1102 and export the target cascaded audio according to the user's input instructions.

[0222] like Figure 11 As shown in (b), the mobile phone 10 can receive user input to select the audio format to export under the "Export Format" option of the audio publishing interface 1102, such as selecting "Audio Format 1". The mobile phone 10 can also receive user input to specify the name (e.g., name A) and path of the target medley audio file under the "Export Path" option of the audio publishing interface 1102. Furthermore, the mobile phone 10 can receive user input to enable the "Save Project" function under the "Save Project" option of the audio publishing interface 1102.

[0223] In this way, when the user operates on the "Export" button on the audio publishing interface 1102 (for example, by clicking it), the mobile phone 10 will save the target medley audio according to the user's instructions.

[0224] In some embodiments, the method portion of determining transitional audio information and generating target audio (i.e., steps S102-S104) in the method provided by this application can also be performed during the process of the terminal device playing audio to the user in real time. In this case, the method portion of determining transitional audio information and generating target audio in the method provided by this application can be implemented by a functional module of an App capable of providing audio listening.

[0225] As an example, the App capable of providing audio listening can be a cloud music App. For simplicity, the following description takes the cloud music App as an example.

[0226] Specifically, the cloud music App can provide a mix mode when providing a music listening mode to a user. The mix mode can be implemented by executing steps S102-S104 in the method provided by the present application on a terminal device running the cloud music App, or a server connected to the cloud music App. For simplicity, the following description takes the mix mode implemented by executing steps S102-S104 in the method provided by the present application on a terminal device running the cloud music App as an example.

[0227] In this case, optionally, when the terminal device plays music for the user through the cloud music App running in the terminal device, the music played by the terminal device can be music automatically recommended by the cloud music media library, or music in the local media library, which is not limited by the present application.

[0228] In this way, when the terminal device determines to play music for the user through the above mix mode through interaction with the user, the terminal device can take the current music being played for the user and the next music to be played for the user as two target audios. And based on the two target audios and a preset mix order, the terminal device can execute the above-mentioned S102-S104 to generate a first target mix audio obtained by mixing the two target audios. The preset mix order is: the current music being played for the user by the terminal device→the next music to be played for the user by the terminal device.

[0229] It should be noted that when the music currently played by the terminal device is music automatically recommended by the cloud music media library, the terminal device can determine the next music automatically recommended for the user in the process of playing the current music for the user, which is the next music to be played for the user by the terminal device.

[0230] It should be noted that in the process of playing the current music for the user, the terminal device can complete the mixing of the two target audios and obtain the first target mix audio.

[0231] Optionally, the terminal device can play the first target mix audio for the user after playing the current music for the user. Further, the terminal device can play the original next music for the user after playing the first target mix music for the user.

[0232] As an example, if the current music being played to the user by the terminal device is music 1, and the next music originally to be played is music 2, the terminal device can play the first target mixtape audio to the user after playing music 1 to the user. Then, the terminal device plays music 2 to the user after playing the first target mixtape music to the user.

[0233] Similarly, when the terminal device plays the originally next music to the user, the originally next music becomes the new current music being played to the user by the terminal device. In this way, the terminal device can repeat the above process to generate a second target mixtape audio obtained by mixing the new current music and the next music of the new current music. And the terminal device can play the second mixtape audio to the user after playing the new current music to the user.

[0234] It can be seen that when the terminal device plays music to the user in the mixtape mode, the terminal device can dynamically generate mixtape audio of the current music and the next music, and play the mixtape audio to the user, thereby improving the user experience.

[0235] It can also be understood that when the terminal device plays music to the user in the mixtape mode, the terminal device can only play the audio segments of the current music and the next music used to generate the mixtape audio when playing the current music and the next music to the user, for example, only play the chorus / climax part of the current music and the next music, and the embodiments of the present application do not limit this.

[0236] It should be noted that when the terminal device generates the target mixtape audio (e.g., the first target mixtape audio, the second target mixtape audio,..., the qth target mixtape audio, q is a positive integer) based on the determined two target audios and the preset mixtape order, in S103, the terminal device only needs to execute the process of generating the transition audio information once, and does not need to receive the second operation input by the user.

[0237] In addition, the cloud music App has preset rendering mode and preset export mode of the target mixtape audio in the mixtape mode. The preset rendering mode includes at least one of the sound source separation processing mode, the mixing processing mode, the sound migration mode, etc. The preset export mode includes the export format of the mixtape audio, and instructions whether to save the engineering of the target mixtape audio, etc. Therefore, in S104, the terminal device does not need to obtain the rendering mode and the export mode of the target mixtape audio by interacting with the user.

[0238] It should be understood that the preset rendering mode and the preset export mode of the target mashup audio in the mashup mode of the cloud music App can be pre-configured through the interaction between the terminal device and the user, or can be configured through the interaction with the user during the process of playing music to the user by the terminal device, which is not limited here. It should also be understood that after the preset rendering mode and the preset export mode of the target mashup audio are configured, the terminal device can also update the pre-configured preset rendering mode and preset export mode through the interaction with the user during the process of playing music to the user, which is not limited in the embodiments of the application.

[0239] Of course, the preset export mode can also not include an indication of whether to save the project of the target mashup audio. In this case, the terminal device can receive the indication of whether to save the project of the target mashup audio input by the user through the interaction with the user when stopping playing music to the user after the interaction with the user, and save the project of the target mashup audio based on the indication input by the user. It can be understood that before this, the terminal device can cache all the projects of the dynamically generated target mashup audio.

[0240] It should also be understood that the name of the above-mentioned "mashup mode" is only an example and is not limited to the embodiments of the application.

[0241] In summary, the embodiments of the application provide a method for processing audio data. Through this method, the embodiments of the application can generate m-1 transition audio information for linking m audio segments in the MIDI domain based on m audio segments. In this way, the MIDI information of the m audio segments can be linked through the m-1 transition audio information, so as to obtain the target mashup audio after the m audio segments are mashed up. As can be seen, when multiple audio segments are mashed up by the method of the application, the terminal device can generate a new transition audio segment for linking the multiple audio segments, so that the method provided by the embodiments of the application does not need to consider the similarity of the audio segments for obtaining the target mashup audio. That is, the method provided by the embodiments of the application can obtain more rich and diverse mashup audios.

[0242] Moreover, since the MIDI information of the audio is the most original form of the audio, it records the information of the note pitch, note dynamics, note duration, etc. Therefore, compared with directly mashing up multiple audio segments in the time domain, the transition audio information for linking two audio segments generated in the method provided by the embodiments of the application is generated based on audio music theory, so that the mashup audio obtained based on the transition audio information is more smooth and natural in hearing. Moreover, processing data in the MIDI domain is also more conducive to the flexibility and consistency of the mashup audio in the later rendering.

[0243] In addition, the user can participate in the method provided by the embodiment of the present application when the m audio segments are mixed, so that the mixed audio that satisfies the user can be obtained, that is, the user experience is high.

[0244] The above describes the scheme provided by the embodiment of the present application from the perspective of the method. To implement the above functions, the hardware structure and / or software module corresponding to each function are included. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driven hardware depends on the specific application of the technical solution and the design constraint conditions. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0245] The embodiment of the present application can divide the functional modules of the processing device of the audio data according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be implemented in the form of hardware or software functional module. It should be noted that the division of the modules in the embodiment of the present application is illustrative, and is only a logical function division. When actually implemented, there can be another division method.

[0246] As shown in Figure 12 , Figure 12 A structure diagram of an audio data processing device 120 provided by the embodiment of the present application is shown. The processing device 120 can be used to execute the above audio data processing method, for example, to execute the method shown in Figure 3 . The processing device 120 can include an acquisition unit 121, a determination unit 122, and a generation unit 123.

[0247] The acquisition unit 121 is configured to acquire m audio segments. The determination unit 122 is configured to determine m-1 transition audio information according to the m audio segments. The generation unit 123 is configured to generate a target mixed audio according to the m audio segments and the m-1 transition audio information. The m-1 transition audio information is used to connect the m audio segments. For the first transition audio information in the m-1 transition audio information, the first transition audio information is used to connect a first audio segment and a second audio segment that are sequentially continuous in the m audio segments. Here, the sequence of the m audio segments refers to the mixing order of the m audio segments.

[0248] As an example, in combination with Figure 3The acquisition unit 121 can be configured to perform S101, the determination unit 122 can be configured to perform S102, and the generation unit 123 can be configured to perform S103-S104.

[0249] Optionally, the determination unit 122 is specifically configured to determine the first transition audio information according to the first information of the first audio segment and the second information of the second audio segment. The first information includes MIDI information and audio feature information of the first audio segment, the second information includes MIDI information and audio feature information of the second audio segment, and the first transition audio information includes MIDI information of the first transition audio corresponding to the first transition audio information.

[0250] As an example, in combination with Figure 3 The determination unit 122 can be configured to perform S102.

[0251] Optionally, the audio feature information includes at least one of main melody track position information, style tags, emotion tags, rhythm information, beat information, or key information of the audio segment.

[0252] Optionally, the determination unit 122 is specifically configured to determine the first transition audio information according to the first information of the first audio segment, the second information of the second audio segment, and a preset neural network model.

[0253] As an example, in combination with Figure 3 The determination unit 122 can be configured to perform S102.

[0254] Optionally, when the first audio segment is located before the second audio segment in the target mashup audio, the first transition audio information is determined based on a feature vector used to represent the first transition audio information, and the feature vector of the first transition audio information is determined based on a first vector and a second vector. The first vector is a feature vector generated at the time sequence end of the first audio segment according to the first information, and the second vector is a feature vector generated at the time sequence start of the second audio segment according to the second information.

[0255] Optionally, the determination unit 122 is further configured to determine k target audios in response to a first operation of a user. The acquisition unit 121 is specifically configured to extract m audio segments from the k target audios. 2≤k≤m, and k is an integer.

[0256] As an example, in combination with Figure 3 The determination unit 122 and the acquisition unit 121 can be configured to perform S101,

[0257] Optionally, the determination unit 122 is further configured to determine a mashup order of the m audio segments before determining m-1 transition audio information according to the m audio segments.

[0258] Optionally, the determining unit 122 is further configured to determine the sequence of the m audio segments again in response to a second operation of the user. The determining unit 122 is further configured to determine the m-1 transition audio information again according to the determined sequence of the m audio segments and the m audio segments. The generating unit 123 is further configured to generate the target mashup audio again according to the determined m-1 transition audio information and the m audio segments.

[0259] Optionally, the processing apparatus 120 further comprises a rendering unit 124 configured to render the target mashup audio according to the third operation of the user.

[0260] As an example, in combination with Figure 3 , the rendering unit 124 can be configured to perform S104.

[0261] Optionally, the processing apparatus 120 further comprises an output unit 125 configured to output the target mashup audio.

[0262] The specific description of the above optional mode can be referred to the foregoing method embodiments, and will not be described herein again. In addition, the explanation and beneficial effect of any one of the processing apparatus 120 provided above can be referred to the corresponding method embodiments described above, and will not be described herein again.

[0263] As an example, in combination with Figure 1 , the acquiring unit 121 and the output unit 125 in the processing apparatus 120 can be implemented by the touch screen 150 and the processor 110 in Figure 1 . The determining unit 122, the generating unit 123 and the rendering unit 124 can be implemented by the processor 110 in Figure 1 to execute the program code in the internal storage 120 in Figure 1 .

[0264] Figure 13 A structure diagram of a signal bearing medium for carrying a computer program product is shown, which is used to store a computer program product or a computer program for executing a computer process on a computing device.

[0265] As shown in Figure 13 , the signal bearing medium 130 can include one or more program instructions, which when run by one or more processors can provide the functions or part of the functions described above for Figure 3 . Therefore, for example, one or more features of S101-S104 in Figure 3 may be undertaken by one or more instructions associated with the signal bearing medium 130. In addition, the program instructions in Figure 13 also describe example instructions.

[0266] In some examples, the signal-bearing medium 130 can include a computer- readable medium 131, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital tape, memory, read-only memory (ROM), random access memory (RAM), etc.

[0267] In some embodiments, the signal-bearing medium 130 can include a computer- recordable medium 132, such as, but not limited to, a floppy disk, a floppy disk drive, a CD-ROM, a CD-R / RW, a DVD-ROM, a DVD-R / RW, etc.

[0268] In some embodiments, the signal-bearing medium 130 can include a communications medium 133, such as, but not limited to, a digital and / or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).

[0269] The signal-bearing medium 130 can be conveyed by a wireless form of the communications medium 133 (e.g., wireless communications medium conforming with the IEEE 1902.11 standard or other transmission protocol). The one or more program instructions can be, for example, computer-executable or logic-implementing instructions.

[0270] In some examples, such as for Figure 3 The described audio data processing apparatus can be configured to provide various operations, functions, or actions in response to the one or more program instructions of the computer-readable medium 131, the computer-recordable medium 132, and / or the communications medium 133.

[0271] It should be understood that the arrangements described herein are for purposes of example only. As such, those skilled in the art will appreciate that other arrangements and other elements (e.g., machines, interfaces, functions, orders, and

[0272] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer executes the instructions, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0273] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for processing audio data, characterized in that, include: Get m audio segments, where m is an integer greater than or equal to 2; Based on the m audio segments, m-1 transition audio information is determined, and the m-1 transition audio information is used to connect the m audio segments; wherein, the first transition audio information in the m-1 transition audio information is used to connect the first audio segment and the second audio segment that are ordered consecutively in the m audio segments, and the order is the medley order of the m audio segments; Based on the m audio segments and the m-1 transition audio information, generate the target medley audio; The step of determining m-1 transition audio information based on the m audio segments includes: Based on the first information of the first audio segment and the second information of the second audio segment, the first transition audio information is determined. The first transition audio information includes the instrument digital interface (MIDI) information of the first transition audio corresponding to the first transition audio information. The first information includes the MIDI information and audio feature information of the first audio segment, and the second information includes the MIDI information and audio feature information of the second audio segment. Furthermore, multiple audio tracks in the MIDI information of the first audio segment and multiple audio tracks in the MIDI information of the second audio segment are the same.

2. The method according to claim 1, characterized in that, The audio feature information includes at least one of the following: the position information of the main melody track of the audio segment, style tag, emotion tag, rhythm information, beat information, or key signature information.

3. The method according to claim 1 or 2, characterized in that, The step of determining the first transition audio information based on the first information of the first audio segment and the second information of the second audio segment includes: The first transition audio information is determined based on the first information of the first audio segment, the second information of the second audio segment, and a preset neural network model.

4. The method according to claim 3, characterized in that, If, in the target medley of audio, the first audio segment precedes the second audio segment, then: The first transition audio information is determined based on a feature vector used to characterize the first transition audio information. The feature vector of the first transition audio information is determined based on a first vector and a second vector. The first vector is a feature vector generated at the end of the first audio segment based on the first information, and the second vector is a feature vector generated at the beginning of the second audio segment based on the second information.

5. The method according to any one of claims 1-4, characterized in that, The acquisition of m audio segments includes: In response to the user's first action, determine k target audios, where 2≤k≤m, and k is an integer; Extract the m audio segments from the k target audios.

6. The method according to any one of claims 1-5, characterized in that, Before determining m-1 transition audio information based on the m audio segments, the method further includes: Determine the medley order of the m audio segments.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to the user's second operation, the medley order of the m audio segments is re-determined; Based on the re-determined medley order and the m audio segments, m-1 transition audio information segments are re-determined; Based on the newly determined m-1 transition audio information and the m audio segments, the target medley audio is regenerated.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: In response to a third user action, the target medley audio is rendered.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Output the target medley audio.

10. An audio data processing apparatus, characterized in that, include: The acquisition unit is used to acquire m audio segments, where m is an integer greater than or equal to 2; A determining unit is configured to determine m-1 transition audio information based on the m audio segments, wherein the m-1 transition audio information is used to connect the m audio segments; wherein, the first transition audio information in the m-1 transition audio information is used to connect the first audio segment and the second audio segment that are ordered consecutively in the m audio segments, wherein the order is the medley order of the m audio segments; a generating unit is configured to generate a target medley audio based on the m audio segments and the m-1 transition audio information; The determining unit is specifically configured to determine the first transition audio information based on the first information of the first audio segment and the second information of the second audio segment. The first transition audio information includes the instrument digital interface (MIDI) information of the first transition audio corresponding to the first transition audio information. The first information includes the MIDI information and audio feature information of the first audio segment, and the second information includes the MIDI information and audio feature information of the second audio segment. Furthermore, multiple audio tracks in the MIDI information of the first audio segment are the same as multiple audio tracks in the MIDI information of the second audio segment.

11. The apparatus according to claim 10, characterized in that, The audio feature information includes at least one of the following: the position information of the main melody track of the audio segment, style tag, emotion tag, rhythm information, beat information, or key signature information.

12. The apparatus according to claim 10 or 11, characterized in that, The determining unit is specifically used to determine the first transition audio information based on the first information of the first audio segment, the second information of the second audio segment, and a preset neural network model.

13. The apparatus according to claim 12, characterized in that, If, in the target medley of audio, the first audio segment precedes the second audio segment, then: The first transition audio information is determined based on a feature vector used to characterize the first transition audio information. The feature vector of the first transition audio information is determined based on a first vector and a second vector. The first vector is a feature vector generated at the end of the first audio segment based on the first information, and the second vector is a feature vector generated at the beginning of the second audio segment based on the second information.

14. The apparatus according to any one of claims 10-13, characterized in that, The determining unit is further configured to, in response to the user's first operation, determine k target audios, where 2≤k≤m, and k is an integer; The acquisition unit is specifically used to extract the m audio segments from the k target audios.

15. The apparatus according to any one of claims 10-14, characterized in that, The determining unit is further configured to determine the medley order of the m audio segments before determining m-1 transition audio information based on the m audio segments.

16. The apparatus according to any one of claims 10-15, characterized in that, The determining unit is further configured to, in response to a second operation by the user, redetermine the medley order of the m audio segments; and further configured to, based on the redetermined medley order and the m audio segments, redetermine m-1 transition audio information. The generation unit is further configured to regenerate the target medley audio based on the newly determined m-1 transition audio information and the m audio segments.

17. The apparatus according to any one of claims 10-16, characterized in that, The device further includes: A rendering unit is used to render the target medley audio in response to a third user action.

18. The apparatus according to any one of claims 10-17, characterized in that, The device further includes: The output unit is used to output the target cascaded audio.

19. An audio data processing apparatus, characterized in that, include: One or more processors and a transmission interface, wherein the one or more processors receive or transmit data through the transmission interface, and the one or more processors are configured to invoke program instructions stored in memory to perform the method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes program instructions that, when executed on a computer or processor, cause the computer or processor to perform the method of any one of claims 1-9.

21. A computer program product, characterized in that, When the computer program product is run on an audio data processing device, it causes the device to perform the method as described in any one of claims 1-9.

Images