Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device based on artificial intelligence, equipment, a storage medium and a program product. The method comprises the following steps: acquiring first audio data and second audio data; performing emotion migration processing on the second audio data and the first audio data through a first generator and a second generator to obtain third audio data and fourth audio data; obtaining a first probability that the third audio data has a first emotion and a second probability that the fourth audio data has a second emotion; obtaining a first generation loss corresponding to the first probability and a second generation loss corresponding to the second probability, and obtaining a first content loss of the second audio data and the second recovery audio data and a second content loss of the first audio data and the first recovery audio data; the first generator and the second generator are updated based on the first generation loss, the second generation loss, the first content loss, and the second content loss. According to the invention, the emotion migration effect of the generator can be optimized.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an artificial intelligence-based data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] In the related technology, technology A is applied to the field of music editing, usually generating new music based on music rules or from random noise, but the generated new music can only meet the requirements of melody content. However, the music generated only based on the requirements of melody content will be very stiff and cannot meet the user's music editing needs. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium and computer program product based on artificial intelligence, which can use music emotion as a training basis so that the trained generator can achieve a stable and reliable music emotion transfer effect.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an artificial intelligence-based data processing method, including:

[0007] Acquire first audio data having a first emotion and second audio data having a second emotion;

[0008] The second audio data is processed by the first generator for emotion migration toward the first emotion to obtain third audio data, and the first audio data is processed by the second generator for emotion migration toward the second emotion to obtain fourth audio data; the third audio data is processed by the second generator for emotion migration toward the second emotion to obtain second restored audio data, and the fourth audio data is processed by the first generator for emotion migration toward the first emotion to obtain first restored audio data;

[0009] Obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator;

[0010] Acquire a first generated loss corresponding to the first emotion based on the first probability, acquire a second generated loss corresponding to the second emotion based on the second probability, and acquire a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data;

[0011] updating the first generator based on the first generation loss, the first content loss, and the second content loss, and updating the second generator based on the second generation loss, the first content loss, and the second content loss;

[0012] Among them, the updated first generator is used to perform emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion transfer processing corresponding to the second emotion on the original audio.

[0013] The present application provides an artificial intelligence-based data processing method, the method comprising:

[0014] Acquire an original audio file, and extract original audio data from the original audio file;

[0015] Performing an emotion migration process toward a first emotion on the original audio data by a first generator to obtain target audio data having the same content as the original audio data and having the first emotion;

[0016] Decoding the target audio data into a target audio file;

[0017] Among them, the first generator is trained by the data processing method provided in the embodiment of the present application.

[0018] The present application provides an artificial intelligence-based data processing method, the method comprising:

[0019] displaying at least one candidate emotion option and at least one candidate music;

[0020] In response to an adding operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music, synthesized music having the same content as the target music and having the target emotion is generated.

[0021] The present application provides an artificial intelligence-based data processing device, the device comprising:

[0022] A data module, used to obtain first audio data with a first emotion and second audio data with a second emotion;

[0023] A generation module, used for performing emotion migration processing on the second audio data towards the first emotion through a first generator to obtain third audio data, and performing emotion migration processing on the first audio data towards the second emotion through a second generator to obtain fourth audio data; performing emotion migration processing on the third audio data towards the second emotion through the second generator to obtain second restored audio data, and performing emotion migration processing on the fourth audio data towards the first emotion through the first generator to obtain first restored audio data;

[0024] a discriminator module, configured to obtain a first probability that the third audio data has the first emotion through a first discriminator, and to obtain a second probability that the fourth audio data has the second emotion through a second discriminator;

[0025] a loss module, configured to obtain a first generated loss corresponding to the first emotion based on the first probability, obtain a second generated loss corresponding to the second emotion based on the second probability, and obtain a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data;

[0026] An updating module, used for updating the first generator based on the first generation loss, the first content loss and the second content loss, and updating the second generator based on the second generation loss, the first content loss and the second content loss; wherein the updated first generator is used for performing emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used for performing emotion transfer processing corresponding to the second emotion on the original audio.

[0027] The present application provides an artificial intelligence-based data processing device, the device comprising:

[0028] An acquisition module, used for acquiring an original audio file and extracting original audio data from the original audio file;

[0029] A migration module, configured to perform an emotion migration process on the original audio data toward a first emotion through a first generator, so as to obtain target audio data having the same content as the original audio data and having the first emotion;

[0030] A decoding module is used to decode the target audio data into a target audio file; wherein the first generator is trained by the data processing method provided in an embodiment of the present application.

[0031] The present application provides an artificial intelligence-based data processing device, the device comprising:

[0032] A display module, used for displaying at least one candidate emotion option and at least one candidate music;

[0033] A synthesis module is used to generate synthesized music with the same content as the target music and having the target emotion in response to an adding operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music.

[0034] An embodiment of the present application provides an electronic device, including:

[0035] A memory for storing computer executable instructions;

[0036] The processor is used to implement the artificial intelligence-based data processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0037] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the artificial intelligence-based data processing method provided in the embodiment of the present application.

[0038] An embodiment of the present application provides a computer program product, including computer executable instructions, which, when executed by a processor, implement the artificial intelligence-based data processing method provided in the embodiment of the present application.

[0039] The embodiments of the present application have the following beneficial effects:

[0040] The first audio data and the second audio data of two emotions are obtained, and then the two audio data are respectively transferred to each other's emotions through two generators, so as to obtain the third audio data and the fourth audio data with interchangeable emotions, and the first probability and the second probability for evaluating the emotion transfer effect of the third audio data and the fourth audio data can be generated through the emotion discriminators corresponding to the two emotions respectively, and then the generation loss is obtained based on the first probability and the second probability, and the generator is trained in combination with the content loss between the real audio data and the generated audio data, so that the two generators have effective transfer capabilities for the two musical emotions respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of the structure of an artificial intelligence-based data processing system provided in an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 3A-3D It is a flowchart of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0044] Figure 4 It is a flowchart of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0045] Figure 5 It is a flowchart of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0046] Figure 6A-6C It is a schematic diagram of an interface of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0047] Figure 7 It is a training schematic diagram of the data processing method based on artificial intelligence provided in an embodiment of the present application;

[0048] Figure 8 It is a schematic diagram of a generator of an artificial intelligence-based data processing method provided in an embodiment of the present application;

[0049] Fig. 9 is a schematic diagram of a discriminator of an artificial intelligence-based data processing method provided in an embodiment of the present application;

[0050] Fig.10 It is a flowchart of a data processing method based on artificial intelligence provided in an embodiment of the present application;

[0051] Fig.11 It is a schematic diagram of audio file conversion of the artificial intelligence-based data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Transformer: The first deep learning model that completely relies on the self-attention mechanism to calculate the representation of its input and output. The basic structure of Transformer consists of an encoder and a decoder.

[0058] 2) Transformer Encoder: Transformer encoder is responsible for encoding the input sequence. Its basic structure consists of multi-head self-attention, residual block, and multi-layer perceptron.

[0059] 3) Musical Instrument Digital Interface (MIDI): Transmits notes, control parameters and other instructions, and controls MIDI devices through instructions (such as which note to play, how loud, etc.). MIDI is the most widely used music standard format in the music arrangement industry.

[0060] 4) Generative Adversarial Networks (GAN): It is an unsupervised deep learning model. Its basic structure consists of a generator and a discriminator.

[0061] 5) Multi-layer Perceptron (MLP): The layers of the multi-layer perceptron are fully connected. The bottom layer is the input layer, the middle layer is the hidden layer, and the last layer is the output layer.

[0062] 7) Classification Segment (CLS): The classification segment is a semantic feature vector that can represent the entire text.

[0063] The related art mainly edits the original music based on music rules to generate new music. For example, the related art needs to manually set the generation rules and generate new music based on the generation rules. When implementing the embodiments of the present application, the applicant found that the related art did not consider synthesizing emotional factors into the old music to generate new music during the process of generating music, resulting in the newly generated music having the same emotion as the old music, or the newly generated music having inappropriate emotion, thereby failing to meet the user's music editing needs.

[0064] In the embodiment of the present application, two pieces of music with different emotions are input, and the emotions contained in the music are interchanged using a deep learning model. Two pieces of music with different emotions can be generated at the same time. At the same time, the generated music is similar to the input music in melody and rhythm, and the music generation result of the entire scheme is predictable.

[0065] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium and computer program product based on artificial intelligence, which can use music emotion as a training basis so that the trained generator can achieve a stable and reliable music emotion transfer effect.

[0066] See also Figure 1 , Figure 1 It is a structural diagram of an artificial intelligence-based data processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0067] In some embodiments, the artificial intelligence-based data processing method provided in the embodiments of the present application can be implemented collaboratively by a terminal and a server, the terminal 400 receives a training request, and sends the training request to the server 200, and the server 200 performs the following processing based on the training request: obtaining first audio data having a first emotion and second audio data having a second emotion; performing emotion migration processing on the second audio data toward the first emotion through a first generator to obtain third audio data, and performing emotion migration processing on the first audio data toward the second emotion through a second generator to obtain fourth audio data; performing emotion migration processing on the third audio data toward the second emotion through the second generator to obtain second restored audio data, and performing emotion migration processing on the fourth audio data toward the first emotion through the first generator to obtain first restored audio data; obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator; obtaining a first generation loss corresponding to the first emotion based on the first probability, and obtaining a second generation loss corresponding to the second emotion based on the second probability. The first generator is updated based on the first generation loss, the first content loss and the second content loss, and the second generator is updated based on the second generation loss, the first content loss and the second content loss; the updated first generator is used to perform emotion migration processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion migration processing corresponding to the second emotion on the original audio; the server 200 deploys the updated second generator and the first generator, the terminal 400 receives the music emotion migration request based on the first emotion, the terminal 400 sends the music emotion migration request and the original music to the server 200, the server 200 obtains the original audio file, and extracts the original audio data from the original audio file; the original audio data is processed towards the first emotion by the first generator, and the target audio data having the same content as the original audio data and the first emotion is obtained; the target audio data is decoded into a target audio file, and the target audio file is sent to the terminal 400 to play the target music.

[0068] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0069] In some embodiments, the terminal or server can implement the artificial intelligence-based data processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a music APP or a live broadcast APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0070] See also Figure 2 , Figure 2 It is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device may be a terminal or a server. In the embodiment of the present application, the electronic device is taken as an example for description. Figure 2 The server shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0071] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0073] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0074] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0076] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0077] A network communication module 252, used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB), etc.;

[0078] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripherals and displaying content and information);

[0079] The input processing module 254 is used to detect one or more user inputs or interactions from one of the one or more input devices 232 and translate the detected inputs or interactions.

[0080] In some embodiments, the artificial intelligence-based data processing device provided in the embodiments of the present application can be implemented in software. Figure 2 The data processing device 255-1 based on artificial intelligence stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data module 2551, a generation module 2552, a discrimination module 2553, a loss module 2554 and an update module 2555, Figure 2 The data processing device 255-2 based on artificial intelligence stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 2556, migration module 2557, decoding module 2558, these modules are logical, so according to the functions implemented, they can be arbitrarily combined or further split. The functions of each module will be explained below.

[0081] The artificial intelligence-based data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0082] See also Figure 3A , Figure 3A is a flow chart of the data processing method based on artificial intelligence provided by the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0083] In step 101, first audio data having a first emotion and second audio data having a second emotion are acquired.

[0084] As an example, the first audio data and the second audio data are obtained by preprocessing the audio file. Since the audio file is in MIDI format, the audio file needs to be preprocessed to obtain the audio data. The data preprocessing process is described in detail below.

[0085] In some embodiments, see Figure 3B In step 101, the first audio data with the first emotion and the second audio data with the second emotion are obtained, which can be obtained by Figure 3B Steps 1011 to 1012 are implemented as shown.

[0086] In step 1011, a first audio file having a first emotion and a second audio file having a second emotion are obtained.

[0087] As an example, the first emotion here can be a happy emotion, the second emotion here can be a sad emotion, the first audio file here can be a music clip file with a happy emotion, and the second audio file here can be a music clip file with a sad emotion. In order to ensure the uniformity of the input of the generator, music clip files with the same time length can be obtained.

[0088] In step 1012, key information extraction is performed on the first audio file to obtain first audio data; key information extraction is performed on the second audio file to obtain second audio data.

[0089] In some embodiments, the key information includes pitch information and sound intensity information. In step 1012, the first audio file is subjected to key information extraction processing to obtain the first audio data, which can be achieved by the following technical scheme: extracting the pitch information and sound intensity information at each moment from the first audio file; performing the following processing for each moment: encoding the moment as the column number of a two-dimensional matrix, encoding the pitch information as the row number of the two-dimensional matrix, and encoding the sound intensity information as the numerical value of the position indicated by the column number and the row number in the two-dimensional matrix; using the two-dimensional matrix composed of the encoding results of multiple moments as the first audio data. Through the embodiment of the present application, the first audio file is serialized into a matrix, so that the model can learn the data, so that the deep learning model can be applied to the field of music editing.

[0090] As an example, the pitch information, tone intensity information, and tone duration recorded in the first audio file (an audio file in MIDI format) are represented in the form of a two-dimensional matrix, thereby realizing the serialization of the audio file. Fig.11 , each column of the matrix represents a unit time length of a performance, and each row of the matrix represents a pitch. For example, the value of the element in the i-th row and j-th column of the matrix is ​​m, and m>0, indicating that the note with pitch i at the j-th moment is played with a force of m.

[0091] In some embodiments, after the second audio data is processed by the first generator for emotion migration toward the first emotion to obtain the third audio data, the following processing is performed for each position in the two-dimensional matrix representing the third audio data: the column number, row number and value of the position are obtained; the column number of the position is decoded into the moment of the corresponding position, the row number of the position is decoded into the pitch information of the moment of the corresponding position, and the value of the position is decoded into the sound intensity information of the moment of the corresponding position; the following processing is performed for the moment corresponding to each position: the pitch information and sound intensity information of the moment are used as the key information of the moment; the key information of the moments corresponding to multiple positions are synthesized into a third audio file. The embodiment of the present application can convert the third audio data into a playable third audio file, which is convenient for storage and playback.

[0092] As an example, the third audio data is represented by the data form of a two-dimensional matrix. In order to obtain the third audio file that can be played, it is necessary to perform the reverse processing of data preprocessing, that is, convert the two-dimensional matrix into a third audio file (an audio file in MIDI format). Each position of the two-dimensional matrix is ​​decoded, and the column number (column number corresponds to time information), row number (corresponding to pitch information) and value of the position (corresponding to sound intensity information) of the position are obtained. The column number of the position is decoded into the time of the corresponding position, the row number of the position is decoded into the pitch information of the time of the corresponding position, and the value of the position is decoded into the sound intensity information of the time of the corresponding position. For example, position A in the two-dimensional matrix is ​​in the i-th row and j-th column, and the value of position A is m. After decoding, the key information of the j-th moment can be obtained: the note with a pitch of i is played with a force of m (sound intensity information), and finally the key information of the moments corresponding to multiple positions are synthesized into the third audio file.

[0093] In some embodiments, the key information includes pitch information and sound intensity information. In step 1012, the second audio file is subjected to key information extraction processing to obtain the second audio data, which can be achieved by the following technical scheme: extracting the pitch information and sound intensity information at each moment from the second audio file; performing the following processing for each moment: encoding the moment as the column number of a two-dimensional matrix, encoding the pitch information as the row number of the two-dimensional matrix, and encoding the sound intensity information as the numerical value of the position indicated by the column number and the row number in the two-dimensional matrix; using the two-dimensional matrix composed of the encoding results of multiple moments as the second audio data. By serializing the second audio file into a matrix through the embodiment of the present application, the model can learn the data, so that the deep learning model can be applied to the field of music editing.

[0094] As an example, the pitch information, tone intensity information, and tone duration recorded in the second audio file (an audio file in MIDI format) are represented in the form of a two-dimensional matrix, thereby realizing the serialization of the audio file. Fig.11 , each column of the matrix represents a unit time length of a performance, and each row of the matrix represents a pitch. For example, the value of the element in the i-th row and j-th column of the matrix is ​​m, and m>0, indicating that the note with pitch i at the j-th moment is played with a force of m.

[0095] In some embodiments, after the second generator performs emotion migration processing on the second audio data toward the first emotion to obtain the fourth audio data, the following processing is performed for each position in the two-dimensional matrix representing the fourth audio data: the column number, row number and value of the position are obtained; the column number of the position is decoded into the moment of the corresponding position, the row number of the position is decoded into the pitch information of the moment of the corresponding position, and the value of the position is decoded into the sound intensity information of the moment of the corresponding position; the following processing is performed for the moment corresponding to each position: the pitch information and sound intensity information of the moment are used as the key information of the moment; the key information of the moments corresponding to multiple positions are synthesized into a fourth audio file. Through the embodiment of the present application, the fourth audio data can be converted into a playable fourth audio file, which is convenient for storage and playback.

[0096] As an example, the fourth audio data is represented by the data form of a two-dimensional matrix. In order to obtain the fourth audio file that can be played, it is necessary to perform the reverse processing of the data preprocessing, that is, convert the two-dimensional matrix into the fourth audio file (audio file in MIDI format). Each position of the two-dimensional matrix is ​​decoded, and the column number (column number corresponds to time information), row number (corresponding to pitch information) and value of the position (corresponding to sound intensity information) of the position are obtained. The column number of the position is decoded into the time of the corresponding position, the row number of the position is decoded into the pitch information of the time of the corresponding position, and the value of the position is decoded into the sound intensity information of the time of the corresponding position. For example, position A in the two-dimensional matrix is ​​in the i-th row and j-th column, and the value of position A is m. After decoding, the key information of the j-th moment can be obtained: the note with a pitch of i is played with a force of m (sound intensity information), and finally the key information of the moments corresponding to multiple positions are synthesized into the fourth audio file.

[0097] In step 102, the second audio data is processed by the first generator for emotion transfer toward the first emotion to obtain third audio data, and the first audio data is processed by the second generator for emotion transfer toward the second emotion to obtain fourth audio data; the third audio data is processed by the second generator for emotion transfer toward the second emotion to obtain second restored audio data, and the fourth audio data is processed by the first generator for emotion transfer toward the first emotion to obtain first restored audio data.

[0098] As an example, the third audio data here is audio data generated based on the second audio data, the third audio data has different emotions compared to the second audio data, the emotion of the third audio data tends to or completely has the first emotion, and the second audio data is known audio data with the second emotion. The fourth audio data here is audio data generated based on the first audio data, the difference is that the fourth audio data has different emotions compared to the first audio data, the emotion of the fourth audio data tends to or completely has the second emotion, and the first audio data is known audio data with the first emotion. The second restored audio data here is audio data generated based on the third audio data, the second restored audio data has different emotions compared to the third audio data, the emotion of the third audio data tends to or completely has the first emotion, and the emotion of the second restored audio data tends to or completely has the second emotion. The first restored audio data here is audio data generated based on the fourth audio data, the difference is that the fourth audio data has different emotions compared to the first audio data, the emotion of the fourth audio data tends to or completely has the second emotion, and the first restored audio data tends to or completely has the first emotion.

[0099] As an example, the emotion migration toward the first emotion here refers to encoding the second audio data and other data processing so that the emotion represented by the processed data changes from the second emotion to the first emotion, thereby expecting to obtain the third audio data having the first emotion, and the emotion migration toward the second emotion here refers to encoding the first audio data and other data processing so that the emotion represented by the processed data changes from the first emotion to the second emotion, thereby expecting to obtain the fourth audio data having the second emotion.

[0100] As an example, see formula (1):

[0101] x ba =G a→b (x a ) (1);

[0102] Among them, x a is the first audio data with the first emotion, x ba is the fourth audio data having the second emotion, G a→b is the second generator.

[0103] As an example, see formula (2):

[0104] x ab =G b→a (x b ) (2);

[0105] Among them, x bis the second audio data with the second emotion, x ab is the third audio data having the first emotion, G b→a is the first generator.

[0106] The embodiment of the present application provides two generators, including a first generator and a second generator. The first generator has the ability to assign a first emotion to input audio data, and the second generator has the ability to assign a second emotion to input audio data, so that the emotion migration process toward the first emotion can be completed by the first generator, and the emotion migration process toward the second emotion can be completed by the second generator. The emotion migration process here does not change the melodic content of the audio data itself, but only completes the change of emotion. In addition, in the process of emotion migration processing, the music emotion data will not be input into the generator, but the generator itself has the migration ability for a certain emotion, and this migration ability is obtained through training.

[0107] The following is a detailed introduction to the forward propagation process of audio data in the generator and the reverse update process of parameters in the generator. Figure 3C Introduce the forward propagation process of audio data in the generator.

[0108] In some embodiments, the first generator includes a multi-layer perception network, a first attention encoding network, and N upsampling encoding networks, where N is a positive integer; see Figure 3C In step 102, the first generator performs emotion migration processing on the second audio data toward the first emotion to obtain third audio data, which can be obtained by Figure 3C The implementation of steps 1021 to 1023 is shown.

[0109] In step 1021, multi-layer perception processing corresponding to the first emotion is performed on the second audio data through a multi-layer perception network to obtain a multi-layer perception result.

[0110] In step 1022, the first attention coding network performs first attention coding processing corresponding to the first emotion on the multi-layer perception result to obtain a first attention coding result.

[0111] In step 1023, the first attention coding result is subjected to upsampling coding processing corresponding to the first emotion through N upsampling coding networks to obtain an upsampling coding result, and the upsampling result is used as the third audio data.

[0112] By combining upsampling coding with attention coding in the embodiment of the present application, the accuracy of coding representation is improved, thereby improving the final emotion transfer capability.

[0113] See also Figure 8, each generator includes a multi-layer perceptron (MLP), 6 upsampling modules, and 7 encoding modules. The pre-processed input audio data is processed by the multi-layer perceptron, then processed by the encoding module, and finally up-sampled. After repeated several times, the output audio data is obtained. The multi-layer perceptron network here is the multi-layer perceptron, and the multi-layer perceptron processing is multiple full-connection processing. The parameters involved in each full-connection processing are all parameters for the emotion transfer processing toward the first emotion. The first attention encoding network is a transformer encoder network, which includes a self-attention processing mechanism. The first attention encoding processing here is implemented by the transformer encoder network. The parameters involved in the transformer encoder network are all parameters for the emotion transfer processing toward the first emotion. The upsampling encoding network includes an upsampling network and a transformer encoder network. The parameters involved in the upsampling encoding network are also parameters for the emotion transfer processing toward the first emotion.

[0114] In some embodiments, the second generator includes a multi-layer perception network, a third attention coding network, and N upsampling coding networks, where N is a positive integer; in step 102, the first audio data is processed by the second generator for emotion migration toward the second emotion to obtain the fourth audio data, which can be achieved by the following technical scheme: the first audio data is processed by the multi-layer perception network for multi-layer perception corresponding to the second emotion to obtain the multi-layer perception result; the second sequence is processed by the third attention coding network for third attention coding corresponding to the second emotion to obtain the third attention coding result; the third attention coding result is processed by N upsampling coding networks for upsampling corresponding to the second emotion to obtain the upsampling coding result, and the upsampling result is used as the fourth audio data. The embodiment of the present application combines upsampling coding with attention coding to improve the accuracy of coding representation, thereby improving the final emotion migration capability.

[0115] See also Figure 8, each generator includes a multi-layer perceptron (MLP), 6 upsampling modules, and 7 encoding modules. The pre-processed input audio data is processed by the multi-layer perceptron, then processed by the encoding module, and finally up-sampled. After repeated several times, the output audio data is obtained. The multi-layer perceptron network here is the multi-layer perceptron, and the multi-layer perceptron processing is multiple full-connection processing. The parameters involved in each full-connection processing are all parameters used for the emotion transfer processing toward the second emotion. The third attention encoding network is a transformer encoder network, which includes a self-attention processing mechanism. The third attention encoding processing here is implemented through the transformer encoder network. The parameters involved in the transformer encoder network are all parameters used for the emotion transfer processing toward the second emotion. The upsampling encoding network includes an upsampling network and a transformer encoder network. The parameters involved in the upsampling encoding network are also parameters used for the emotion transfer processing toward the second emotion.

[0116] It can be seen from the above description that the network structures of the first generator and the second generator are the same, and the only difference is that the parameters of the network structure are different. The parameters of the first generator can help to realize the emotion transfer function toward the first emotion, and the parameters of the second generator can help to realize the emotion transfer function toward the second emotion. The reason for the different parameters is that the loss functions used in the update are different. The first generator and the second generator are updated separately based on their corresponding loss functions to realize the corresponding emotion transfer functions.

[0117] In addition to the above structure, the generator provided in the embodiment of the present application may also adopt a convolutional neural network (CNN) structure, a recurrent convolutional network (RNN) or a long short-term memory recursive network (LSTM), etc.

[0118] After the audio data is forward propagated in the generator, the loss used to implement the reverse update of the parameters in the generator is obtained through steps 103 and 104.

[0119] In step 103, a first probability that the third audio data has a first emotion is obtained by a first discriminator, and a second probability that the fourth audio data has a second emotion is obtained by a second discriminator.

[0120] As an example, the first discriminator here can be a pre-trained network with the ability to discriminate the first emotion, and the second discriminator here can be a pre-trained network with the ability to discriminate the second emotion. Therefore, it is not necessary to update the discriminator during the training of the generator, which can effectively improve the training efficiency. In addition, the discriminator can also be trained while training the generator. The solution of training the generator and the discriminator at the same time will be introduced in the subsequent embodiments.

[0121] In some embodiments, the first discriminator includes N downsampling encoding networks and a second attention encoding network, where N is a positive integer; see Figure 3D In step 103, the first probability that the third audio data has the first emotion is obtained by the first discriminator, which can be obtained by Figure 3D Steps 1031 to 1032 are implemented as shown.

[0122] In step 1031, downsampling encoding processing corresponding to the first emotion is performed on the third audio data through N downsampling encoding networks to obtain a downsampling encoding result.

[0123] In step 1032, the down-sampling encoding result and the classification mark constitute the input of the second attention encoding network, and the input is subjected to second attention encoding processing through the second attention encoding network to obtain the encoding value corresponding to the classification mark, and the encoding value is used as the first probability.

[0124] The embodiment of the present application combines downsampling coding with attention coding to improve the accuracy of coding representation, thereby improving the final emotion discrimination ability.

[0125] See also Fig. 9, each discriminator includes 6 downsampling modules and 7 encoding modules. First, the audio data output by the generator is input into the encoding module, and then the downsampling process is repeated for multiple times to obtain the encoding representation. The CLS symbol is added after the obtained encoding representation, and finally the encoding module is passed. The output result of the position corresponding to the CLS symbol is the discrimination result, and the discrimination result is true or false. The downsampling encoding network includes a transformer encoder network and a downsampling network. The parameters involved in the downsampling encoding network are parameters for the emotion migration process toward the first emotion. For the downsampling encoding result, it is necessary to add a classification identifier after the downsampling encoding result, that is, the [CLS] identifier in the transformer encoder network. The second attention encoding network is a transformer encoder network. The transformer encoder network includes a self-attention processing mechanism. The parameters involved in the transformer encoder network are all parameters for the emotion migration process toward the second emotion. Through the transformer encoder network, each position of the input data can be encoded based on the attention mechanism to obtain the encoding value corresponding to each position. In the embodiment of the present application, the encoding value at the position where the classification identifier is located will be used as the probability of the discriminator output.

[0126] In some embodiments, the second discriminator includes N downsampling coding networks and a fourth attention coding network; obtaining the second probability that the fourth audio data has the second emotion through the second discriminator in step 103 can be achieved through the following technical scheme: performing downsampling coding processing corresponding to the second emotion on the fourth audio data through N downsampling coding networks to obtain a downsampling coding result; the downsampling coding result and the classification mark constitute the input of the fourth attention coding network, and the fourth attention coding network performs the fourth attention coding processing on the input to obtain the coding value corresponding to the classification mark as the second probability. Through the embodiment of the present application, the downsampling coding is combined with the attention coding to improve the accuracy of the coding representation, thereby improving the final emotion discrimination ability.

[0127] See also Fig. 9, each discriminator includes 6 downsampling modules and 7 encoding modules. First, the audio data output by the generator is input into the encoding module, and then the downsampling process is repeated for multiple times to obtain the encoding representation. The CLS symbol is added after the obtained encoding representation, and finally the encoding module is passed. The output result of the position corresponding to the CLS symbol is the discrimination result, and the discrimination result is true or false. The downsampling encoding network includes a transformer encoder network and a downsampling network. The parameters involved in the downsampling encoding network are parameters for the emotion migration process toward the first emotion. For the downsampling encoding result, it is necessary to add a classification identifier after the downsampling encoding result, that is, the [CLS] identifier in the transformer encoder network. The fourth attention encoding network is a transformer encoder network. The transformer encoder network includes a self-attention processing mechanism. The parameters involved in the transformer encoder network are all parameters for the emotion migration process toward the second emotion. Through the transformer encoder network, each position of the input data can be encoded based on the attention mechanism to obtain the encoding value corresponding to each position. In the embodiment of the present application, the encoding value at the position where the classification identifier is located will be used as the probability of the discriminator output.

[0128] It can be seen from the above description that the network structures of the first discriminator and the second discriminator are the same, and the only difference is that the parameters of the network structure are different. The parameters of the first discriminator can help to distinguish whether the first emotion is included, and the parameters of the second generator can help to distinguish whether the second emotion is included. The reason for the different parameters is that the loss functions used in the update are different. The first discriminator and the second discriminator are updated separately based on their corresponding loss functions, so that the corresponding emotion discrimination function can be realized.

[0129] In step 104, a first generated loss corresponding to the first emotion is obtained based on the first probability, a second generated loss corresponding to the second emotion is obtained based on the second probability, and a first content loss of the second audio data and the third audio data and a second content loss of the first audio data and the fourth audio data are obtained.

[0130] In some embodiments, in step 104, obtaining a first generation loss corresponding to the first emotion based on the first probability and obtaining a second generation loss corresponding to the second emotion based on the second probability can be achieved through the following technical solutions: normalizing the difference between the first probability and the value one to obtain a first normal result, and using the first normal result as the first generation loss corresponding to the first emotion; normalizing the difference between the second probability and the value one to obtain a second normal result, and using the second normal result as the second generation loss corresponding to the second emotion. The generation loss can be constructed through the embodiments of the present application, thereby improving the accuracy of emotion transfer.

[0131] As an example, see formula (3):

[0132]

[0133] Among them, x a is the first audio data, G a→b (x a ) is the fourth audio data with the second emotion, D b (G a→b (x a )) is the second probability that the fourth audio data output by the second discriminator has the second emotion, It is the second generation loss.

[0134] As an example, see formula (4):

[0135]

[0136] Among them, x b is the second audio data with the second emotion, G b→a (x b ) is the third audio data with the first emotion, D a (G b→a (x b )) is the first probability that the third audio data output by the first discriminator has the first emotion, is the first generation loss.

[0137] In some embodiments, obtaining the first content loss of the second audio data and the third audio data and the second content loss of the first audio data and the fourth audio data in step 104 can be achieved by the following technical solution: normalizing the difference between the second audio data and the second restored audio data to obtain a third normal form result, and using the third normal form result as the first content loss; normalizing the difference between the first audio data and the first restored audio data to obtain a fourth normal form result, and using the fourth normal form result as the second content loss. Through the embodiments of the present application, content consistency loss can be constructed to ensure that content will not be lost during the process of emotion transfer, thereby improving the training effect of the generator.

[0138] As an example, see formula (5):

[0139] L ca =||x a -G b→a (x ba )||2 (5);

[0140] Among them, x a is the first audio data, x ba is the fourth audio data having the second emotion, G b→a (x ba ) is the first restored audio data obtained by performing emotion migration processing on the fourth audio data toward the first emotion, L ca It is the second content loss.

[0141] As an example, see formula (6):

[0142] L cb =||x b -G a→b (x ab )||2 (6);

[0143] Among them, x b is the second audio data, x ab is the third audio data having the first emotion, G a→b (x ab ) is the second restored audio data obtained by performing emotion migration processing on the third audio data toward the second emotion, L cb It is the first content loss.

[0144] Step 105 is used to implement the process of reverse updating of parameters in the generator.

[0145] In step 105 , the first generator is updated based on the first generation loss, the first content loss, and the second content loss, and the second generator is updated based on the second generation loss, the first content loss, and the second content loss.

[0146] As an example, the first generator is updated based on the first generation loss, the first content loss and the first fusion loss of the second content loss, that is, the parameter change value of the first generator that minimizes the first fusion loss is obtained through the gradient descent algorithm, and the parameters of the first generator are updated based on the parameter change value, and the second generator is updated based on the second generation loss, the first content loss and the second content loss. The second generator is updated based on the second fusion loss, that is, the parameter change value of the second generator that minimizes the second fusion loss is obtained through the gradient descent algorithm, and the parameters of the second generator are updated based on the parameter change value.

[0147] As an example, the updated first generator is used to perform emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion transfer processing corresponding to the second emotion on the original audio.

[0148] If the generator and the discriminator are trained at the same time, the scheme introduced in steps 101 to 105 can be regarded as the last round of iteration of the adversarial generative network training (that is, only the generator is updated). The iterative process before the last round of iteration will involve the simultaneous update of the generator and the discriminator. The following describes the process of obtaining the first generator, the second generator, the first discriminator and the second discriminator through the previous round of iteration.

[0149] In some embodiments, before obtaining the first probability that the third audio data has the first emotion through the first discriminator, the fifth audio data having the first emotion and the sixth audio data having the second emotion are obtained; the sixth audio data is subjected to emotion migration processing toward the first emotion through the historical first generator to obtain the seventh audio data; the fifth audio data is subjected to emotion migration processing toward the second emotion through the historical second generator to obtain the eighth audio data; the seventh audio data is subjected to emotion migration processing toward the second emotion through the historical second generator to obtain the sixth restored audio data, and the eighth audio data is subjected to emotion migration processing toward the first emotion through the historical first generator to obtain the fifth restored audio data; the third probability that the seventh audio data has the first emotion and the fourth probability that the fifth audio data has the first emotion are obtained through the historical first discriminator, and the fifth probability that the eighth audio data has the second emotion and the sixth probability that the sixth audio data has the second emotion are obtained through the historical second discriminator. ; Based on the third probability and the fourth probability, obtain the first discriminant loss corresponding to the first emotion, and based on the fifth probability and the sixth probability, obtain the second discriminant loss corresponding to the second emotion; update the historical first discriminant based on the first discriminant loss, and update the historical second discriminant based on the second discriminant loss; obtain the third generation loss corresponding to the first emotion based on the third probability, obtain the fourth generation loss corresponding to the second emotion based on the fifth probability, and obtain the third content loss of the sixth audio data and the sixth restored audio data and the fourth content loss of the fifth audio data and the fifth restored audio data; update the historical first generator based on the third generation loss, the third content loss and the fourth content loss, and update the historical second generator based on the fourth generation loss, the third content loss and the fourth content loss; use the updated historical first generator as the first generator, the updated historical second generator as the second generator, the updated historical first discriminator as the first discriminator, and the updated historical second discriminator as the second discriminator. Through the embodiments of the present application, the generator and the discriminator can be trained at the same time without pre-training the discriminator. Therefore, it is equivalent to using the same data for training, which makes the coupling ability of the discriminator and the generator stronger, that is, it has more targeted judgment ability, which can improve the judgment accuracy of the discriminator, thereby improving the training effect of the generator.

[0150] As an example, for the step of acquiring the fifth audio data having the first emotion and the sixth audio data having the second emotion, the fifth audio data and the sixth audio data are acquired in the same manner as the first audio data and the second audio data, or the fifth audio data and the first audio data here may be the same, and the sixth audio data and the second audio data here may be the same.

[0151] As an example, the emotion migration process of the sixth audio data toward the first emotion is performed by the historical first generator to obtain the seventh audio data; the emotion migration process of the fifth audio data toward the second emotion is performed by the historical second generator to obtain the eighth audio data, which is similar to step 102. The network structure of the historical first generator here is the same as the network structure of the first generator, and the network structure of the historical second generator here is the same as the network structure of the second generator, and the only difference is that the first generator is obtained by updating the historical first generator, and the second generator is obtained by updating the historical second generator.

[0152] As an example, the process of obtaining the third probability that the seventh audio data has the first emotion and the fourth probability that the fifth audio data has the first emotion through the historical first discriminator, and obtaining the fifth probability that the eighth audio data has the second emotion and the sixth probability that the sixth audio data has the second emotion through the historical second discriminator is similar to step 103. The network structure of the historical first discriminator here is the same as that of the first discriminator, and the network structure of the historical second discriminator here is the same as that of the second discriminator, and the only difference is that the first discriminator is obtained by updating the historical first discriminator, and the second discriminator is obtained by updating the historical second discriminator.

[0153] As an example, the process here of obtaining the third generation loss corresponding to the first emotion based on the third probability, obtaining the fourth generation loss corresponding to the second emotion based on the fifth probability, and obtaining the third content loss of the sixth audio data and the sixth restored audio data and the fourth content loss of the fifth audio data and the fifth restored audio data is similar to step 104, and reference may be made to the specific implementation of step 104.

[0154] As an example, the process of updating the first historical generator based on the third generation loss, the third content loss and the fourth content loss, and updating the second historical generator based on the fourth generation loss, the third content loss and the fourth content loss is similar to step 105, and the specific implementation of step 105 can be referred to.

[0155] In some embodiments, the above-mentioned acquisition of the first discriminant loss corresponding to the first emotion based on the third probability and the fourth probability, and the acquisition of the second discriminant loss corresponding to the second emotion based on the fifth probability and the sixth probability can be achieved through the following technical scheme: normalizing the difference between the fourth probability and the value one to obtain the fifth normal result; normalizing the third probability to obtain the sixth normal result; fusing the fifth normal result with the sixth normal result to obtain the first discriminant loss; normalizing the difference between the sixth probability and the value one to obtain the seventh normal result; normalizing the fifth probability to obtain the eighth normal result; fusing the seventh normal result with the eighth normal result to obtain the second discriminant loss. By constructing the loss of the discriminator through the embodiment of the present application, the discriminant ability of the discriminator can be interpretably enhanced, thereby improving the training effect of the generator.

[0156] As an example, see formula (7):

[0157]

[0158] Among them, x a is the fifth audio data, D a (x a )) is the fourth probability that the fifth audio data output by the historical first discriminator has the first emotion, x ab is the seventh audio data, D a (x ab )) is the third probability that the seventh audio data has the first emotion, is the first discriminant loss.

[0159] As an example, see formula (8):

[0160]

[0161] Among them, x b is the sixth audio data, D b (x b )) is the sixth probability that the sixth audio data output by the historical second discriminator has the second emotion, (x ba )) is the eighth audio data, D b (x ba )) is the fifth probability that the eighth audio data output by the historical second discriminator has the second emotion, is the second discriminant loss.

[0162] In some embodiments, see Figure 4 , Figure 4 is a flow chart of the data processing method based on artificial intelligence provided by the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0163] In step 201, an original audio file is obtained, and original audio data is extracted from the original audio file.

[0164] As an example, the original audio file here is an audio file in MIDI format. The implementation method of extracting original audio data from the original audio file can refer to the implementation method of obtaining the first audio data or obtaining the second audio data in step 101.

[0165] In step 202, the original audio data is processed by a first generator for emotion transfer toward a first emotion, thereby obtaining target audio data having the same content as the original audio data and having the first emotion. The first generator in step 202 is trained by the data processing method provided in an embodiment of the present application.

[0166] As an example, the implementation of step 202 here can refer to the implementation of step 102, the only difference is that the generator used is a trained generator.

[0167] In step 203, the target audio data is decoded into a target audio file.

[0168] As an example, the implementation method of decoding the target audio data into the target audio file here can refer to the implementation method of converting the third audio data into a playable third audio file.

[0169] Through the embodiments of the present application, the generator of the corresponding emotion can be flexibly called, so as to realize the emotion transfer quickly and easily.

[0170] In some embodiments, see Figure 5 , Figure 5 is a flow chart of the data processing method based on artificial intelligence provided by the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained.

[0171] In step 301, at least one candidate emotion option and at least one candidate music are displayed.

[0172] In step 302, in response to an operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music, synthesized music having the same content as the target music and having the target emotion is generated.

[0173] As an example, in response to the user's selection operation, the target music is selected in the "music library", and in response to the user's selection operation, the target emotion is selected in the "emotion library", and finally, by triggering the synthesis operation, the synthesized music can be generated. The synthesized music has the same content as the target music, but has a target emotion, where the target emotion can be the first emotion or the second emotion. First, the target music file corresponding to the target music is obtained, and then the target music file is converted into target music data, and then the generator corresponding to the target emotion is called to perform emotion migration processing on the target music data toward the corresponding target emotion to obtain synthesized music data, and then the synthesized music data is decoded into a synthesized music file for storage or playback.

[0174] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0175] In some embodiments, the artificial intelligence-based data processing method provided in the embodiments of the present application can be implemented collaboratively by a terminal and a server. The terminal receives a training request and sends the training request to the server. The server performs the following processing based on the training request: obtaining first audio data having a first emotion and second audio data having a second emotion; performing emotion migration processing on the second audio data toward the first emotion through a first generator to obtain third audio data, and performing emotion migration processing on the first audio data toward the second emotion through a second generator to obtain fourth audio data; obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator; obtaining a first generation loss corresponding to the first emotion based on the first probability, obtaining a second generation loss corresponding to the second emotion based on the second probability, and obtaining a first content loss between the second audio data and the third audio data and a first content loss between the first audio data and the fourth audio data second content loss; updating the first generator based on the first generation loss, the first content loss and the second content loss, and updating the second generator based on the second generation loss, the first content loss and the second content loss; the updated first generator is used to perform emotion migration processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion migration processing corresponding to the second emotion on the original audio; the server deploys the updated second generator and the first generator, the terminal receives a music emotion migration request based on the first emotion, the terminal sends the music emotion migration request and the original music to the server, the server obtains the original audio file, and extracts the original audio data from the original audio file; the original audio data is subjected to emotion migration processing towards the first emotion by the first generator, and target audio data having the same content as the original audio data and the first emotion is obtained; the target audio data is decoded into a target audio file, and the target audio file is sent to the terminal to play the target music.

[0176] In some embodiments, the function of music emotion transfer can be added to the music app, and users can specify the source music and target emotion to generate creative new music. It is suitable for users who need a personalized music experience, especially those who want to express their emotions through music. This product can also be used in the field of music creation to provide creators with creative inspiration and music materials. Generate new music based on the content of music A and the emotion of music B, see Fig. 6A , users need to select source music in the "Music Library", select target emotion in the "Emotion Library", adjust various parameters as needed, and finally click the "Generate" button to generate new music. Users can listen to the new music, save it locally or share it with friends.

[0177] In some embodiments, the present application embodiment can also be applied to page advertisement soundtracks. When entering an APP, the page advertisement is usually silent. The present application embodiment can be used to generate background music that matches the advertisement for different advertisement pages with specified emotions. Generate new music based on the content of music A and the emotion of music B, see Figure 6B , select the source music and target emotion to generate new music, and bind the corresponding advertisement. After adding background music to advertisement A, the corresponding background music will be played when advertisement A is played on the page.

[0178] In some embodiments, the embodiments of the present application can also be applied to the short video background music library to generate new music based on the content of music A and the emotion of music B. Figure 6C , start the short video APP, and in the process of making a short video, enter the background music function interface by triggering the "Add background music" control. Select the source music in the "Music Library" of this interface, select the target emotion in the "Emotion Library", and then adjust the various parameters as needed. Finally, click the "Generate" button to generate new music as the background music for the short video. After generating the music, you can click "Save" to store it in your personal cloud for future use.

[0179] The embodiment of the present application adopts a recurrent generative adversarial network as a training framework, uses Transformer as the basic unit structure for extracting and generating music data features, and proposes a training method for a generator model for performing music emotion transfer processing.

[0180] First, obtain a music dataset and preprocess it. Obtain music files of the Musical Instrument Digital Interface (MIDI) type with different emotions as a dataset. Preprocess each music file in the dataset. The preprocessing process is as follows: Represent the music in the Musical Instrument Digital Interface file in the form of pitch, volume, and duration. For example, a music file contains 88 NOTE_ON events, 88 NOTE_OFF events, 100 TIME_SHIFT events, and 32 VELOCITY events. NOTE_ON indicates the start of a note. When the synthesizer NOTE_ON, it starts playing the note with the corresponding pitch and volume. When the synthesizer NOTE_OFF, the synthesizer terminates the note. VELOCITY indicates the strength of the note when it is played. Taking the piano as an example, the loudness of the note depends on the strength or speed of the piano player hitting the keys. The TIME_SHIFT event moves the current time forward by a corresponding number of quantized time steps. Play the music files in the dataset and annotate the emotions contained in the music: happy and sad. According to the annotation information, the music in the data set is divided into two categories. The embodiment of the present application intends to realize the automatic generation of music with different emotions by converting these two types of music data. During training, it is necessary to ensure the diversity of the training data. At the same time, the model requires that the length of the input data is fixed. Therefore, the original music file is divided into equal time segments to obtain several music clips. For example, if 20 seconds is the fixed length of the input music, then a music file of 1 minute and 5 seconds takes 0-20 seconds, 20-40 seconds, and 40-60 seconds respectively, and the last 5 seconds are ignored, generating a total of 3 music clips.

[0181] Next, we build a training framework for the emotional music generation network. Figure 7 , the training framework includes 2 generators and 2 discriminators, see Figure 8 Each generator includes a multi-layer perceptron (MLP), 6 upsampling modules, and 7 encoding modules. The pre-processed input music data is processed by the multi-layer perceptron, then processed by the encoding module, and finally processed by upsampling. After repeated several times, the output music data is obtained. Fig. 9 Each discriminator contains 6 downsampling modules and 7 encoding modules. First, the music data output by the generator is input into the encoding module, and then it is downsampled and repeated many times to obtain the encoded representation. The CLS symbol is added to the obtained encoded representation, and finally it passes through the encoding module. The output result at the position corresponding to the CLS symbol is the discrimination result, which is true or false.

[0182] Next, the training framework of the emotional music generation network is trained. The two emotional training data sets are input into the network for training. The Adam optimizer is used to optimize the parameters of the generator and the discriminator. The loss function can use the least squares loss and the cycle consistency loss. In the model training stage, in order to learn the optimal parameters of the model and verify the effect of the training model, the data set is divided into a training set and a test set, and the ratio of the training set to the test set is 8:2. Specifically, Figure 7 The real music 1 with emotion a is recorded as x a , through the generator G a→b Generate music 1 with emotion b, denoted as x ba Similarly, the real music 2 with emotion b is denoted by x b , through the generator G b→a Generate music 2 with emotion a, denoted as x ab The above process can be seen in formula (9) and formula (10):

[0183] x ba =G a→b (x a ) (9);

[0184] x ab =G b→a (x b ) (10);

[0185] Among them, x a is the first audio data with the first emotion (real music 1), x ba is the fourth audio data with the second emotion (generated music 1), G a→b is the second generator. x b is the second audio data (real music 2) having the second emotion, x ab is the third audio data with the first emotion (generated music 2), G b→a is the first generator.

[0186] The discriminator that determines whether music contains emotion a (regardless of real music or generated music) is denoted as D a , the discriminator that determines whether the music contains emotion b (regardless of real music or generated music) is denoted as D b .

[0187] The embodiment of the present application trains the generator to minimize the following loss function. At this time, the discriminator outputs the probability that the generated music contains the converted emotion, see formula (11) and formula (12):

[0188]

[0189]

[0190] Among them, x a is the first audio data, G a→b (x a ) is the fourth audio data with the second emotion, D b (G a→b (x a )) is the second probability that the fourth audio data output by the second discriminator has the second emotion, is the second generation loss. x b is the second audio data with the second emotion, G b→a (x b ) is the third audio data with the first emotion, D a (G b→a (x b )) is the first probability that the third audio data output by the first discriminator has the first emotion, is the first generation loss.

[0191] In order to make the discriminator have stronger emotion differentiation ability, the discriminator is trained to minimize the following loss function, see formula (13) and formula (14):

[0192]

[0193]

[0194] Among them, x a is the fifth audio data, D a (x a )) is the fourth probability that the fifth audio data output by the historical first discriminator has the first emotion, x ab is the seventh audio data, D a (x ab )) is the third probability that the seventh audio data has the first emotion, is the first discriminant loss. x b is the sixth audio data, D b (x b )) is the sixth probability that the sixth audio data output by the historical second discriminator has the second emotion, (x ba )) is the eighth audio data, D b (x ba )) is the fifth probability that the eighth audio data output by the historical second discriminator has the second emotion, is the second discriminant loss.

[0195] In order to make the generated music similar to the original music in melody, the embodiment of the present application also uses consistency loss to train the generator. The cycle consistency loss is used to ensure that the song content is not lost, see formula (15):

[0196] L c =||x a -G b→a (x ba )||2+||x b -G a→b (x ab )||2 (15);

[0197] Among them, x a is the first audio data, x ba is the fourth audio data having the second emotion, G b→a (x ba ) is the first restored audio data obtained by performing emotion migration processing on the fourth audio data toward the first emotion, x b is the second audio data, x ab is the third audio data having the first emotion, G a→b (x ab ) is the second restored audio data obtained by performing emotion migration processing on the third audio data toward the second emotion.

[0198] The encoding module in the generator module involves self-attention processing. The process of self-attention processing is shown in formula (16):

[0199]

[0200] Among them, Attention(Q,K,V) is the output of the attention module, K T is the transposed matrix of the mapping result obtained by mapping the input of the attention module based on the key matrix parameters, Q is the mapping result obtained by mapping the input of the attention module based on the query matrix parameters, V is the mapping result obtained by mapping the input of the attention module based on the value matrix parameters, d k is the input data dimension. The self-attention processing here is implemented through the attention module in the transformer encoder.

[0201] The embodiment of the present application can enable music of two different emotions to achieve emotional transfer while retaining the content, so that the generation of emotional music no longer relies on a large amount of manually labeled music data, and no longer defines the emotional distribution function visually. Instead, it cleverly avoids this problem through two adversarial generative networks, making the results of emotional music generation more controllable.

[0202] In summary, the embodiment of the present application first obtains two MIDI format music data sets with different emotions, then reads the music files in MIDI format into data sequences, constructs a deep neural network structure, sets network parameters, inputs the two emotional training data sets into the generator and discriminator, and starts training. After the training is completed, the test data set is input into the generator to obtain the data sequence after emotion transfer, and finally the data sequence is decoded into emotional music.

[0203] Combine the following Fig.10 The scheme of the embodiment of the present application is described in detail.

[0204] In step 1001, two MIDI format music data sets with different emotions are obtained. This time, 400 piano pieces are used, including 200 piano pieces with emotion A and 200 piano pieces with emotion B.

[0205] In step 1002, the music file in MIDI format is read into a data sequence. Because MIDI carries the performance data and control data information of specific real-time notes, Python's pretty-midi and MusPy toolkits can be used to convert MIDI files into serialized note information. The specific process is to represent the pitch, intensity, and duration of the notes recorded in MIDI in the form of a two-dimensional matrix, see Fig.11 , each column of the matrix represents a unit time length of a performance, and each row of the matrix represents a pitch. For example, the value of the element in the i-th row and j-th column of the matrix is ​​m, and m>0, which means that the note with pitch i at the j-th moment is played with a force of m.

[0206] In step 1003, the test set and the training set are divided.

[0207] In step 1004, a deep neural network structure is constructed. The training framework includes two generators and two discriminators. a→b It can transform the real music 1 with emotion a into emotion b; the generator G b→a The real music 2 with emotion b is varied to have emotion a. When training the model, two pieces of music with different emotions are input at the same time. Through the recurrent adversarial network, the music variation with the other emotion is generated, and the discriminator is used to determine whether the music variation contains the target emotion, thereby optimizing the model parameters so that it can generate music variation that meets the target emotion.

[0208] In step 1005, the generator and the discriminator are trained, the batch size of the training is 64, the number of iterations is 100, and the network training is optimized using the Adam optimizer with a decay rate of 0.0002. After the network training is completed, the trained parameters are saved.

[0209] In step 1006, the trained network parameters are loaded, and the music data sequence that needs to be converted into emotions is input into the network to obtain a new music data sequence.

[0210] In step 1007, the obtained music sequence is converted into a MIDI file output using Python's pretty-midi and MusPy toolkits.

[0211] The embodiment of the present application can supplement the existing music material library by changing the existing music melody, rhythm, intensity, etc. In addition, the embodiment of the present application proposes a method for generating music with different emotions, which can better meet the actual needs of video soundtrack, music performance, etc.

[0212] In addition to Transformer, the basic structure of the generator provided in the embodiment of the present application can also adopt a convolutional neural network (CNN) structure, a recurrent convolutional network (RNN) or a long short-term memory recursive network (LSTM), etc.

[0213] The following is a description of an exemplary structure of a data processing device based on artificial intelligence provided by an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the artificial intelligence-based data processing device 255-1 of the memory 250 may include: a data module for obtaining first audio data having a first emotion and second audio data having a second emotion; a generation module for performing an emotion migration process on the second audio data toward the first emotion through a first generator to obtain third audio data, and performing an emotion migration process on the first audio data toward the second emotion through a second generator to obtain fourth audio data; a discrimination module for obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator; a loss module for obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator; and a loss module for obtaining a first probability that the third audio data has the first emotion through a second discriminator. A first generation loss corresponding to a first emotion is obtained based on a first probability, a second generation loss corresponding to a second emotion is obtained based on a second probability, and a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data is obtained; an updating module is used to update the first generator based on the first generation loss, the first content loss and the second content loss, and to update the second generator based on the second generation loss, the first content loss and the second content loss; wherein the updated first generator is used to perform emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion transfer processing corresponding to the second emotion on the original audio.

[0214] In some embodiments, the data module is also used to: obtain a first audio file with a first emotion and a second audio file with a second emotion; perform key information extraction processing on the first audio file to obtain first audio data; perform key information extraction processing on the second audio file to obtain second audio data.

[0215] In some embodiments, the key information includes pitch information and sound intensity information, and the data module is also used to: extract the pitch information and sound intensity information of each moment from the first audio file; perform the following processing for each moment: encode the moment as the column number of a two-dimensional matrix, encode the pitch information as the row number of the two-dimensional matrix, and encode the sound intensity information as the numerical value of the position indicated by the column number and the row number in the two-dimensional matrix; and use the two-dimensional matrix composed of the encoding results of multiple moments as the first audio data.

[0216] In some embodiments, after the second audio data is processed by the first generator for emotion migration toward the first emotion to obtain the third audio data, the data module is also used to: perform the following processing for each position in the two-dimensional matrix representing the third audio data: obtain the column number, row number and numerical value of the position; decode the column number of the position into the moment of the corresponding position, decode the row number of the position into the pitch information of the moment of the corresponding position, and decode the numerical value of the position into the sound intensity information of the moment of the corresponding position; perform the following processing for the moment corresponding to each position: use the pitch information and sound intensity information of the moment as the key information of the moment; synthesize the key information of the moments corresponding to multiple positions into a third audio file.

[0217] In some embodiments, the first generator includes a multi-layer perception network, a first attention coding network and N upsampling coding networks; the generation module is also used to: perform multi-layer perception processing corresponding to the first emotion on the second audio data through the multi-layer perception network to obtain a multi-layer perception result; perform first attention coding processing corresponding to the first emotion on the multi-layer perception result through the attention coding network to obtain an attention coding result; perform upsampling coding processing corresponding to the first emotion on the attention coding result through N upsampling coding networks to obtain an upsampling coding result, and use the upsampling result as the third audio data.

[0218] In some embodiments, the first discriminator includes N downsampling coding networks and a second attention coding network; the discriminant module is also used to: perform downsampling coding processing on the third audio data corresponding to the first emotion through the N downsampling coding networks to obtain a downsampling coding result; the downsampling coding result and the classification identifier constitute the input of the second attention coding network, and perform second attention coding processing on the input through the second attention coding network to obtain the coding value corresponding to the classification identifier, and use the coding value as the first probability.

[0219] In some embodiments, the loss module is also used to: normalize the difference between the first probability and the numerical value one to obtain a first normalized result, and use the first normalized result as a first generated loss corresponding to the first emotion; normalize the difference between the second probability and the numerical value one to obtain a second normalized result, and use the second normalized result as a second generated loss corresponding to the second emotion.

[0220] In some embodiments, the loss module is further used to: normalize the difference between the second audio data and the third audio data to obtain a third normal form result, and use the third normal form result as the first content loss; normalize the difference between the first audio data and the fourth audio data to obtain a fourth normal form result, and use the fourth normal form result as the second content loss.

[0221] In some embodiments, the discrimination module is also used to: obtain the fifth audio data having the first emotion and the sixth audio data having the second emotion before obtaining the first probability that the third audio data has the first emotion through the first discriminator; perform emotion migration processing on the sixth audio data toward the first emotion through the historical first generator to obtain the seventh audio data; perform emotion migration processing on the fifth audio data toward the second emotion through the historical second generator to obtain the eighth audio data; obtain the third probability that the seventh audio data has the first emotion and the fourth probability that the fifth audio data has the first emotion through the historical first discriminator, and obtain the fifth probability that the eighth audio data has the second emotion and the sixth probability that the sixth audio data has the second emotion through the historical second discriminator; obtain the first discrimination loss corresponding to the third probability and the fourth probability and the corresponding The fifth probability and the second discriminant loss of the sixth probability; updating the historical first discriminator based on the first discriminant loss, and updating the historical second discriminator based on the second discriminant loss; obtaining the third generation loss corresponding to the third probability and the fourth generation loss corresponding to the fifth probability, and obtaining the third content loss corresponding to the sixth audio data and the sixth restored audio data and the fourth content loss corresponding to the fifth audio data and the fifth restored audio data; updating the historical first generator based on the third generation loss, the third content loss and the fourth content loss, and updating the historical second generator based on the fourth generation loss, the third content loss and the fourth content loss; using the updated historical first generator as the first generator, the updated historical second generator as the second generator, the updated historical first discriminator as the first discriminator, and the updated historical second discriminator as the second discriminator.

[0222] In some embodiments, the discrimination module is further used to: normalize the difference between the fourth probability and the numerical value one to obtain a fifth normal form result; normalize the third probability to obtain a sixth normal form result; fuse the fifth normal form result with the sixth normal form result to obtain a first discrimination loss; normalize the difference between the sixth probability and the numerical value one to obtain a seventh normal form result; normalize the fifth probability to obtain an eighth normal form result; fuse the seventh normal form result with the eighth normal form result to obtain a second discrimination loss.

[0223] In some embodiments, Figure 2 As shown, the software modules stored in the artificial intelligence-based data processing device 255-2 of the memory 250 may include: an acquisition module, used to acquire the original audio file and extract the original audio data from the original audio file; a migration module, used to perform emotion migration processing on the original audio data toward the first emotion through a first generator, to obtain target audio data having the same content as the original audio data and having the first emotion; a decoding module, used to decode the target audio data into a target audio file; wherein the first generator is trained by the data processing method provided in an embodiment of the present application.

[0224] In some embodiments, the software modules stored in the artificial intelligence-based data processing device in the memory may include: a display module for displaying at least one candidate emotion option and at least one candidate music; a synthesis module for generating synthesized music having the same content as the target music and having the target emotion in response to an adding operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music.

[0225] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the data processing method based on artificial intelligence described in the embodiment of the present application.

[0226] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the artificial intelligence-based data processing method provided by the embodiment of the present application.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0228] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0229] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0230] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0231] To summarize, through the embodiments of the present application, the first audio data and the second audio data of two emotions are obtained, and then the two audio data are respectively migrated to each other's emotions through two generators, so as to obtain the third audio data and the fourth audio data with interchangeable emotions, and the emotion discriminators corresponding to the two emotions respectively can generate the first probability and the second probability for evaluating the emotion migration effect of the third audio data and the fourth audio data, and then the generation loss is obtained based on the first probability and the second probability, and the generator is trained in combination with the content loss between the real audio data and the generated audio data, so that the two generators have effective migration capabilities for the two musical emotions respectively.

[0232] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method based on artificial intelligence, characterized in that: The method comprises: Acquire first audio data having a first emotion and second audio data having a second emotion; Performing an emotion migration process toward the first emotion on the second audio data by a first generator to obtain third audio data, and performing an emotion migration process toward the second emotion on the first audio data by a second generator to obtain fourth audio data; Performing emotion migration processing toward the second emotion on the third audio data by the second generator to obtain second restored audio data, and performing emotion migration processing toward the first emotion on the fourth audio data by the first generator to obtain first restored audio data; Obtaining a first probability that the third audio data has the first emotion through a first discriminator, and obtaining a second probability that the fourth audio data has the second emotion through a second discriminator; Acquire a first generated loss corresponding to the first emotion based on the first probability, acquire a second generated loss corresponding to the second emotion based on the second probability, and acquire a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data; updating the first generator based on the first generation loss, the first content loss, and the second content loss, and updating the second generator based on the second generation loss, the first content loss, and the second content loss; Among them, the updated first generator is used to perform emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used to perform emotion transfer processing corresponding to the second emotion on the original audio.

2. The method according to claim 1, characterized in that The obtaining of first audio data with a first emotion and second audio data with a second emotion includes: Acquire a first audio file having the first emotion and a second audio file having the second emotion; Performing key information extraction processing on the first audio file to obtain the first audio data; Perform key information extraction processing on the second audio file to obtain the second audio data.

3. The method according to claim 2, characterized in that The key information includes pitch information and sound intensity information, and the step of extracting the key information from the first audio file to obtain the first audio data includes: Extracting pitch information and sound intensity information at each moment from the first audio file; The following processing is performed for each of the said moments: Encoding the time as a column number of a two-dimensional matrix, encoding the pitch information as a row number of the two-dimensional matrix, and encoding the tone intensity information as a value of a position indicated by the column number and the row number in the two-dimensional matrix; A two-dimensional matrix formed by encoding results at multiple moments is used as the first audio data.

4. The method according to claim 3, characterized in that After performing emotion migration processing toward the first emotion on the second audio data by the first generator to obtain third audio data, the method further includes: The following processing is performed for each position in the two-dimensional matrix representing the third audio data: Obtain the column number, row number and value of the position; Decoding the column number of the position into the time corresponding to the position, decoding the row number of the position into the pitch information of the time corresponding to the position, and decoding the value of the position into the sound intensity information of the time corresponding to the position; The following processing is performed for each moment corresponding to the position: the pitch information and the sound intensity information at the moment are used as the key information of the moment; The key information of the moments corresponding to the multiple positions is synthesized into a third audio file.

5. The method according to claim 1, characterized in that The first generator includes a multi-layer perception network, a first attention encoding network, and N upsampling encoding networks, where N is a positive integer; The step of performing an emotion migration process on the second audio data toward the first emotion by the first generator to obtain third audio data includes: Performing multi-layer perception processing corresponding to the first emotion on the second audio data through the multi-layer perception network to obtain a multi-layer perception result; Performing a first attention encoding process corresponding to the first emotion on the multi-layer perception result through the first attention encoding network to obtain a first attention encoding result; The first attention coding result is upsampled and encoded corresponding to the first emotion through the N upsampling coding networks to obtain an upsampling coding result, and the upsampling result is used as the third audio data.

6. The method according to claim 1, characterized in that The first discriminator includes N downsampling encoding networks and a second attention encoding network, where N is a positive integer; The obtaining, by a first discriminator, a first probability that the third audio data has the first emotion comprises: Performing downsampling encoding processing corresponding to the first emotion on the third audio data through the N downsampling encoding networks to obtain a downsampling encoding result; The down-sampling encoding result and the classification mark constitute the input of the second attention encoding network, and the second attention encoding network performs second attention encoding processing on the input to obtain the encoding value corresponding to the classification mark, and use the encoding value as the first probability.

7. The method according to claim 1, characterized in that The step of acquiring a first generated loss corresponding to the first emotion based on the first probability, and acquiring a second generated loss corresponding to the second emotion based on the second probability, comprises: Normalize the difference between the first probability and the value one to obtain a first normalized result, and use the first normalized result as a first generated loss corresponding to the first emotion; The difference between the second probability and the value one is normalized to obtain a second normalized result, and the second normalized result is used as the second generated loss corresponding to the second emotion.

8. The method according to claim 1, characterized in that The obtaining of a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data includes: Normalize the difference between the second audio data and the second restored audio data to obtain a third normalized result, and use the third normalized result as the first content loss; Normalization is performed on the difference between the first audio data and the first restored audio data to obtain a fourth normalization result, and the fourth normalization result is used as the second content loss.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire fifth audio data having the first emotion and sixth audio data having the second emotion; Performing an emotion migration process toward the first emotion on the sixth audio data through the first history generator to obtain seventh audio data, and performing an emotion migration process toward the second emotion on the fifth audio data through the second history generator to obtain eighth audio data; Performing an emotion migration process toward the second emotion on the seventh audio data through the second history generator to obtain sixth restored audio data, and performing an emotion migration process toward the first emotion on the eighth audio data through the first history generator to obtain fifth restored audio data; Obtaining, by means of a historical first discriminator, a third probability that the seventh audio data has the first emotion and a fourth probability that the fifth audio data has the first emotion, and obtaining, by means of a historical second discriminator, a fifth probability that the eighth audio data has the second emotion and a sixth probability that the sixth audio data has the second emotion; Based on the third probability and the fourth probability, obtain a first discrimination loss corresponding to the first emotion, and based on the fifth probability and the sixth probability, obtain a second discrimination loss corresponding to the second emotion; updating the historical first discriminator based on the first discriminant loss, and updating the historical second discriminator based on the second discriminant loss; Acquire a third generation loss corresponding to the first emotion based on the third probability, acquire a fourth generation loss corresponding to the second emotion based on the fifth probability, and acquire a third content loss between the sixth audio data and the sixth restored audio data and a fourth content loss between the fifth audio data and the fifth restored audio data; updating the historical first generator based on the third generation loss, the third content loss, and the fourth content loss, and updating the historical second generator based on the fourth generation loss, the third content loss, and the fourth content loss; The updated historical first generator is used as the first generator, the updated historical second generator is used as the second generator, the updated historical first discriminator is used as the first discriminator, and the updated historical second discriminator is used as the second discriminator.

10. The method according to claim 9, characterized in that The obtaining, based on the third probability and the fourth probability, a first discrimination loss corresponding to the first emotion, and the obtaining, based on the fifth probability and the sixth probability, a second discrimination loss corresponding to the second emotion, comprises: Normalizing the difference between the fourth probability and the value 1 to obtain a fifth normalized result; Normalizing the third probability to obtain a sixth normalized result; Fusing the fifth normal form result with the sixth normal form result to obtain the first discrimination loss; Normalizing the difference between the sixth probability and the value 1 to obtain a seventh normalized result; Normalizing the fifth probability to obtain an eighth normal form result; The seventh normal form result and the eighth normal form result are fused to obtain the second discrimination loss.

11. A data processing method based on artificial intelligence, characterized in that: The method comprises: Acquire an original audio file, and extract original audio data from the original audio file; Performing an emotion migration process toward a first emotion on the original audio data by a first generator to obtain target audio data having the same content as the original audio data and having the first emotion; Decoding the target audio data into a target audio file; Wherein, the first generator is trained by the data processing method described in any one of claims 1 to 10.

12. A data processing method based on artificial intelligence, characterized in that: The method comprises: displaying at least one candidate emotion option and at least one candidate music; In response to an adding operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music, synthesized music having the same content as the target music and having the target emotion is generated.

13. A data processing device based on artificial intelligence, characterized in that: The device comprises: A data module, used to obtain first audio data with a first emotion and second audio data with a second emotion; A generation module, used for performing emotion migration processing on the second audio data towards the first emotion through a first generator to obtain third audio data, and performing emotion migration processing on the first audio data towards the second emotion through a second generator to obtain fourth audio data; performing emotion migration processing on the third audio data towards the second emotion through the second generator to obtain second restored audio data, and performing emotion migration processing on the fourth audio data towards the first emotion through the first generator to obtain first restored audio data; a discriminator module, configured to obtain a first probability that the third audio data has the first emotion through a first discriminator, and to obtain a second probability that the fourth audio data has the second emotion through a second discriminator; a loss module, configured to obtain a first generated loss corresponding to the first emotion based on the first probability, obtain a second generated loss corresponding to the second emotion based on the second probability, and obtain a first content loss between the second audio data and the second restored audio data and a second content loss between the first audio data and the first restored audio data; An updating module, used for updating the first generator based on the first generation loss, the first content loss and the second content loss, and updating the second generator based on the second generation loss, the first content loss and the second content loss; wherein the updated first generator is used for performing emotion transfer processing corresponding to the first emotion on the original audio data, and the updated second generator is used for performing emotion transfer processing corresponding to the second emotion on the original audio.

14. A data processing device based on artificial intelligence, characterized in that: The device comprises: An acquisition module, used for acquiring an original audio file and extracting original audio data from the original audio file; A migration module, configured to perform an emotion migration process on the original audio data toward a first emotion through a first generator, so as to obtain target audio data having the same content as the original audio data and having the first emotion; A decoding module, used for decoding the target audio data into a target audio file; wherein the first generator is trained by the data processing method described in any one of claims 1 to 10.

15. A data processing device based on artificial intelligence, characterized in that: The device comprises: A display module, used for displaying at least one candidate emotion option and at least one candidate music; A synthesis module is used to generate synthesized music with the same content as the target music and having the target emotion in response to an adding operation of adding a target emotion corresponding to a target emotion option in the at least one candidate emotion option to a target music in the at least one candidate music.

16. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, used to implement the artificial intelligence-based data processing method described in any one of claims 1 to 10, claim 11 and claim 12 when executing the computer-executable instructions stored in the memory.

17. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the artificial intelligence-based data processing method described in any one of claims 1 to 10, claim 11 and claim 12 is implemented.

18. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the artificial intelligence-based data processing method described in any one of claims 1 to 10, claim 11 and claim 12 is implemented.

Images