Method, device and storage medium for training a neural network model

Abstract

The application provides a neural network model training method, device and storage medium, relates to the technical field of artificial intelligence, and the method comprises the following steps: performing model compression on a first neural network model after training to obtain a second neural network model, wherein the parameter quantity of the second neural network model is smaller than that of the first neural network model; inputting a first training sample into the second neural network model to obtain first prediction data, and calculating a first loss value corresponding to the first prediction data; inputting the first training sample into the first neural network model to obtain second prediction data, and calculating a second loss value corresponding to the second prediction data; and training the second neural network model based on the first loss value and the second loss value to obtain a second neural network model after training. The application can obtain a neural network model suitable for a terminal.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and storage medium for training a neural network model. Background Technology

[0002] Neural network models play a crucial role in artificial intelligence, and to achieve better reasoning performance, the number of parameters in neural network models is becoming increasingly large. Typically, large models can have hundreds of millions, billions, or even more parameters.

[0003] Servers possess abundant storage and computing resources, sufficient to deploy and run large models, while terminals have very limited storage and computing resources, potentially unable to run large models with numerous parameters. Therefore, obtaining a neural network model that can be deployed and run on a terminal becomes a pressing issue. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for training a neural network model, which can obtain a neural network model with a relatively small number of parameters suitable for a terminal. The technical solution is as follows:

[0005] Firstly, a method for training a neural network model is provided, the method comprising:

[0006] The trained first neural network model is compressed to obtain a second neural network model, wherein the number of parameters in the second neural network model is less than the number of parameters in the first neural network model.

[0007] Input the first training sample into the second neural network model to obtain the first prediction data, and calculate the first loss value corresponding to the first prediction data;

[0008] The first training sample is input into the first neural network model to obtain the second prediction data, and the second loss value corresponding to the second prediction data is calculated.

[0009] Based on the first loss value and the second loss value, the second neural network model is trained to obtain the trained second neural network model.

[0010] In one possible implementation, training the second neural network model based on the first loss value and the second loss value includes:

[0011] Multiply the first loss value by the first weight to obtain the first product;

[0012] Multiply the second loss value by the second weight to obtain the second product;

[0013] Add the first product and the second product to obtain the fusion loss value;

[0014] The second neural network model is trained based on the fusion loss value.

[0015] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

[0016] In one possible implementation, before inputting the first training sample into the second neural network model, the method further includes:

[0017] Obtain the actual master audio clips of the sample songs;

[0018] The actual master audio segment is downsampled to obtain the downsampled audio segment;

[0019] The downsampled audio segment is upsampled to obtain an upsampled audio segment, which is used as the first training sample. The sampling rate of the upsampled audio segment is the same as the sampling rate of the real master audio segment.

[0020] In one possible implementation, the duration of the actual master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0021] In one possible implementation, the first predicted data is a first predicted master audio segment, the second predicted data is a second predicted master audio segment, and the calculation of the first loss value corresponding to the first predicted data includes:

[0022] Based on the first predicted master audio segment and the real master audio segment, calculate the first time domain loss value and the first frequency domain loss value corresponding to the first predicted master audio segment;

[0023] The first time-domain loss value and the first frequency-domain loss value are added together to obtain the first loss value corresponding to the first predicted master audio segment;

[0024] The calculation of the second loss value corresponding to the second predicted data includes:

[0025] Based on the second predicted master audio segment and the real master audio segment, calculate the second time-domain loss value and the second frequency-domain loss value corresponding to the second predicted master audio segment;

[0026] The second time-domain loss value and the second frequency-domain loss value are added together to obtain the second loss value corresponding to the second predicted master audio segment.

[0027] In one possible implementation, the first neural network model is an autoencoder.

[0028] In one possible implementation, the trained second neural network model is deployed on a terminal, which, upon invocation of the terminal, generates the master audio of the second song based on the input audio to be processed of the second song.

[0029] Secondly, an apparatus for training a neural network model is provided, the apparatus comprising:

[0030] The compression module is used to compress the trained first neural network model to obtain a second neural network model, wherein the number of parameters in the second neural network model is less than the number of parameters in the first neural network model.

[0031] The training module is used to input the first training sample into the second neural network model to obtain the first predicted data and calculate the first loss value corresponding to the first predicted data; input the first training sample into the first neural network model to obtain the second predicted data and calculate the second loss value corresponding to the second predicted data; and train the second neural network model based on the first loss value and the second loss value to obtain the trained second neural network model.

[0032] In one possible implementation, the training module is used for:

[0033] Multiply the first loss value by the first weight to obtain the first product;

[0034] Multiply the second loss value by the second weight to obtain the second product;

[0035] Add the first product and the second product to obtain the fusion loss value;

[0036] The second neural network model is trained based on the fusion loss value.

[0037] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

[0038] In one possible implementation, the training module is further configured to:

[0039] Obtain the actual master audio clips of the sample songs;

[0040] The actual master audio segment is downsampled to obtain the downsampled audio segment;

[0041] The downsampled audio segment is upsampled to obtain an upsampled audio segment, which is used as the first training sample. The sampling rate of the upsampled audio segment is the same as the sampling rate of the real master audio segment.

[0042] In one possible implementation, the duration of the actual master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0043] In one possible implementation, the first predicted data is a first predicted master audio segment, the second predicted data is a second predicted master audio segment, and the training module is used to:

[0044] Based on the first predicted master audio segment and the real master audio segment, calculate the first time domain loss value and the first frequency domain loss value corresponding to the first predicted master audio segment;

[0045] The first time-domain loss value and the first frequency-domain loss value are added together to obtain the first loss value corresponding to the first predicted master audio segment;

[0046] Based on the second predicted master audio segment and the real master audio segment, calculate the second time-domain loss value and the second frequency-domain loss value corresponding to the second predicted master audio segment;

[0047] The second time-domain loss value and the second frequency-domain loss value are added together to obtain the second loss value corresponding to the second predicted master audio segment.

[0048] In one possible implementation, the first neural network model is an autoencoder.

[0049] In one possible implementation, the trained second neural network model is deployed on a terminal, which, upon invocation of the terminal, generates the master audio of the second song based on the input audio to be processed of the second song.

[0050] Thirdly, a terminal is provided, the terminal including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed as described in the first aspect and any possible method of training the neural network model described in the first aspect.

[0051] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction that is loaded and executed by a processor to perform the operations performed by the method of training the neural network model described in the first aspect and any possible implementation of the first aspect.

[0052] Fifthly, a computer program product is provided, the computer program product storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed as described in the first aspect and any possible method of training the neural network model described in the first aspect.

[0053] The beneficial effects of the technical solutions provided in this application are:

[0054] The technical solution provided in this application compresses a first neural network model that has already been trained to obtain a second neural network model with fewer parameters. Furthermore, to improve the inference performance of the compressed second neural network model, this solution uses the same training samples and the loss values ​​from both the first and second neural network models to train the second neural network model together, making the inference performance of the trained second neural network model closer to that of the first neural network model. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of a real-time scenario provided in an embodiment of this application;

[0057] Figure 2 This is a flowchart of a method for training a neural network model provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of a neural network model training method provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of a neural network model training method provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of a device structure for training a neural network model provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0064] This application provides a method for training a neural network model. This method can be implemented by a computing device, such as a server or server cluster. See also Figure 1 This illustration shows a possible implementation scenario of an embodiment of this application. For a large model with a large number of parameters, a computing device trains the large model. After training, the trained large model can be deployed on the computing device to perform a target inference task, such as music master audio generation, speech recognition, audio feature extraction, image generation, etc. When there is a need for the terminal to perform the target inference task locally, considering the large number of parameters of the large model, it cannot be deployed on the terminal. The large model can be compressed to obtain a smaller model with fewer parameters, but still capable of performing the target inference task. The smaller model can then be deployed on the terminal. However, if the smaller model is directly deployed on the terminal, although it can still perform the target inference task, the inference effect may be poor. In this case, the neural network model training method provided in this application embodiment can be used. After compressing the trained large model to obtain a smaller model, the smaller model is then trained. Furthermore, during the training of the smaller model, the large model guides the smaller model; that is, the smaller model is trained jointly based on the loss values ​​of the large model and the small model, making the inference effect of the smaller model closer to that of the large model. Once the small model has been trained, it can be deployed on the terminal.

[0065] The following describes a method for training a neural network model according to an embodiment of this application. This method can be implemented using a computing device. See [link to documentation]. Figure 2 The processing flow of this method may include the following steps:

[0066] Step 201: Compress the trained first neural network model to obtain the second neural network model.

[0067] In implementation, for a first neural network model with a large number of parameters, the computing device can first train it. The training samples used during training vary depending on the target inference task performed by the first neural network model. For example, in image recognition scenarios, the training samples are images and the true category labels (also called category label ground truths) of each object contained in the image. In this scenario, the image is input into the neural network model, and the neural network model outputs the predicted category labels of each object contained in the image. As another example, in face recognition scenarios, the training samples are images and the true location information of faces in the image (also called location information ground truths). In this case, the image is input into the neural network model, and the neural network model outputs the predicted location information of faces in the image (such as the pixel coordinates of the smallest rectangle containing the face). Yet another example, in master audio generation scenarios, the training samples are the lower sampling rate audio of the song and the actual master audio of the song. The lower sampling rate refers to a sampling rate lower than the master audio. In this scenario, the lower sampling rate audio of the song is input into the neural network model, and the neural network model outputs the predicted master audio of the song.

[0068] The following explanation, using the first neural network model as the master audio generation model and its application to perform master audio generation, is illustrated with accompanying diagrams:

[0069] See Figure 3 The computing device acquires the authentic master audio of the first sample song. The first sample song can be any sample song from the music library. The authentic master audio of the first sample song is the initial version after mixing in the recording studio, representing the highest quality audio of the song. The sampling rate of the authentic master audio can reach 192kHz. A first authentic master audio segment is extracted from the authentic master audio of the first sample song. This first authentic master audio segment contains lyrics, and its duration can be configured according to actual needs, such as 10 seconds.

[0070] Then, the computing device downsamples the first real master audio segment to obtain the downsampled first audio frequency band. This downsampling, also known as downsampling, allows the sampling rate of the first audio frequency band to be the same as that commonly used in high-quality audio, for example, 44.1kHz.

[0071] Next, the computing device upsamples the first audio segment to obtain the upsampled second audio frequency band, and uses the second audio segment as the first training sample. The sampling rate of the second audio segment is the same as that of the actual master audio; for example, the sampling rate of the second audio segment is 192kHz.

[0072] Next, the computing device begins training the first neural network model. The first training sample is input into the first neural network model, which then processes the training sample through encoding and decoding, outputting the first predicted master audio segment. Then, the loss value corresponding to the first predicted master audio segment is calculated. The calculation method for the loss value corresponding to the first predicted master audio segment is explained below:

[0073] First, the computing device calculates the first time-domain loss value and the first frequency-domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the actual master audio segment. The calculation method for the first time-domain loss value can be shown in the following formula (1):

[0074]

[0075] Where t represents time, and J represents the number of first training samples included in a training sample set. The value of J can be configured by technicians according to actual needs. For example, J=1 means that each first training sample is calculated with a separate loss value, and J=2 means that two first training samples are calculated with a single loss value. ||||1 represents the first-order norm, y j (t) represents the j-th first training sample included in the training sample set. This represents the first predicted master audio segment corresponding to the first training sample. L time This represents the first time-domain loss value corresponding to the training sample set.

[0076] The calculation method for the first frequency domain loss value can be shown in the following formulas (2)-(5):

[0077]

[0078] Formula (3) is the specific calculation of the cumulative term in Formula (2), Formula (4) is the specific calculation of the first term of the addition in Formula (3), and Formula (5) is the specific calculation of the second term of the addition in Formula (3). Wherein, L freq This represents the first frequency domain loss value corresponding to the training sample set. M represents the number of STFTs (short-time Fourier transforms). Each STFT has a different resolution, determined by the window length and frame shift. For example, M = 3, where the first STFT has a window length of 4096 and a frame shift of 1024, the second STFT has a window length of 2048 and a frame shift of 512, and the third STFT has a window length of 1024 and a frame shift of 256. F denoted by 2, log represents the logarithm, and STFT() represents the short-time Fourier transform.

[0079] After calculating the first time-domain loss value and the first frequency-domain loss value corresponding to the first predicted master audio segment, the computing device calculates the loss value corresponding to the first predicted master audio segment based on the first time-domain loss value and the first frequency-domain loss value. The calculation method can be shown in the following formula (6):

[0080]

[0081] Among them, L loss This represents the loss value corresponding to a training sample set. When J is greater than 1, L can be used as the loss value. loss This represents the loss value corresponding to the first predicted master audio segment for each first training sample in the training sample set. α represents the scaling factor, which can be multiplied by L. freq Before, you can also ride on L time Previously, the specific value could be configured by technical personnel according to actual needs; for example, α = 0.9.

[0082] Then, the computing device trains the first neural network model based on the calculated loss value. There are many possible training methods, and this embodiment does not limit them. One possible training method is illustrated below:

[0083] After obtaining the loss value, the computing device calculates the gradients corresponding to the parameters in the first neural network model based on the loss value and the backpropagation algorithm. Then, based on the calculated gradients for each parameter, gradient descent is used to adjust the parameters, completing one iteration. Training stops when the loss value of the first neural network model is less than a first threshold, or when the number of training iterations reaches a second threshold. The first and second thresholds can be configured by technicians according to actual needs.

[0084] After the first neural network model is trained, the computing device compresses the first neural network model to obtain a second neural network model. The second neural network model has fewer parameters than the first neural network model, and the second neural network model can also perform the above-mentioned target inference task.

[0085] Model compression can include Model Pruning and Model Quantization. A brief introduction to Model Pruning and Model Quantization follows.

[0086] Model pruning: By deleting unimportant neurons or connections in the first neural network model, the size of the model is reduced, resulting in a second neural network model.

[0087] Model quantization: Model compression is achieved by reducing the data type precision of the parameters of the first neural network model, resulting in the second neural network model.

[0088] The technical solution provided in this application integrates time-domain loss and frequency-domain loss when training the master audio generation model. The time-domain loss is used to constrain the consistency of energy and distribution between the predicted master audio and the real master audio, while the frequency-domain loss is used to characterize the high-frequency spectrum, enabling the master audio generation model to learn in the direction of restoring the real master audio.

[0089] In one possible implementation, where the first neural network model is a master audio generation model, the first neural network model can be an AutoEncoder. Correspondingly, the second neural network model is also an AutoEncoder. The difference between the two is that the number of parameters in the second neural network model is less than the number of parameters in the first neural network model.

[0090] Step 202: Input the second training sample into the second neural network model to obtain the first prediction data, and calculate the first loss value corresponding to the first prediction data.

[0091] In practice, after compressing the first neural network model to obtain the second neural network model, the computing device inputs the second training sample into the second neural network model to obtain the first prediction data and calculates the first loss value corresponding to the first prediction data.

[0092] Depending on the target inference task performed by the second neural network model, the training samples used by the computing device to train the second neural network model will also be different. The following explanation, using the example in step 201, illustrates the training of the second neural network model in the scenario of master audio generation:

[0093] See Figure 4 First, the computing device acquires the actual master audio of the second sample song. The second sample song can be any sample song from the music library. The actual master audio of the second sample song is the initial version after mixing in the recording studio, representing the highest quality audio of the song. The sampling rate of the actual master audio can reach 192kHz. Then, a second actual master audio segment is extracted from the actual master audio of the second sample song. This second actual master audio segment contains the lyrics. Furthermore, since the complexity and number of parameters of the second neural network model are reduced compared to the first neural network model, a larger training field can be used when training the second neural network model, resulting in better inference performance. Accordingly, the duration of the extracted second actual master audio segment can be longer than the first actual master audio segment used in step 201 when training the first neural network model.

[0094] Then, the computing device downsamples the second real master audio segment to obtain the downsampled third audio frequency band. This downsampling, also known as downsampling, allows the sampling rate of the third audio frequency band to be the same as that commonly used in high-quality audio, for example, 44.1kHz.

[0095] Next, the computing device upsamples the third audio segment to obtain the upsampled fourth audio frequency band, which is then used as the second training sample. The sampling rate of the fourth audio segment is the same as that of the actual master audio; for example, the sampling rate of the fourth audio segment is 192kHz.

[0096] Next, the computing device begins training the second neural network model. The second training samples are input into the second neural network model, which then processes them through encoding and decoding, outputting the second predicted master audio segment (i.e., the aforementioned first predicted data). Then, the first loss value corresponding to the second predicted master audio segment is calculated. The calculation of the first loss value corresponding to the second predicted master audio segment is explained below:

[0097] The computing device calculates a second time-domain loss value and a second frequency-domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the second real master audio segment, and calculates a first loss value corresponding to the second predicted master audio segment based on the second time-domain loss value and the second frequency-domain loss value.

[0098] Here, the calculation of the first loss value is the same as the calculation of the loss value corresponding to the first predicted master audio segment in step 201 above, and will not be repeated here.

[0099] Step 203: Input the second training sample into the first neural network model to obtain the second prediction data, and calculate the second loss value corresponding to the second prediction data.

[0100] In practice, step 203 and step 202 can be executed simultaneously, or step 202 can be executed first and then step 203, or step 203 can be executed first and then step 202.

[0101] After obtaining the second training sample, the computing device inputs the second training sample into the first neural network model. The first neural network model encodes and decodes the second training sample, outputs the second prediction data, and then calculates the second loss value corresponding to the second prediction data.

[0102] In the scenario of generating the master audio, referring to the example in step 202 above, the fourth audio segment is the second training sample, see [link / reference]. Figure 4The fourth audio segment is input into the first neural network model. The first neural network model performs encoding and decoding operations on the fourth audio segment and outputs the third predicted master audio segment (i.e., the second predicted data). Then, the second loss value corresponding to the third predicted master audio segment is calculated. The calculation of the second loss value corresponding to the third predicted master audio segment is explained below:

[0103] The computing device calculates the third time-domain loss value and the third frequency-domain loss value corresponding to the third predicted master audio segment based on the third predicted master audio segment and the second real master audio segment. Based on the third time-domain loss value and the third frequency-domain loss value, it calculates the second loss value corresponding to the third predicted master audio segment.

[0104] Here, the calculation of the second loss value is the same as the calculation of the loss value corresponding to the first predicted master audio segment in step 201 above, and will not be repeated here.

[0105] Step 204: Based on the first loss value and the second loss value, train the second neural network model to obtain the trained second neural network model.

[0106] In implementation, after obtaining the first loss value of the second neural network model and the second loss value of the first neural network model, the computing device fuses the first and second loss values ​​to obtain a fused loss value. Then, the second neural network model is trained based on the fused loss value. Here, there are many specific training methods, and this application embodiment does not limit them. The following is an exemplary example of one training method.

[0107] After obtaining the fusion loss value, the computing device calculates the gradients corresponding to the parameters in the second neural network model based on the fusion loss value and the backpropagation algorithm. Then, based on the calculated gradients corresponding to each parameter, the gradient descent method is used to adjust the parameters to complete one iteration.

[0108] The following explains the fusion of the first and second loss values:

[0109] The first loss value and the second loss value can be fused using the following formula (7):

[0110]

[0111] Where L is the fusion loss value of the first loss and the second loss, β large These are the loss weights for the first neural network model. The second loss value, β small These are the loss weights for the second neural network model. Let β be the first loss value. large and β smallThe value of β can be configured by technical personnel according to actual needs. large and β small The value can be adjusted. and The proportion in the fusion loss value. For example, β large =1,β small =1.1. For example, β large +β small =1, where β large =0.4, β small =0.6.

[0112] After obtaining the trained second neural network model, the computing device can directly or indirectly distribute the trained second neural network model to the terminal. Indirect distribution includes, but is not limited to, first storing the trained second neural network model on a storage medium, and then copying it from the storage medium to the terminal. The terminal deploys the trained second neural network model, and when it needs to perform related inference tasks, it calls the locally deployed trained second neural network model to execute the relevant inference tasks. For example, taking master audio generation as an example, when it is necessary to generate master audio, the terminal calls the locally deployed trained second neural network model, inputs the audio to be processed from the song into the trained second neural network model, and the trained second neural network model outputs the master audio of the song.

[0113] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0114] The technical solution provided in this application compresses a first neural network model that has already been trained, resulting in a second neural network model with fewer parameters. Furthermore, to improve the inference performance of the compressed second neural network model, this solution uses the same training samples and the loss values ​​from both the first and second neural network models to train the second neural network model together, making the inference performance of the trained second neural network model closer to that of the first neural network model.

[0115] Based on the same technical concept, embodiments of this application also provide an apparatus for training a neural network model. This apparatus is applied to a computing device, such as... Figure 5 As shown, the device includes a compression module 510 and a training module 520, wherein:

[0116] Compression module 510 is used to compress the trained first neural network model to obtain a second neural network model, wherein the number of parameters of the second neural network model is less than the number of parameters of the first neural network model.

[0117] The training module 520 is used to input the first training sample into the second neural network model to obtain the first prediction data and calculate the first loss value corresponding to the first prediction data; input the first training sample into the first neural network model to obtain the second prediction data and calculate the second loss value corresponding to the second prediction data; and train the second neural network model based on the first loss value and the second loss value to obtain the trained second neural network model.

[0118] In one possible implementation, the training module 520 is used for:

[0119] Multiply the first loss value by the first weight to obtain the first product;

[0120] Multiply the second loss value by the second weight to obtain the second product;

[0121] Add the first product and the second product to obtain the fusion loss value;

[0122] The second neural network model is trained based on the fusion loss value.

[0123] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio of the first song.

[0124] In one possible implementation, the training module 520 is further configured to:

[0125] Obtain the actual master audio clips of the sample songs;

[0126] The actual master audio segment is downsampled to obtain the downsampled audio segment;

[0127] The downsampled audio segment is upsampled to obtain an upsampled audio segment, which is used as the first training sample. The sampling rate of the upsampled audio segment is the same as the sampling rate of the real master audio segment.

[0128] In one possible implementation, the duration of the actual master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0129] In one possible implementation, the first prediction data is a first predicted master audio segment, the second prediction data is a second predicted master audio segment, and the training module 520 is used for:

[0130] Based on the first predicted master audio segment and the real master audio segment, calculate the first time domain loss value and the first frequency domain loss value corresponding to the first predicted master audio segment;

[0131] The first time-domain loss value and the first frequency-domain loss value are added together to obtain the first loss value corresponding to the first predicted master audio segment;

[0132] Based on the second predicted master audio segment and the real master audio segment, calculate the second time-domain loss value and the second frequency-domain loss value corresponding to the second predicted master audio segment;

[0133] The second time-domain loss value and the second frequency-domain loss value are added together to obtain the second loss value corresponding to the second predicted master audio segment.

[0134] In one possible implementation, the first neural network model is an autoencoder.

[0135] In one possible implementation, the trained second neural network model is deployed on a terminal, which, upon invocation of the terminal, generates the master audio of the second song based on the input audio to be processed of the second song.

[0136] The technical solution provided in this application compresses a first neural network model that has already been trained, resulting in a second neural network model with fewer parameters. Furthermore, to improve the inference performance of the compressed second neural network model, this solution uses the same training samples and the loss values ​​from both the first and second neural network models to train the second neural network model together, making the inference performance of the trained second neural network model closer to that of the first neural network model.

[0137] It should be noted that the neural network model training apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when training a neural network model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device can be divided into different functional modules to complete all or part of the functions described above. In addition, the neural network model training apparatus and the neural network model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0138] Figure 6A structural block diagram of a terminal 600 provided in an exemplary embodiment of this application is shown. The terminal 600 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 600 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names. The terminal 600 may deploy the aforementioned second neural network model.

[0139] Typically, terminal 600 includes a processor 601 and a memory 602.

[0140] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0141] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices.

[0142] In some embodiments, the terminal 600 may also optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.

[0143] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0144] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0145] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0146] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0147] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0148] The positioning component 608 is used to determine the current geographic location of the terminal 600 in order to enable navigation or LBS (Location Based Service). The positioning component 608 can be a positioning component of GPS (Global Positioning System), BeiDou system, or Galileo system.

[0149] Power supply 609 is used to supply power to the various components in terminal 600. Power supply 609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0150] In some embodiments, the terminal 600 further includes one or more sensors 610. The one or more sensors 610 include, but are not limited to: an accelerometer 611, a gyroscope 612, a pressure sensor 613, a fingerprint sensor 614, an optical sensor 615, and a proximity sensor 616.

[0151] Accelerometer 611 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 600. For example, accelerometer 611 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 601 can control display screen 605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 611. Accelerometer 611 can also be used for games or for acquiring user motion data.

[0152] The gyroscope sensor 612 can detect the orientation and rotation angle of the terminal 600. The gyroscope sensor 612, in conjunction with the accelerometer sensor 611, can collect 3D motion data from the user on the terminal 600. Based on the data collected by the gyroscope sensor 612, the processor 601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0153] The pressure sensor 613 can be disposed on the side bezel of the terminal 600 and / or on the lower layer of the display screen 605. When the pressure sensor 613 is disposed on the side bezel of the terminal 600, it can detect the user's grip signal on the terminal 600, and the processor 601 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 613. When the pressure sensor 613 is disposed on the lower layer of the display screen 605, the processor 601 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 605. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0154] The fingerprint sensor 614 is used to collect the user's fingerprint. The processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 614 can be located on the front, back, or side of the terminal 600. When the terminal 600 has physical buttons or a manufacturer's logo, the fingerprint sensor 614 can be integrated with the physical buttons or manufacturer's logo.

[0155] An optical sensor 615 is used to collect ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 based on the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is decreased. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 based on the ambient light intensity collected by the optical sensor 615.

[0156] The proximity sensor 616, also known as a distance sensor, is typically mounted on the front panel of the terminal 600. The proximity sensor 616 is used to detect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from a screen-on state to a screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from a screen-off state to a screen-on state.

[0157] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0158] Figure 7 This is a schematic diagram of a computing device 1000 provided in an embodiment of this application. The computing device 1000 can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 1001 and one or more memories 1002. The memories 1002 store at least one instruction, which is loaded and executed by the processors 1001 to implement the methods provided in the various method embodiments described above. Of course, the computing device may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computing device may also include other components for implementing device functions, which will not be elaborated upon here.

[0159] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the method for training a neural network model in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0160] In an exemplary embodiment, a computer program product is also provided, which stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the method for training the neural network model described above.

[0161] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample songs and neural network models involved in this application were obtained with full authorization.

[0162] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0163] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a neural network model, characterized in that, The method includes: The trained first neural network model is compressed to obtain a second neural network model, wherein the number of parameters in the second neural network model is less than the number of parameters in the first neural network model. The first training sample is input into the second neural network model to obtain the first predicted master audio segment. Based on the first predicted master audio segment and the corresponding real master audio segment, the first time domain loss value and the first frequency domain loss value corresponding to the first predicted master audio segment are calculated. Based on the first time domain loss value and the first frequency domain loss value, the first loss value corresponding to the first predicted master audio segment is calculated. The first training sample is input into the first neural network model to obtain the second predicted master audio segment. Based on the second predicted master audio segment and the corresponding real master audio segment, the second time domain loss value and the second frequency domain loss value corresponding to the second predicted master audio segment are calculated. Based on the second time domain loss value and the second frequency domain loss value, the second loss value corresponding to the second predicted master audio segment is calculated. Multiply the first loss value by the first weight to obtain the first product, and multiply the second loss value by the second weight to obtain the second product; Add the first product and the second product to obtain the fusion loss value; The second neural network model is trained based on the fusion loss value.

2. The method according to claim 1, characterized in that, The first neural network model is used to generate the master audio of the first song based on the input audio of the first song.

3. The method according to claim 1, characterized in that, Before inputting the first training sample into the second neural network model, the method further includes: Obtain the actual master audio clips of the sample songs; The actual master audio segment is downsampled to obtain the downsampled audio segment; The downsampled audio segment is upsampled to obtain an upsampled audio segment, which is used as the first training sample. The sampling rate of the upsampled audio segment is the same as the sampling rate of the real master audio segment.

4. The method according to claim 1, characterized in that, The duration of the actual master audio segment is greater than the duration of the audio segment used to train the first neural network model.

5. The method according to claim 1, characterized in that, The first loss value corresponding to the first predicted master audio segment is calculated based on the first time-domain loss value and the first frequency-domain loss value. The first time-domain loss value and the first frequency-domain loss value are added together to obtain the first loss value corresponding to the first predicted master audio segment; The step of calculating the second loss value corresponding to the second predicted master audio segment based on the second time-domain loss value and the second frequency-domain loss value includes: The second time-domain loss value and the second frequency-domain loss value are added together to obtain the second loss value corresponding to the second predicted master audio segment.

6. The method according to claim 1, characterized in that, The first neural network model is an autoencoder.

7. The method according to any one of claims 1-6, characterized in that, The trained second neural network model is deployed on a terminal, and when invoked by the terminal, it generates the master audio of the second song based on the input audio to be processed of the second song.

8. A computing device, characterized in that, The computing device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operation performed by the method for training a neural network model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operation of the method for training a neural network model as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded and executed by a processor to perform the operation of the method for training a neural network model as described in any one of claims 1 to 7.

Images