Neural network model training method and device, and storage medium

By performing model compression and joint training on large neural network models, the problem of insufficient resources for terminal equipment running large models is solved, and the model's inference effect at the terminal is improved.

CN119089942AActive Publication Date: 2024-12-06TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411185893.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-12-06
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

Due to limited storage and computing resources, terminal devices are difficult to deploy and run large neural network models with huge parameters, resulting in poor inference results.

Method used

By compressing the trained neural network model, a model with small parameters is generated, and the loss value of the large model and the loss value of the small model are trained together to improve the inference effect of the small model and make it close to the performance of the large model .

Benefits of technology

The parameter amount of neural network model deployed on terminal devices is reduced, while improving the inference effect of the model, so that it can run efficiently under limited resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089942A_ABST
    Figure CN119089942A_ABST
Patent Text Reader

Abstract

The invention provides a neural network model training method and device and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the model compression of a trained first neural network model, and obtaining a second neural network model, the parameter quantity of the second network model is smaller than that of the first neural network model; inputting a first training sample into the second neural network model to obtain first prediction data, and calculating a first loss value corresponding to the first prediction data; inputting the first training sample into the first neural network model to obtain second prediction data, and calculating a second loss value corresponding to the second prediction data; and training the second neural network model based on the first loss value and the second loss value to obtain a trained second neural network model. According to the invention, the neural network model suitable for the terminal can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for training a neural network model. Background Art

[0002] Neural network models play a vital role in artificial intelligence. In order to achieve better reasoning effects, the number of parameters in neural network models is increasing. Usually, the number of parameters in large models can reach hundreds of millions, billions, or even more.

[0003] The server has a large amount of storage and computing resources, which are sufficient to deploy and run large models, while the storage and computing resources of the terminal are very limited and may not be able to run large models with huge parameters. In this case, how to obtain a neural network model that can be deployed and run on the terminal has become an urgent problem to be solved. Summary of the invention

[0004] The embodiment of the present application provides a method, device and storage medium for training a neural network model, which can obtain a neural network model with a small number of parameters suitable for a terminal. The technical solution is as follows:

[0005] In a first aspect, a method for training a neural network model is provided, the method comprising:

[0006] Performing model compression on the trained first neural network model to obtain a second neural network model, wherein the parameter amount of the second neural network model is smaller than the parameter amount of the first neural network model;

[0007] Inputting the first training sample into the second neural network model to obtain first prediction data, and calculating a first loss value corresponding to the first prediction data;

[0008] Inputting the first training sample into the first neural network model to obtain second prediction data, and calculating a second loss value corresponding to the second prediction data;

[0009] Based on the first loss value and the second loss value, the second neural network model is trained to obtain a trained second neural network model.

[0010] In a possible implementation, the training of the second neural network model based on the first loss value and the second loss value includes:

[0011] Multiplying the first loss value by a first weight to obtain a first product;

[0012] Multiplying the second loss value by a second weight to obtain a second product;

[0013] Adding the first product and the second product to obtain a fusion loss value;

[0014] Based on the fusion loss value, the second neural network model is trained.

[0015] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

[0016] In a possible implementation, before inputting the first training sample into the second neural network model, the method further includes:

[0017] Get the real master audio clip of the sample song;

[0018] Downsampling the real master audio segment to obtain a downsampled audio segment;

[0019] The downsampled audio segment is upsampled to obtain an upsampled audio segment as a first training sample, wherein a sampling rate of the upsampled audio segment is the same as a sampling rate of the real master audio segment.

[0020] In a possible implementation, the duration of the real master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0021] In a possible implementation, the first prediction data is a first prediction master audio segment, the second prediction data is a second prediction master audio segment, and the calculating the first loss value corresponding to the first prediction data includes:

[0022] Calculating a first time domain loss value and a first frequency domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the real master audio segment;

[0023] Adding the first time domain loss value and the first frequency domain loss value to obtain a first loss value corresponding to the first predicted master audio segment;

[0024] The calculating a second loss value corresponding to the second prediction data includes:

[0025] Calculating a second time domain loss value and a second frequency domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the real master audio segment;

[0026] The second time domain loss value and the second frequency domain loss value are added to obtain a second loss value corresponding to the second predicted master audio segment.

[0027] In a possible implementation, the first neural network model is an autoencoder.

[0028] In a possible implementation, the trained second neural network model is deployed on a terminal, and when called by the terminal, the master audio of the second song is generated based on the input audio to be processed of the second song.

[0029] In a second aspect, a device for training a neural network model is provided, the device comprising:

[0030] A compression module, used for compressing the trained first neural network model to obtain a second neural network model, wherein the parameter amount of the second neural network model is less than the parameter amount of the first neural network model;

[0031] A training module is used to input the first training sample into the second neural network model to obtain first prediction data, and calculate a first loss value corresponding to the first prediction data; input the first training sample into the first neural network model to obtain second prediction data, and calculate a second loss value corresponding to the second prediction data; based on the first loss value and the second loss value, train the second neural network model to obtain a trained second neural network model.

[0032] In a possible implementation, the training module is used to:

[0033] Multiplying the first loss value by a first weight to obtain a first product;

[0034] Multiplying the second loss value by a second weight to obtain a second product;

[0035] Adding the first product and the second product to obtain a fusion loss value;

[0036] Based on the fusion loss value, the second neural network model is trained.

[0037] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

[0038] In a possible implementation, the training module is further used to:

[0039] Get the real master audio clip of the sample song;

[0040] Downsampling the real master audio segment to obtain a downsampled audio segment;

[0041] The downsampled audio segment is upsampled to obtain an upsampled audio segment as a first training sample, wherein a sampling rate of the upsampled audio segment is the same as a sampling rate of the real master audio segment.

[0042] In a possible implementation, the duration of the real master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0043] In a possible implementation, the first prediction data is a first prediction master audio segment, the second prediction data is a second prediction master audio segment, and the training module is used to:

[0044] Calculating a first time domain loss value and a first frequency domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the real master audio segment;

[0045] Adding the first time domain loss value and the first frequency domain loss value to obtain a first loss value corresponding to the first predicted master audio segment;

[0046] Calculating a second time domain loss value and a second frequency domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the real master audio segment;

[0047] The second time domain loss value and the second frequency domain loss value are added to obtain a second loss value corresponding to the second predicted master audio segment.

[0048] In a possible implementation, the first neural network model is an autoencoder.

[0049] In a possible implementation, the trained second neural network model is deployed on a terminal, and when called by the terminal, the master audio of the second song is generated based on the input audio to be processed of the second song.

[0050] In a third aspect, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for training a neural network model as described in the first aspect and any possible implementation of the first aspect.

[0051] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the operations performed by the method for training a neural network model as described in the first aspect and any possible implementation of the first aspect.

[0052] In a fifth aspect, a computer program product is provided, wherein at least one instruction is stored in the computer program product, and the instruction is loaded and executed by a processor to implement the operations performed by the method for training a neural network model as described in the first aspect and any possible implementation of the first aspect.

[0053] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0054] The technical solution provided by the embodiment of the present application performs model compression on the first neural network model that has been trained to obtain a second neural network model with a smaller number of parameters than the first neural network model. In addition, in order to make the second neural network model obtained by model compression have a better reasoning effect, in this solution, the loss values ​​of the same training samples in the first neural network model and the second neural network model are used to train the second neural network model together, so that the reasoning effect of the trained second neural network model is closer to the first neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 is a schematic diagram of a real-time scenario provided in an embodiment of the present application;

[0057] Figure 2 It is a flow chart of a method for training a neural network model provided in an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of a neural network model training provided in an embodiment of the present application;

[0059] Figure 4 is a schematic diagram of a neural network model training provided in an embodiment of the present application;

[0060] Figure 5 It is a schematic diagram of the structure of a device for training a neural network model provided in an embodiment of the present application;

[0061] Figure 6 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0062] Figure 7 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0064] The present application embodiment provides a method for training a neural network model, which can be implemented by a computing device, which can be a server, a server cluster, etc. Figure 1 , shows a possible implementation scenario of the embodiment of the present application. For a Large Model with a large number of parameters, the computing device trains the large model. After the training is completed, the trained large model can be deployed on the computing device to perform the target reasoning task. The target reasoning task can be music master audio generation, speech recognition, audio feature extraction, image generation, etc. In the case where the terminal has a need to perform the target reasoning task locally, considering that the large model has a large number of parameters and cannot be deployed on the terminal, the large model can be compressed to obtain a small model with a small number of parameters but can also perform the target reasoning task. The small model can be deployed on the terminal. However, if the small model is directly deployed on the terminal, although the target reasoning task can also be performed, the reasoning effect may be poor. In this case, the method for training a neural network model provided in the embodiment of the present application can be adopted. After the trained large model is compressed and the small model is obtained, the small model is continued to be trained. When training the small model, the large model is used to guide the small model, that is, based on the loss value of the large model and the loss value of the small model, the small model is trained together so that the reasoning effect of the small model is closer to the large model. After the small model is trained, it can be deployed on the terminal.

[0065] The following is a method for training a neural network model provided in an embodiment of the present application. The method can be implemented by a computing device. Figure 2 , the processing flow of the method may include the following steps:

[0066] Step 201: compress the trained first neural network model to obtain a second neural network model.

[0067] In implementation, for the first neural network model with a large number of parameters, the computing device may first train it, and the training samples used in the training of the first neural network model may be different according to the different target reasoning tasks performed by the first neural network model. For example, in the image recognition scenario, the training samples are the real category identifications of the image and the objects contained in the image (also referred to as the true value of the category identification). In this scenario, the image is input into the neural network model, and the neural network model outputs the predicted category identifications of the objects contained in the image. For another example, in the face recognition scenario, the training samples are the real position information of the image and the face in the image (also referred to as the true value of the position information). In this case, the image is input into the neural network model, and the neural network model outputs the predicted position information of the face in the image (such as the pixel coordinates of the smallest rectangular box containing the face). For another example, in the master audio generation scenario, the training samples are the audio of the song with a lower sampling rate and the real master audio of the song, wherein the lower sampling rate refers to the sampling rate lower than the master audio. In this scenario, the audio of the song with a lower sampling rate is input into the neural network model, and the neural network model outputs the predicted master audio of the song.

[0068] The following takes the first neural network model as a master audio generation model to perform master audio generation as an example, and describes the training of the first neural network model in conjunction with the accompanying drawings:

[0069] See also Figure 3 , the computing device obtains the real master audio of the first sample song, wherein the first sample song can be any sample song in the music library, the real master of the first sample song is the original version of the first sample song after the mixing in the recording studio is completed, and is the song audio with the highest sound quality of the first sample song, and the sampling rate of the real master audio can reach 192kHz. A first real master audio segment is intercepted from the real master audio of the first sample song, wherein the first real master audio segment is a segment containing lyrics, and the duration of the first real master audio segment can be configured according to actual needs, such as 10 seconds.

[0070] Then, the computing device downsamples the first real master audio segment to obtain a downsampled first audio frequency band. Downsampling can also be called downsampling, and the sampling rate of the first audio frequency band can be the same as the sampling rate of high-quality audio that is currently commonly used. For example, the sampling rate of the first audio frequency band can be 44.1kHz.

[0071] Then, the computing device upsamples the first audio segment to obtain an upsampled second audio frequency band, and uses the second audio segment as a first training sample, wherein the sampling rate of the second audio segment is the same as the sampling rate of the real master audio, for example, the sampling rate of the second audio segment is 192 kHz.

[0072] Next, the computing device starts training the first neural network model, inputs the first training sample into the first neural network model, and the first neural network model encodes, decodes, and processes the first training sample, and then outputs a first predicted master audio segment. Then, the loss value corresponding to the first predicted master audio segment is calculated. The following is an explanation of the method for calculating the loss value corresponding to the first predicted master audio segment:

[0073] First, the computing device calculates a first time domain loss value and a first frequency domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the real master audio segment. The calculation method of the first time domain loss value can be shown in the following formula (1):

[0074]

[0075] Where t represents time, J represents the number of first training samples included in a training sample set, and the value of J can be configured by technicians according to actual needs. For example, J=1, J=1 means that each first training sample calculates a loss value separately, and J=2 means that two first training samples calculate a loss value. ||||1 represents the first-order norm, y j (t) represents the jth first training sample included in the training sample set, represents the first predicted master audio segment corresponding to the first training sample. time Represents the first time domain loss value corresponding to the training sample set.

[0076] The calculation method of the first frequency domain loss value can be shown in the following formulas (2)-(5):

[0077]

[0078] Formula (3) is the specific calculation of the cumulative term in formula (2), formula (4) is the specific calculation of the first term of the addition in formula (3), and formula (5) is the specific calculation of the second term of the addition in formula (3). freq represents the first frequency domain loss value corresponding to the training sample set, M represents the number of STFTs (short-time Fourier transform), the resolution of each STFT is different, and the resolution of STFT is determined by the window length and frame shift. For example, M=3, where the window length of the first STFT is 4096 and the frame shift is 1024, the window length of the second STFT is 2048 and the frame shift is 512, and the window length of the third STFT is 1024 and the frame shift is 256, |||| F represents the second-order norm, log represents logarithm, and STFT() represents short-time Fourier transform.

[0079] After calculating the first time domain loss value and the first frequency domain loss value corresponding to the first predicted master audio segment, the computing device calculates the loss value corresponding to the first predicted master audio segment based on the first time domain loss value and the first frequency domain loss value. The computing method may be as shown in the following formula (6):

[0080]

[0081] Among them, L loss Represents the loss value corresponding to a training sample set. When J is greater than 1, L loss As the loss value corresponding to the first predicted master audio segment corresponding to each first training sample in the training sample set. α represents the scaling factor, which can be multiplied by L freq Before, you can also take L time Before, the specific value can be configured by the technicians according to the actual needs, for example, α=0.9.

[0082] Then, the computing device trains the first neural network model according to the calculated loss value. Here, there can be many training methods, which are not limited in the present embodiment of the application. The following is an exemplary example of a possible training method:

[0083] After obtaining the loss value, the computing device calculates the gradient corresponding to the parameters in the first neural network model according to the loss value and the back propagation algorithm, and then uses the gradient descent method to adjust the parameters according to the calculated gradient corresponding to each parameter to complete a round of iteration. The training is stopped until the loss value of the first neural network model is less than the first threshold, or the number of training iterations reaches the second threshold. The first threshold and the second threshold can be configured by the technician according to actual needs.

[0084] After the training of the first neural network model is completed, the computing device compresses the first neural network model to obtain a second neural network model. The parameter amount of the second neural network model is smaller than that of the first neural network model, and the second neural network model can also perform the above-mentioned target reasoning task.

[0085] Among them, model compression can be Model Pruning, Model Quantification, etc. The following is a brief introduction to model pruning and model quantization.

[0086] Model pruning: By deleting unimportant neurons or connections in the first neural network model, the size of the model is reduced to obtain the second neural network model.

[0087] Model quantization: Model compression is achieved by reducing the data type precision of the parameters of the first neural network model to obtain the second neural network model.

[0088] The technical solution provided in the embodiment of the present application integrates time domain loss values ​​and frequency domain loss values ​​when training the master audio generation model. The time domain loss values ​​are used to constrain the consistency of energy and distribution between the predicted master audio and the real master audio, and the frequency domain loss values ​​are used to characterize the high-frequency spectrum, so that the master audio generation model learns in the direction of restoring the real master audio.

[0089] In one possible implementation, when the first neural network model is a master audio generation model, the first neural network model can be an AutoEncoder, and correspondingly, the second neural network model is also an AutoEncoder. The difference between the two is that the parameter amount of the second neural network model is smaller than the parameter amount of the first neural network model.

[0090] Step 202: input the second training sample into the second neural network model to obtain first prediction data, and calculate a first loss value corresponding to the first prediction data.

[0091] During implementation, after compressing the first neural network model to obtain the second neural network model, the computing device inputs the second training sample into the second neural network model to obtain the first prediction data, and calculates the first loss value corresponding to the first prediction data.

[0092] According to different target reasoning tasks performed by the second neural network model, the training samples used by the computing device when training the second neural network model are also different. The training of the second neural network model is described below in the scenario of master audio generation in conjunction with the example in step 201:

[0093] See also Figure 4 First, the computing device obtains the real master audio of the second sample song, wherein the second sample song can be any sample song in the music library, and the real master of the second sample song is the original version of the second sample song after the mixing in the recording studio is completed, which is the song audio with the highest sound quality of the second sample song, and the sampling rate of the real master audio can reach 192kHz. Then, a second real master audio segment is intercepted from the real master audio of the second sample song, wherein the second real master audio segment is a segment containing lyrics. In addition, since the complexity and parameter amount of the second neural network model are reduced compared to the first neural network model, a larger training field of view can be used when training the second neural network model, so that the second neural network model can achieve better reasoning effect. Accordingly, the duration of the intercepted second real master audio segment can be greater than the first real master audio segment used when training the first neural network model in the above step 201.

[0094] Then, the computing device downsamples the second real master audio segment to obtain a downsampled third audio frequency band. Downsampling can also be called downsampling, and the sampling rate of the third audio frequency band can be the same as the sampling rate of high-quality audio that is currently commonly used. For example, the sampling rate of the third audio frequency band can be 44.1kHz.

[0095] Then, the computing device upsamples the third audio segment to obtain an upsampled fourth audio frequency band, and uses the fourth audio segment as a second training sample, wherein the sampling rate of the fourth audio segment is the same as the sampling rate of the real master audio, for example, the sampling rate of the fourth audio segment is 192 kHz.

[0096] Next, the computing device starts training the second neural network model, inputs the second training sample into the second neural network model, and the second neural network model encodes, decodes, and processes the second training sample, and then outputs the second predicted master audio segment (i.e., the first predicted data mentioned above). Then, the first loss value corresponding to the second predicted master audio segment is calculated. The following is an explanation of calculating the first loss value corresponding to the second predicted master audio segment:

[0097] The computing device calculates the second time domain loss value and the second frequency domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the second real master audio segment, and calculates the first loss value corresponding to the second predicted master audio segment based on the second time domain loss value and the second frequency domain loss value.

[0098] Here, the calculation of the first loss value is the same as the calculation of the loss value corresponding to the first predicted master audio segment in the above step 201, and will not be repeated here.

[0099] Step 203: input the second training sample into the first neural network model to obtain second prediction data, and calculate a second loss value corresponding to the second prediction data.

[0100] In implementation, step 203 and step 202 may be performed simultaneously, or step 202 may be performed first and then step 203, or step 203 may be performed first and then step 202.

[0101] After acquiring the second training sample, the computing device inputs the second training sample into the first neural network model, the first neural network model encodes, decodes, and processes the second training sample, outputs second prediction data, and then calculates a second loss value corresponding to the second prediction data.

[0102] In the scenario of master audio generation, combined with the example in step 202 above, the fourth audio segment is the second training sample, see Figure 4, the fourth audio segment is input into the first neural network model, the first neural network model performs encoding, decoding and other operations on the fourth audio segment, and outputs a third predicted master audio segment (i.e., second predicted data). Then, the second loss value corresponding to the third predicted master audio segment is calculated. The calculation of the second loss value corresponding to the third predicted master audio segment is described below:

[0103] The computing device calculates a third time domain loss value and a third frequency domain loss value corresponding to the third predicted master audio segment based on the third predicted master audio segment and the second real master audio segment, and calculates a second loss value corresponding to the third predicted master audio segment based on the third time domain loss value and the third frequency domain loss value.

[0104] Here, the calculation of the second loss value is the same as the calculation of the loss value corresponding to the first predicted master audio segment in the above step 201, and will not be repeated here.

[0105] Step 204: Based on the first loss value and the second loss value, train the second neural network model to obtain a trained second neural network model.

[0106] In implementation, after obtaining the first loss value of the second neural network model and the second loss value of the first neural network model, the computing device fuses the first loss value and the second loss value to obtain a fused loss value, and then trains the second neural network model according to the fused loss value. Here, there can be many specific training methods, which are not limited in the embodiments of the present application. An exemplary training method is listed below for illustration.

[0107] After obtaining the fusion loss value, the computing device calculates the gradients corresponding to the parameters in the second neural network model based on the fusion loss value and the back propagation algorithm, and then adjusts the parameters using the gradient descent method based on the calculated gradients corresponding to each parameter to complete a round of iteration.

[0108] The following is an explanation of the fusion of the first loss value and the second loss value:

[0109] The first loss value and the second loss value are fused as shown in the following formula (7):

[0110]

[0111] Among them, L is the fusion loss value of the first loss value and the second loss value, β large is the loss weight of the first neural network model, is the second loss value, β small is the loss weight of the second neural network model, is the first loss value, where β large and β smallThe value of can be configured by technicians according to actual needs. large and β small The value of can be adjusted and The proportion of the fusion loss value. For example, β large =1,β small =1.1. For another example, β large +β small =1, where β large =0.4,β small =0.6.

[0112] After obtaining the trained second neural network model, the computing device can send the trained second neural network model directly or indirectly to the terminal, wherein the indirect sending includes but is not limited to first storing the trained second neural network model in a storage medium, and then copying it from the storage medium to the terminal. The terminal deploys the trained second neural network model, and when it is necessary to perform related reasoning tasks, the terminal calls the locally deployed trained second neural network model to perform related reasoning tasks. For example, taking master audio generation as an example, when it is necessary to generate master audio, the terminal calls the locally deployed trained second neural network model, inputs the song's audio to be processed into the trained second neural network model, and the trained second neural network model outputs the master audio of the song.

[0113] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0114] The technical solution provided in the embodiment of the present application performs model compression on the first neural network model that has been trained to obtain a second neural network model with a smaller number of parameters than the first neural network model. In addition, in order to make the second neural network model obtained by model compression have a better reasoning effect, in this solution, the loss values ​​of the same training samples in the first neural network model and the second neural network model are used to train the second neural network model together, so that the reasoning effect of the trained second neural network model is closer to the first neural network model.

[0115] Based on the same technical concept, the embodiment of the present application also provides a device for training a neural network model, which is applied to a computing device, such as Figure 5 As shown, the apparatus includes a compression module 510 and a training module 520, wherein:

[0116] A compression module 510 is used to compress the trained first neural network model to obtain a second neural network model, wherein the parameter amount of the second neural network model is smaller than the parameter amount of the first neural network model;

[0117] The training module 520 is used to input the first training sample into the second neural network model to obtain the first prediction data, and calculate the first loss value corresponding to the first prediction data; input the first training sample into the first neural network model to obtain the second prediction data, and calculate the second loss value corresponding to the second prediction data; based on the first loss value and the second loss value, the second neural network model is trained to obtain a trained second neural network model.

[0118] In a possible implementation, the training module 520 is used to:

[0119] Multiplying the first loss value by a first weight to obtain a first product;

[0120] Multiplying the second loss value by a second weight to obtain a second product;

[0121] Adding the first product and the second product to obtain a fusion loss value;

[0122] Based on the fusion loss value, the second neural network model is trained.

[0123] In one possible implementation, the first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

[0124] In a possible implementation, the training module 520 is further configured to:

[0125] Get the real master audio clip of the sample song;

[0126] Downsampling the real master audio segment to obtain a downsampled audio segment;

[0127] The downsampled audio segment is upsampled to obtain an upsampled audio segment as a first training sample, wherein a sampling rate of the upsampled audio segment is the same as a sampling rate of the real master audio segment.

[0128] In a possible implementation, the duration of the real master audio segment is greater than the duration of the audio segment used to train the first neural network model.

[0129] In a possible implementation, the first prediction data is a first prediction master audio segment, the second prediction data is a second prediction master audio segment, and the training module 520 is used to:

[0130] Calculating a first time domain loss value and a first frequency domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the real master audio segment;

[0131] Adding the first time domain loss value and the first frequency domain loss value to obtain a first loss value corresponding to the first predicted master audio segment;

[0132] Calculating a second time domain loss value and a second frequency domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the real master audio segment;

[0133] The second time domain loss value and the second frequency domain loss value are added to obtain a second loss value corresponding to the second predicted master audio segment.

[0134] In a possible implementation, the first neural network model is an autoencoder.

[0135] In a possible implementation, the trained second neural network model is deployed on a terminal, and when called by the terminal, the master audio of the second song is generated based on the input audio to be processed of the second song.

[0136] The technical solution provided in the embodiment of the present application performs model compression on the first neural network model that has been trained to obtain a second neural network model with a smaller number of parameters than the first neural network model. In addition, in order to make the second neural network model obtained by model compression have a better reasoning effect, in this solution, the loss values ​​of the same training samples in the first neural network model and the second neural network model are used to train the second neural network model together, so that the reasoning effect of the trained second neural network model is closer to the first neural network model.

[0137] It should be noted that: the apparatus for training a neural network model provided in the above embodiment only uses the division of the above functional modules as an example when training a neural network model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computing device is divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for training a neural network model provided in the above embodiment and the method embodiment for training a neural network model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0138] Figure 6The structural block diagram of a terminal 600 provided by an exemplary embodiment of the present application is shown. The terminal 600 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 600 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names. The terminal 600 may deploy the above-mentioned second neural network model.

[0139] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0140] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0141] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices.

[0142] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609.

[0143] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0144] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0145] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, set on the front panel of the terminal 600; in other embodiments, the display screen 605 can be at least two, respectively set on different surfaces of the terminal 600 or in a folding design; in other embodiments, the display screen 605 can be a flexible display screen, set on the curved surface or folding surface of the terminal 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0146] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0147] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0148] The positioning component 608 is used to locate the current geographical location of the terminal 600 to implement navigation or LBS (Location Based Service). The positioning component 608 can be a positioning component of the GPS (Global Positioning System), Beidou system or Galileo system.

[0149] The power supply 609 is used to power various components in the terminal 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0150] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0151] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0152] The gyro sensor 612 can detect the body direction and rotation angle of the terminal 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the terminal 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0153] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's holding signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0154] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or a manufacturer logo is set on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0155] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0156] The proximity sensor 616, also called a distance sensor, is usually arranged on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0157] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0158] Figure 7 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. The computing device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 1001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the computing device may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the computing device may also include other components for implementing device functions, which will not be described in detail here.

[0159] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in a terminal to complete the method of training a neural network model in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0160] In an exemplary embodiment, a computer program product is also provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the above-mentioned method for training a neural network model.

[0161] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample songs, neural network models, etc. involved in this application are all obtained with full authorization.

[0162] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0163] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a neural network model, characterized in that: The method comprises: Performing model compression on the trained first neural network model to obtain a second neural network model, wherein the parameter amount of the second neural network model is smaller than the parameter amount of the first neural network model; Inputting the first training sample into the second neural network model to obtain first prediction data, and calculating a first loss value corresponding to the first prediction data; Inputting the first training sample into the first neural network model to obtain second prediction data, and calculating a second loss value corresponding to the second prediction data; Based on the first loss value and the second loss value, the second neural network model is trained to obtain a trained second neural network model.

2. The method according to claim 1, characterized in that The step of training the second neural network model based on the first loss value and the second loss value includes: Multiplying the first loss value by a first weight to obtain a first product; Multiplying the second loss value by a second weight to obtain a second product; Adding the first product and the second product to obtain a fusion loss value; Based on the fusion loss value, the second neural network model is trained.

3. The method according to claim 1 or 2, characterized in that: The first neural network model is used to generate the master audio of the first song based on the input audio to be processed of the first song.

4. The method according to claim 3, characterized in that Before inputting the first training sample into the second neural network model, the method further includes: Get the real master audio clip of the sample song; Downsampling the real master audio segment to obtain a downsampled audio segment; The downsampled audio segment is upsampled to obtain an upsampled audio segment as a first training sample, wherein a sampling rate of the upsampled audio segment is the same as a sampling rate of the real master audio segment.

5. The method according to claim 4, characterized in that The duration of the real master audio segment is greater than the duration of the audio segment used to train the first neural network model.

6. The method according to claim 4 or 5, characterized in that: The first prediction data is a first prediction master audio segment, the second prediction data is a second prediction master audio segment, and the calculating of a first loss value corresponding to the first prediction data includes: Calculating a first time domain loss value and a first frequency domain loss value corresponding to the first predicted master audio segment based on the first predicted master audio segment and the real master audio segment; Adding the first time domain loss value and the first frequency domain loss value to obtain a first loss value corresponding to the first predicted master audio segment; The calculating a second loss value corresponding to the second prediction data includes: Calculating a second time domain loss value and a second frequency domain loss value corresponding to the second predicted master audio segment based on the second predicted master audio segment and the real master audio segment; The second time domain loss value and the second frequency domain loss value are added to obtain a second loss value corresponding to the second predicted master audio segment.

7. The method according to claim 6, characterized in that The first neural network model is an autoencoder AutoEncoder.

8. The method according to any one of claims 3 to 7, characterized in that: The trained second neural network model is used to be deployed on a terminal, and when called by the terminal, the master audio of the second song is generated based on the input audio to be processed of the second song.

9. A computing device, characterized in that The computing device includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for training a neural network model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the method for training a neural network model as described in any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product stores at least one instruction, which is loaded and executed by a processor to implement the operations performed by the method for training a neural network model as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Neural network model training method and device, image processing method and device and terminal equipment

    CN111950723A

  • Neural network training method and device, equipment and storage medium

    CN117010461A

  • Model compression method and apparatus

    US20230229912A1