Cross-platform model deployment method and device and storage medium

By reading pre-trained audio repair models from the general platform in cross-platform model deployment, and modifying and refactoring parameters according to the target model structure of the platform to be transplanted, the problem of low cross-platform deployment efficiency is solved, and the implementation of the Efficient model deployment and resource conservation.

CN118963777AActive Publication Date: 2024-11-15TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411167935.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-15
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

When deploying cross-platform models, the existing technology requires modification and retraining of the model structure, resulting in long time periods, affecting efficiency and resource consumption.

Method used

By reading the pre-trained audio repair model from the general platform, modify the model structure according to the target model structure of the platform to be transplanted, and reconstruct the model parameters through differential comparison to achieve cross-platform deployment without retraining.

Benefits of technology

Improve the efficiency of cross-platform model deployment and save the resource cost consumed by retraining.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963777A_ABST
    Figure CN118963777A_ABST
Patent Text Reader

Abstract

The invention discloses a cross-platform model deployment method and device and a storage medium, and relates to the field of model deployment, and the method comprises the steps: reading a pre-trained universal audio restoration model from a universal platform; modifying a general model structure of the general audio restoration model based on a target model structure required by the to-be-transplanted platform so as to export a modified audio restoration model; the model parameters of the modified audio restoration model are the same as the model parameters of the general audio restoration model; and determining a target model parameter based on the target model structure, reconstructing the model parameter of the modified audio restoration model based on a difference comparison result between the target model parameter and the model parameter of the modified audio restoration model, and deploying the target audio restoration model obtained after reconstruction to the to-be-transplanted platform. According to the method and the device, by modifying the model structure and the model parameters, cross-platform deployment of the audio repair model is realized under the condition that the audio repair model of which the model structure is modified does not need to be retrained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of model deployment, and in particular to a cross-platform model deployment method, device and storage medium. Background Art

[0002] Considering that platforms built on different processors have different characteristics, that is, platforms built on different processors have different deployments for the same model. Therefore, when deploying the same model across platforms, it is usually necessary to first modify the structure of the model to be deployed according to the model deployment characteristics of the transplantation platform, and then retrain the modified model and deploy it to the transplantation platform after training. However, the model training cycle is long, which is likely to have a negative impact on the efficiency of cross-platform deployment of the model. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a cross-platform model deployment method, device and storage medium, which can modify the model structure and model parameters accordingly, and can realize cross-platform deployment of the audio repair model without retraining the audio repair model after the model structure is modified. The specific scheme is as follows:

[0004] In a first aspect, the present application provides a cross-platform model deployment method, comprising:

[0005] Reading a pre-trained universal audio restoration model from a universal platform; the universal platform is a platform built based on the first processor;

[0006] The general model structure of the general audio repair model is modified based on the target model structure required by the to-be-ported platform to derive a modified audio repair model; the to-be-ported platform is a platform built based on the second processor; the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model;

[0007] Determining target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio restoration model based on a difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model;

[0008] The target audio restoration model is deployed to the platform to be transplanted.

[0009] Optionally, the modifying the general model structure of the general audio restoration model based on the target model structure required by the to-be-ported platform includes:

[0010] Modifying the convolution structure in the general model structure of the general audio restoration model accordingly based on the target model convolution structure required by the platform to be transplanted;

[0011] The structural parameters of the convolution structure include any one or a combination of the number of input channels, the number of output channels, the size of the convolution kernel and the convolution step size.

[0012] Optionally, determining the target model parameters based on the target model structure includes:

[0013] Corresponding target model parameters are determined based on the structural parameters of the target model convolution structure; wherein the target model parameters include at least one of a model weight and a model offset.

[0014] Optionally, reconstructing the model parameters of the modified audio restoration model based on the difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain the target audio restoration model includes:

[0015] Performing a difference comparison between the target model parameters and the model parameters of the modified audio restoration model to obtain a difference comparison result;

[0016] Utilizing the difference comparison result and based on the target model parameters, the parameters in the model parameters of the modified audio repair model that are different from the target model parameters are modified, and the parameters in the model parameters of the modified audio repair model that are not different from the target model parameters are retained to obtain the target audio repair model.

[0017] Optionally, before reading the pre-trained universal audio restoration model from the universal platform, the method further includes:

[0018] Building a to-be-trained audio restoration model based on an encoder and a decoder having a convolutional layer, and reconstructing audio having the same sampling rate as the real master audio based on the real master audio;

[0019] Input the reconstructed audio into the audio restoration model to be trained, use the encoder to reduce the dimension of the reconstructed audio to obtain latent variables, and use the decoder to increase the dimension of the latent variables to obtain the restored master audio;

[0020] A training loss between the real master audio and the repaired master audio is determined, and the audio repair model to be trained is trained based on the training loss to obtain the pre-trained universal audio repair model.

[0021] Optionally, the reconstructing the reconstructed audio having the same sampling rate as the real master audio based on the real master audio includes:

[0022] Downsampling the real master audio to obtain downsampled audio; the sampling rate of the downsampled audio is less than the sampling rate of the real master audio;

[0023] Upsampling the downsampled audio to obtain a reconstructed audio having the same sampling rate as the real master audio;

[0024] The highest audio frequency in the reconstructed audio is smaller than the highest audio frequency in the real master audio.

[0025] Optionally, determining a training loss between the real master audio and the restored master audio includes:

[0026] Determine the time domain loss based on the audio difference between the real master audio and the repaired master audio at different times;

[0027] Determine frequency domain loss based on short-time Fourier transform of the real master audio and the restored master audio at different resolutions;

[0028] A training loss between the real master audio and the restored master audio is determined based on the time domain loss and the frequency domain loss.

[0029] Optionally, determining the frequency domain loss based on short-time Fourier transform of the real master audio and the repaired master audio at different resolutions includes:

[0030] Determining different resolutions based on different preset window lengths and preset frame shifts corresponding to the preset window lengths;

[0031] Based on the preset window length and the preset frame shift corresponding to the preset window length, respectively, short-time Fourier transform is performed on the real master audio and the restored master audio to obtain corresponding transform values, and target transform values ​​are determined based on the transform values ​​corresponding to the real master audio and the restored master audio at the same resolution;

[0032] The frequency domain loss is determined based on the target transformation values ​​of the real master audio and the restored master audio at different resolutions.

[0033] In a second aspect, the present application provides an electronic device, including:

[0034] Memory, used to store computer programs;

[0035] A processor is used to execute the computer program to implement the aforementioned cross-platform model deployment method.

[0036] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned cross-platform model deployment method.

[0037] In a fourth aspect, the present application provides a computer program product, including a computer program / instruction, which implements the aforementioned cross-platform model deployment method when executed by a processor.

[0038] In the present application, a pre-trained universal audio repair model is read from a universal platform; the universal platform is a platform built based on a first processor; the universal model structure of the universal audio repair model is modified based on the target model structure required by the platform to be transplanted to derive a modified audio repair model; the platform to be transplanted is a platform built based on a second processor; the model parameters of the modified audio repair model are the same as the model parameters of the universal audio repair model; the target model parameters are determined based on the target model structure, and based on the difference comparison results between the target model parameters and the model parameters of the modified audio repair model, the model parameters of the modified audio repair model are reconstructed to obtain a target audio repair model; and the target audio repair model is deployed to the platform to be transplanted. It can be seen that the present application first modifies the general model structure of the general audio repair model according to the target model structure required by the platform to be transplanted to obtain a modified audio repair model, and then compares the target model parameters corresponding to the target model structure with the model parameters of the modified audio repair model, thereby reconstructing the model parameters of the modified audio repair model according to the difference comparison results to deploy the reconstructed target audio repair model to the platform to be transplanted. In this way, the cross-platform deployment of the audio repair model can be achieved without retraining the audio repair model after the model structure is modified, which not only improves the efficiency of cross-platform model deployment, but also saves the resource cost consumed by retraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A cross-platform model deployment architecture diagram disclosed in this application;

[0041] Figure 2 A flow chart of a cross-platform model deployment method disclosed in this application;

[0042] Figure 3 A comparison diagram of the model structure disclosed in this application;

[0043] Figure 4 A comparison diagram of model parameters disclosed in this application;

[0044] Figure 5 A training flow chart of an audio restoration model disclosed in this application;

[0045] Figure 6 A specific audio restoration model training flow chart disclosed in this application;

[0046] Figure 7 A spectrum diagram of a real master audio disclosed in this application;

[0047] Figure 8 A spectrum diagram of reconstructed audio disclosed in the present application;

[0048] Fig. 9 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] When deploying the same model across platforms, it is usually necessary to first structurally modify the model to be deployed according to the model deployment characteristics of the transplantation platform, and then retrain the modified model and deploy it to the transplantation platform after training. However, the model training cycle is long, which can easily have a negative impact on the efficiency of cross-platform deployment of the model. To this end, the present application provides a cross-platform model deployment method, which can achieve cross-platform deployment of the audio repair model without retraining the audio repair model after the model structure is modified, thereby improving the efficiency of cross-platform deployment of the model by making corresponding modifications to the model structure and model parameters.

[0051] In the cross-platform model deployment method of this application, the system framework used can be specifically referred to in Figure 1 As shown, it may specifically include: a general platform 01 and a platform to be transplanted 02 connected with the general platform 01 , and may also include a third-party platform 03 connected with the general platform 01 and the platform to be transplanted 02 .

[0052] In this application, a pre-trained general audio restoration model is deployed in the general platform 01; and the processors used by the general platform 01 and the platform to be transplanted 02 are different. For example, the general platform 01 can use a CPU (Central Processing Unit), and the platform to be transplanted 02 can use an NPU (Neural Processing Unit).

[0053] In a specific embodiment, the cross-platform model deployment method of the present application is applied to a general platform 01, that is, the general platform 01 is used to execute the steps of the cross-platform model deployment method, including reading a pre-trained general audio repair model from a local, and modifying the general model structure of the general audio repair model accordingly based on the target model structure required by the platform to be transplanted 02, so as to derive a modified audio repair model; wherein the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model; then, based on the target model structure, the target model parameters that can be learned and adjusted inside the model are determined, and based on the difference comparison results between the target model parameters and the model parameters of the modified audio repair model, the model parameters of the modified audio repair model are reconstructed to obtain the target audio repair model; finally, the target audio repair model is deployed to the platform to be transplanted 02.

[0054] In another specific implementation, the cross-platform model deployment method of the present application can also be applied to a third-party platform 03 that establishes a connection with the general platform 01 and the platform to be transplanted 02, that is, the third-party platform 03 is used to perform the steps of the cross-platform model deployment method, including reading a pre-trained general audio repair model from the general platform 01, and modifying the general model structure of the general audio repair model accordingly based on the target model structure required by the platform to be transplanted 02, so as to derive a modified audio repair model; wherein the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model; then, the target model parameters are determined based on the target model structure, and based on the difference comparison results between the target model parameters and the model parameters of the modified audio repair model, the model parameters of the modified audio repair model are reconstructed to obtain the target audio repair model; finally, the target audio repair model is deployed to the platform to be transplanted 02. It should be noted that there is no specific limitation on the processor used by the third-party platform 03, and both a CPU processor and an NPU processor can be used.

[0055] See also Figure 2 As shown, an embodiment of the present invention discloses a cross-platform model deployment method, including:

[0056] Step S11: read a pre-trained universal audio restoration model from a universal platform; the universal platform is a platform built based on the first processor.

[0057] The application subject of the cross-platform model deployment method proposed in the embodiment of the present invention can be either a general platform or a third-party platform; it should be noted that the third-party platform is a platform that communicates with the general platform and the platform to be transplanted. Among them, the general platform is a platform built on the basis of the first processor. Generally, due to the strong compatibility of the CPU processor, the first processor usually uses a CPU processor, but other types of processors can also be used. The platform to be transplanted is a platform built on the basis of the second processor, and the processor used by the platform to be transplanted is different from that used by the general platform. For example, the second processor can use an NPU processor, etc.

[0058] In this embodiment, when the universal audio repair model needs to be deployed across platforms to the platform to be ported, it is necessary to first read the pre-trained universal audio repair model from the universal platform through the universal platform or a third-party platform. It should be noted that the universal audio repair model is used to repair and restore the audio to improve the quality of the audio so that it can achieve an effect close to the master audio.

[0059] Step S12: modify the general model structure of the general audio repair model based on the target model structure required by the platform to be transplanted to derive a modified audio repair model; the platform to be transplanted is a platform built based on the second processor; the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model.

[0060] In this embodiment, since different platforms have different characteristics for the deployment of the same model, that is, different platforms have differences in model structure and model parameters for the deployment of the same model. Therefore, before deploying the universal audio repair model to the platform to be transplanted, it is necessary to first determine the target model structure required by the platform to be transplanted, and then modify the universal model structure of the universal audio repair model accordingly based on the target model structure required by the platform to be transplanted, so as to obtain a modified audio repair model. It should be pointed out that the model structure of the modified audio repair model is the target model structure, and the model parameters of the modified audio repair model are the model parameters of the universal audio repair model, and the model parameters are parameters that can be learned and adjusted within the model, such as model weights, model offsets, etc.

[0061] According to one of the embodiments, the target model convolution structure required by the to-be-ported platform can be determined based on the computational efficiency of different convolution kernels of the to-be-ported platform, and then the convolution structure in the general model structure of the general audio repair model is modified accordingly based on the target model convolution structure required by the to-be-ported platform, thereby obtaining a modified audio repair model. The structural parameters of the convolution structure include any one or a combination of the number of input channels, the number of output channels, the convolution kernel size, and the convolution step size.

[0062] like Figure 3 As shown in the figure, taking the general platform using the CPU processor and the platform to be transplanted using the NPU processor as an example, since the NPU processor has a higher computational efficiency for the convolution kernel with a convolution kernel size of (1,8) than for the convolution kernel with a convolution kernel size of (8,1), the NPU processor is more suitable for using the convolution kernel with a convolution kernel size of (1,8), so that the target model convolution structure required by the platform to be transplanted can be determined. Specifically, the target model convolution structure required by the platform to be transplanted is a two-dimensional convolution Conv2d in the encoder Encoder, and according to Figure 3 From Conv2d(2,64,kernel_size=(1,8),stride=(1,4)), we can see that the structural parameters of the two-dimensional convolution include the number of input channels 2, the number of output channels 64, the convolution kernel size (1,8) and the convolution step size (1,4); the target model convolution structure required by the to-be-ported platform is the two-dimensional transposed convolution ConvTranspose2d in the decoder Decoder. The structural parameters of the two-dimensional transposed convolution include the number of input channels 64, the number of output channels 2, the convolution kernel size (1,8) and the convolution step size (1,4), corresponding to Figure 3 ConvTranspose2d(64,2,kernel_size=(1,8),stride=(1,4)) in the CPU model structure of the general audio restoration model. Based on this, the convolution structure in the CPU model structure of the general audio restoration model can be modified accordingly according to the target model convolution structure required by the ported platform. The convolution structure in the CPU model structure is a two-dimensional convolution Conv2d in the encoder Encoder, and the structural parameters of the two-dimensional convolution include the number of input channels 2, the number of output channels 64, the convolution kernel size (8,1) and the convolution stride (4,1), corresponding to Figure 3 Conv2d(2,64,kernel_size=(8,1),stride=(4,1)) in the CPU model structure; the convolution structure in the decoder is a two-dimensional transposed convolution ConvTranspose2d. The structural parameters of the two-dimensional transposed convolution include the number of input channels 64, the number of output channels 2, the convolution kernel size (8,1) and the convolution stride (4,1), corresponding to Figure 3 ConvTranspose2d(64,2,kernel_size=(8,1),stride=(4,1)) in.

[0063] Step S13: determining target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio restoration model based on a difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model.

[0064] In this embodiment, since there is a correlation between the model structure and the model parameters, the corresponding target model parameters can be determined based on the target model structure required by the platform to be transplanted, and then the target model parameters and the model parameters of the modified audio repair model are compared to obtain the difference comparison results, and the model parameters of the modified audio repair model are reconstructed based on the difference comparison results to obtain the target audio repair model.

[0065] According to one of the embodiments, the corresponding target model parameters are determined based on the structural parameters of the target model convolution structure required by the platform to be transplanted. Specifically, the model weight weight is determined according to the number of input channels, the number of output channels and the convolution kernel size in the structural parameters of the target model convolution structure, and the model offset bias is determined according to the number of output channels in the structural parameters of the target model convolution structure.

[0066] like Figure 3 and Figure 4As shown in the figure, taking the NPU processor used in the platform to be ported as an example, the target model convolution structure required by the platform to be ported is Conv2d(2,64,kernel_size=(1,8),stride=(1,4)) in the encoder Encoder; accordingly, the target model parameters required by the platform to be ported are encoder.0.0.weighttorch.Size([64,2,1,8]) and encoder.0.0.bias torch.Size(

[64] ) in the encoder Encoder, that is, in the 0.0 version of the encoder, the dimension size of the model weight tensor is 4, and the first dimension is 64, the second dimension is 2, the third dimension is 1, and the fourth dimension is 8; in the 0.0 version of the encoder, the dimension size of the model bias tensor is 1, and the first dimension is 64. Furthermore, the target model convolution structure required by the platform to be ported is ConvTranspose2d(64,2,kernel_size=(1,8),stride=(1,4)) in the decoder Decoder; correspondingly, the target model parameters required by the platform to be ported are decoder4.3.weight torch.Size([64,2,1,8]) and decoder4.3.biastorch.Size([2]) in the decoder Decoder, that is, in the 4.3 version of the decoder, the dimension size of the model weight tensor is 4, and the first dimension is 64, the second dimension is 2, the third dimension is 1, and the fourth dimension is 8; in the 4.3 version of the decoder, the dimension size of the model bias tensor is 1, and the first dimension is 2.

[0067] The target model parameters and the model parameters of the modified audio restoration model are subjected to a difference comparison to obtain a difference comparison result. Then, based on the difference comparison result and the target model parameters, the model parameters of the modified audio restoration model that differ from the target model parameters are modified, and the model parameters of the modified audio restoration model that do not differ from the target model parameters are retained, thereby obtaining a target audio restoration model.

[0068] Specifically, if the difference comparison result indicates that there are differences between the target model parameters and the model parameters of the modified audio repair model, the model parameters to be adjusted that are different from the model parameters of the modified audio repair model can be determined from the target model parameters, and the model parameters to be adjusted can be used to make corresponding modifications to the model parameters of the modified audio repair model that are different from the target model parameters, and the parameters in the model parameters of the modified audio repair model that are not different from the target model parameters can be directly retained, thereby obtaining the target audio repair model.

[0069] Step S14: deploying the target audio restoration model to the platform to be transplanted.

[0070] In this embodiment, after obtaining the target audio restoration model with modified model structure and model parameters, the target audio restoration model can be deployed to the to-be-ported platform built on the second processor. In this way, the step of retraining the model with modified model structure is avoided, and the efficiency of cross-platform deployment of the model is improved.

[0071] It can be seen that the present application first modifies the general model structure of the general audio repair model according to the target model structure required by the platform to be transplanted to obtain a modified audio repair model, and then compares the target model parameters corresponding to the target model structure with the model parameters of the modified audio repair model, thereby reconstructing the model parameters of the modified audio repair model according to the difference comparison results to deploy the reconstructed target audio repair model to the platform to be transplanted. In this way, the cross-platform deployment of the audio repair model can be achieved without retraining the audio repair model after the model structure is modified, which not only improves the efficiency of cross-platform model deployment, but also saves the resource cost consumed by retraining.

[0072] Based on the previous embodiment, it can be seen that the previous embodiment describes how to transplant the general audio restoration model from the general platform to the platform to be transplanted. Next, this embodiment will elaborate on how to train the general audio restoration model. Figure 5 As shown, the embodiment of the present invention discloses a training process of a general audio restoration model, including:

[0073] Step S21: construct an audio restoration model to be trained based on an encoder and a decoder having a convolutional layer, and reconstruct audio having the same sampling rate as the real master audio based on the real master audio.

[0074] In this embodiment, an audio restoration model to be trained is constructed based on an autoencoder (AE), wherein the autoencoder includes an encoder and a decoder having a convolutional layer. Real master audio is obtained as training data, and reconstructed audio having the same sampling rate as the real master audio is reconstructed based on the real master audio.

[0075] Specifically, Figure 6As shown, the real master audio is downsampled to obtain downsampled audio; wherein the sampling rate of the downsampled audio is less than the sampling rate of the real master audio. For example, the sampling rate of the real master audio is 192kHz, and the sampling rate of the downsampled audio obtained by downsampling is 44.1kHz. The downsampled audio is then upsampled to obtain a reconstructed audio with the same sampling rate as the real master audio. Since the high-frequency spectrum is lost during the downsampling and upsampling processes, although the reconstructed audio has the same sampling rate as the real master audio, the highest audio in the reconstructed audio is lower than the highest audio in the real master audio. As shown Figure 7 and Figure 8 As shown, Figure 7 This is the spectrum of the real master audio. Figure 8 To reconstruct the audio spectrum, it can be found that only the low-frequency spectrum is retained in the reconstructed audio, and the high-frequency spectrum is basically lost; and the audio restoration model can be used to restore the high-frequency spectrum in the audio.

[0076] Step S22: input the reconstructed audio into the audio restoration model to be trained, use the encoder to reduce the dimension of the reconstructed audio to obtain latent variables, and use the decoder to increase the dimension of the latent variables to obtain the restored master audio.

[0077] In this embodiment, Figure 6 As shown, the reconstructed audio is used as the input of the audio restoration model to be trained, and the encoder in the audio restoration model to be trained is used to reduce the dimension of the reconstructed audio to obtain latent variables in a low-dimensional space, and the decoder in the audio restoration model to be trained is used to increase the dimension of the latent variables to obtain a repaired master audio in a high-dimensional space.

[0078] Step S23: determine the training loss between the real master audio and the repaired master audio, and train the audio repair model to be trained based on the training loss to obtain a pre-trained universal audio repair model.

[0079] In this embodiment, a preset loss function is used to calculate the training loss between the real master audio and the repaired master audio, and the audio repair model to be trained is back-propagated based on the training loss to optimize the training of the audio repair model to obtain a pre-trained general audio repair model.

[0080] Specifically, for the training loss between the real master audio and the restored master audio, firstly, the time domain loss is determined based on the audio difference between the real master audio and the restored master audio at different times; then the frequency domain loss is determined based on the short time Fourier transform (STFT) of the real master audio and the restored master audio at different resolutions; finally, the training loss between the real master audio and the restored master audio is determined by combining the time domain loss and the frequency domain loss. ;in, represents the training loss, represents the time domain loss, represents the frequency domain loss, Represents the scaling factor of time domain loss to frequency domain loss.

[0081] For the time domain loss, the audio difference between the real master audio and the restored master audio at different times is calculated, and the audio difference at different times is accumulated to obtain the time domain loss between the real master audio and the restored master audio. ;in, represents the time domain loss, t represents the time, Represents true master audio, Indicates repairing the master audio. represents the 1st-order norm.

[0082] Among them, for frequency domain loss, different resolutions are determined based on different preset window lengths and preset frame shifts corresponding to each preset window length; for example, the preset window length can be 4096, 2048, 1024, etc., and the preset frame shift is generally set to one-fourth of the preset window length. The preset frame shift corresponding to the preset window length 4096 can be 1024, the preset frame shift corresponding to the preset window length 2048 can be 512, and the preset frame shift corresponding to the preset window length 1024 can be 256. Then, based on the preset window length and the preset frame shift corresponding to the preset window length, the real master audio and the repaired master audio are short-time Fourier transformed to obtain corresponding transformation values, and the target transformation value is determined based on the transformation values ​​corresponding to the real master audio and the repaired master audio at the same resolution. Finally, the frequency domain loss is determined based on the target transformation values ​​of the real master audio and the repaired master audio at different resolutions. The formula involved in the frequency domain loss is as follows:

[0083] ;

[0084] in, represents the frequency domain loss, M represents the total number of resolutions of different resolutions, STFT ( ) represents the short-time Fourier transform function, represents the 2nd-order norm, Indicates the target transformation value at the same resolution, and Represents two intermediate parameters used to calculate the target transform value at the same resolution.

[0085] It can be seen that this embodiment can constrain the consistency of energy and distribution between the repaired master audio and the real master audio by adopting time domain loss, thereby ensuring consistency in the time domain; and, this embodiment can restore the high-frequency spectrum in the audio and repair the low-frequency spectrum in the audio by adopting frequency domain loss, so that the audio repair model learns in the direction of restoring the master. Furthermore, this embodiment determines the final training loss between the real master audio and the repaired master audio based on time domain loss and frequency domain loss, and optimizes the model training using the final training loss, thereby ensuring the audio repair effect of the model. In addition, the audio repair model is a model constructed based on an encoder and a decoder with a convolutional layer, so the model structure and model parameters can be modified to achieve cross-platform deployment of the audio repair model without retraining the audio repair model after the model structure is modified, thereby improving the efficiency of cross-platform model deployment.

[0086] Furthermore, the present application also discloses an electronic device. Fig. 9 This is a structural diagram of an electronic device 10 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0087] Fig. 9 A schematic diagram of the structure of an electronic device 10 provided in an embodiment of the present application. The electronic device 10 may specifically include: at least one processor 11, at least one memory 12, a power supply 13, a communication interface 14, an input / output interface 15, and a communication bus 16. The memory 12 is used to store a computer program, which is loaded and executed by the processor 11 to implement the relevant steps in the cross-platform model deployment method disclosed in any of the aforementioned embodiments. In addition, the electronic device 10 in this embodiment may specifically be an electronic computer.

[0088] In this embodiment, the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0089] In addition, the memory 12, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 121, a computer program 122, etc., and the storage method can be temporary storage or permanent storage.

[0090] The operating system 121 is used to manage and control the hardware devices on the electronic device 10 and the computer program 122, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the cross-platform model deployment method performed by the electronic device 10 disclosed in any of the aforementioned embodiments, the computer program 122 can further include a computer program that can be used to complete other specific tasks.

[0091] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the cross-platform model deployment method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0092] Furthermore, the present application also discloses a computer program product, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the cross-platform model deployment method disclosed above. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0094] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0095] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0096] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0097] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cross-platform model deployment method, characterized in that: include: Reading a pre-trained universal audio restoration model from a universal platform; the universal platform is a platform built based on the first processor; Modifying the general model structure of the general audio repair model based on the target model structure required by the to-be-ported platform to derive a modified audio repair model; The platform to be transplanted is a platform built based on the second processor; The model parameters of the modified audio restoration model are the same as the model parameters of the general audio restoration model; Determining target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio restoration model based on a difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model; The target audio restoration model is deployed to the platform to be transplanted.

2. The cross-platform model deployment method according to claim 1, characterized in that: The modifying of the general model structure of the general audio repair model based on the target model structure required by the to-be-ported platform includes: Modifying the convolution structure in the general model structure of the general audio restoration model accordingly based on the target model convolution structure required by the platform to be transplanted; The structural parameters of the convolution structure include any one or a combination of the number of input channels, the number of output channels, the size of the convolution kernel and the convolution step size.

3. The cross-platform model deployment method according to claim 2, characterized in that: The determining of the target model parameters based on the target model structure includes: Corresponding target model parameters are determined based on the structural parameters of the target model convolution structure; wherein the target model parameters include at least one of a model weight and a model offset.

4. The cross-platform model deployment method according to claim 1, characterized in that: The step of reconstructing the model parameters of the modified audio restoration model based on the difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain the target audio restoration model includes: Performing a difference comparison between the target model parameters and the model parameters of the modified audio restoration model to obtain a difference comparison result; Utilizing the difference comparison result and based on the target model parameters, the parameters in the model parameters of the modified audio repair model that are different from the target model parameters are modified, and the parameters in the model parameters of the modified audio repair model that are not different from the target model parameters are retained to obtain the target audio repair model.

5. The cross-platform model deployment method according to any one of claims 1 to 4, characterized in that: Before reading the pre-trained universal audio restoration model from the universal platform, the method further includes: Building a to-be-trained audio restoration model based on an encoder and a decoder having a convolutional layer, and reconstructing audio having the same sampling rate as the real master audio based on the real master audio; Input the reconstructed audio into the audio restoration model to be trained, use the encoder to reduce the dimension of the reconstructed audio to obtain latent variables, and use the decoder to increase the dimension of the latent variables to obtain the restored master audio; A training loss between the real master audio and the repaired master audio is determined, and the audio repair model to be trained is trained based on the training loss to obtain the pre-trained universal audio repair model.

6. The cross-platform model deployment method according to claim 5, characterized in that: The reconstructing audio having the same sampling rate as the real master audio based on the real master audio comprises: Downsampling the real master audio to obtain downsampled audio; the sampling rate of the downsampled audio is less than the sampling rate of the real master audio; Upsampling the downsampled audio to obtain a reconstructed audio having the same sampling rate as the real master audio; The highest audio frequency in the reconstructed audio is smaller than the highest audio frequency in the real master audio.

7. The cross-platform model deployment method according to claim 5, characterized in that: The determining of the training loss between the real master audio and the repaired master audio comprises: Determine the time domain loss based on the audio difference between the real master audio and the repaired master audio at different times; Determine frequency domain loss based on short-time Fourier transform of the real master audio and the restored master audio at different resolutions; A training loss between the real master audio and the restored master audio is determined based on the time domain loss and the frequency domain loss.

8. The cross-platform model deployment method according to claim 7, characterized in that: The determining of frequency domain loss based on short-time Fourier transform of the real master audio and the repaired master audio at different resolutions includes: Determining different resolutions based on different preset window lengths and preset frame shifts corresponding to the preset window lengths; Based on the preset window length and the preset frame shift corresponding to the preset window length, respectively, short-time Fourier transform is performed on the real master audio and the restored master audio to obtain corresponding transform values, and target transform values ​​are determined based on the transform values ​​corresponding to the real master audio and the restored master audio at the same resolution; The frequency domain loss is determined based on the target transformation values ​​of the real master audio and the restored master audio at different resolutions.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the cross-platform model deployment method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the cross-platform model deployment method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Deployment method and architecture for inference cross-platform of neural network model

    CN112101529A

  • Neural network model automatic adaptation method and device

    CN113128682A

  • NPU neural network model deployment method and device based on TVM

    CN114721670A

  • Automated fine-tuning and deployment of pre-trained deep learning models

    US20220398462A1