A cross-platform model deployment method, device and storage medium

Abstract

The application discloses a cross-platform model deployment method and device and a storage medium, relates to the field of model deployment, and comprises the following steps: reading a pre-trained general audio repair model from a general platform; modifying a general model structure of the general audio repair model based on a target model structure required by a to-be-ported platform to derive a modified audio repair model; the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model; determining target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio repair model based on a difference comparison result between the target model parameters and the model parameters of the modified audio repair model; and deploying a target audio repair model obtained after reconstruction to the to-be-ported platform. The application realizes cross-platform deployment of the audio repair model by modifying the model structure and the model parameters, without the need of retraining the audio repair model after modification of the model structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of model deployment, in particular to a cross-platform model deployment method and device and storage medium. BACKGROUND

[0002] It is considered that platforms built based on different processors have different characteristics, that is, the deployment of the same model on platforms built based on different processors is different. Therefore, when deploying the same model across platforms, the model to be deployed usually needs to be structurally modified according to the model deployment characteristics of the transplantation platform, and then the modified model needs to be retrained and deployed to the transplantation platform after training is completed. However, the time period of model training is long, which can negatively affect the efficiency of model cross-platform deployment. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a cross-platform model deployment method, device and storage medium, which can modify the model structure and model parameters accordingly, and can realize the cross-platform deployment of the audio repair model without the need to retrain the audio repair model after modifying the model structure. The specific scheme is as follows:

[0004] In a first aspect, the present application provides a cross-platform model deployment method, comprising:

[0005] reading a pre-trained general audio repair model from a general platform; the general platform is a platform built based on a first processor;

[0006] modifying the general model structure of the general audio repair model based on a target model structure required by a platform to be transplanted to derive a modified audio repair model; the platform to be transplanted is a platform built based on a second processor; the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model;

[0007] determining target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio repair model based on the difference comparison result between the target model parameters and the model parameters of the modified audio repair model to obtain a target audio repair model;

[0008] deploying the target audio repair model to the platform to be transplanted.

[0009] Optionally, the modification of the general model structure of the general audio repair model based on the target model structure required by the platform to be transplanted comprises:

[0010] modifying a convolutional structure in a general model structure of the general audio restoration model based on a target model convolutional structure required by a target platform to be transplanted;

[0011] The structure parameters of the convolutional structure include any one or a combination of input channel number, output channel number, convolution kernel size, and convolution step length.

[0012] Optionally, the determining of the target model parameters based on the target model structure comprises:

[0013] determining corresponding target model parameters based on the structure parameters of the target model convolutional structure; wherein the target model parameters include at least one of model weights and model biases.

[0014] Optionally, the reconstructing of the model parameters of the modified audio restoration model based on the difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model comprises:

[0015] performing a difference comparison between the target model parameters and the model parameters of the modified audio restoration model to obtain a difference comparison result.

[0016] modifying, based on the difference comparison result and the target model parameters, parameters of the model parameters of the modified audio restoration model that are different from the target model parameters, and retaining parameters of the model parameters of the modified audio restoration model that are not different from the target model parameters, to obtain a target audio restoration model.

[0017] Optionally, before the reading of the pre-trained general audio restoration model from the general platform, the method further comprises:

[0018] constructing a to-be-trained audio restoration model based on an encoder and a decoder having convolutional layers, and reconstructing a reconstructed audio having the same sampling rate as a real master audio based on the real master audio;

[0019] inputting the reconstructed audio into the to-be-trained audio restoration model to obtain hidden variables by dimension reduction of the reconstructed audio using the encoder, and to obtain a restored master audio by dimension increase of the hidden variables using the decoder;

[0020] determining a training loss between the real master audio and the restored master audio, and training the to-be-trained audio restoration model based on the training loss to obtain the pre-trained general audio restoration model.

[0021] Optionally, the reconstructing of the reconstructed audio having the same sampling rate as the real master audio based on the real master audio comprises:

[0022] down-sampling the real master audio to obtain a down-sampled audio; a sampling rate of the down-sampled audio is less than a sampling rate of the real master audio;

[0023] up-sampling the down-sampled audio to obtain a reconstructed audio having the same sampling rate as the real master audio;

[0024] wherein a highest audio in the reconstructed audio is less than a highest audio in the real master audio.

[0025] Optionally, the determining the training loss between the real master audio and the repaired master audio comprises:

[0026] determining a time domain loss based on an audio difference between the real master audio and the repaired master audio at different times;

[0027] determining a frequency domain loss based on a short-time Fourier transform of the real master audio and the repaired master audio at different resolutions;

[0028] determining the training loss between the real master audio and the repaired master audio based on the time domain loss and the frequency domain loss.

[0029] Optionally, the determining the frequency domain loss based on the short-time Fourier transform of the real master audio and the repaired master audio at different resolutions comprises:

[0030] determining different resolutions based on different preset window lengths and preset frame shifts respectively corresponding to the preset window lengths;

[0031] performing short-time Fourier transform on the real master audio and the repaired master audio respectively based on the preset window lengths and the preset frame shifts respectively corresponding to the preset window lengths to obtain corresponding transform values, and determining a target transform value based on the transform values respectively corresponding to the real master audio and the repaired master audio at the same resolution;

[0032] determining the frequency domain loss based on the target transform value of the real master audio and the repaired master audio at different resolutions.

[0033] In a second aspect, the present application provides an electronic device comprising:

[0034] a memory for saving a computer program;

[0035] a processor for executing the computer program to implement the foregoing cross-platform model deployment method.

[0036] In a third aspect, the present application provides a computer readable storage medium for storing a computer program, which, when executed by a processor, implements the foregoing cross-platform model deployment method.

[0037] In a fourth aspect, the present application provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implements the foregoing cross-platform model deployment method.

[0038] In the present application, a pre-trained general audio restoration model is read from a general platform; the general platform is a platform constructed based on a first processor; the general model structure of the general audio restoration model is modified based on a target model structure required by a to-be-ported platform to derive a modified audio restoration model; the to-be-ported platform is a platform constructed based on a second processor; the model parameters of the modified audio restoration model are the same as the model parameters of the general audio restoration model; target model parameters are determined based on the target model structure, and the model parameters of the modified audio restoration model are reconstructed based on a difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model; and the target audio restoration model is deployed to the to-be-ported platform. As can be seen, the present application first modifies the general model structure of the general audio restoration model according to the target model structure required by the to-be-ported platform to obtain the modified audio restoration model, and then compares the target model parameters corresponding to the target model structure with the model parameters of the modified audio restoration model in terms of difference, so as to reconstruct the model parameters of the modified audio restoration model according to the difference comparison result, and deploy the target audio restoration model obtained after reconstruction to the to-be-ported platform. In this way, the cross-platform deployment of the audio restoration model can be realized without the need of retraining the audio restoration model after modification of the model structure, which not only improves the efficiency of cross-platform model deployment, but also saves the resource cost consumed by retraining. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0040] Figure 1 A cross-platform model deployment architecture disclosed by the present application;

[0041] Figure 2 A cross-platform model deployment method flowchart disclosed by the present application;

[0042] Figure 3 A model structure comparison chart disclosed by the present application;

[0043] Figure 4 A model parameter comparison chart disclosed by the present application;

[0044] Figure 5 A training flowchart of an audio repair model disclosed by the present application;

[0045] Figure 6 A specific audio repair model training flowchart disclosed by the present application;

[0046] Figure 7 A spectrum chart of a real master audio disclosed by the present application;

[0047] Figure 8 A spectrum chart of a reconstructed audio disclosed by the present application;

[0048] Figure 9 A structure chart of an electronic device disclosed by the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] When deploying the same model across platforms, it is usually necessary to modify the structure of the model to be deployed according to the model deployment characteristics of the transplantation platform, and then retrain the modified model, and deploy it to the transplantation platform after the training is completed. However, the time period of model training is long, which can negatively affect the efficiency of model cross-platform deployment. Therefore, the present application provides a cross-platform model deployment method, which can realize cross-platform deployment of the audio repair model without retraining the audio repair model after modifying the model structure, and improve the efficiency of model cross-platform deployment.

[0051] In the cross-platform model deployment method of the present application, the system framework used can be seen from Figure 1 , which can specifically include a general platform 01 and a to-be-transplanted platform 02 connected with the general platform 01, and can further include a third-party platform 03 connected with the general platform 01 and the to-be-transplanted platform 02.

[0052] In the present application, a pre-trained general audio repair model is deployed in the general platform 01; and the general platform 01 and the processor used by the to-be-ported platform 02 are different. For example, the general platform 01 can use a CPU (Central Processing Unit), and the to-be-ported platform 02 can use an NPU (Neural Processing Unit).

[0053] In one specific embodiment, the cross-platform model deployment method of the present application is applied to the general platform 01, that is, the general platform 01 is used to perform the steps of the cross-platform model deployment method, including reading a pre-trained general audio repair model from the local, and modifying the general model structure of the general audio repair model based on the target model structure required by the to-be-ported platform 02, to export a modified audio repair model; wherein the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model; then determining the target model parameters that can be learned and adjusted inside the model based on the target model structure, and reconstructing the model parameters of the modified audio repair model based on the difference comparison result between the target model parameters and the model parameters of the modified audio repair model, to obtain a target audio repair model; finally, deploying the target audio repair model to the to-be-ported platform 02.

[0054] In another specific embodiment, the cross-platform model deployment method of the present application can also be applied to a third-party platform 03 connected with the general platform 01 and the to-be-ported platform 02, that is, the third-party platform 03 is used to perform the steps of the cross-platform model deployment method, including reading a pre-trained general audio repair model from the general platform 01, and modifying the general model structure of the general audio repair model based on the target model structure required by the to-be-ported platform 02, to export a modified audio repair model; wherein the model parameters of the modified audio repair model are the same as the model parameters of the general audio repair model; then determining the target model parameters based on the target model structure, and reconstructing the model parameters of the modified audio repair model based on the difference comparison result between the target model parameters and the model parameters of the modified audio repair model, to obtain a target audio repair model; finally, deploying the target audio repair model to the to-be-ported platform 02. It should be noted that the processor used by the third-party platform 03 is not specifically limited, which can be a CPU processor or an NPU processor.

[0055] Referring to Figure 2 The embodiment of the present application discloses a cross-platform model deployment method, which comprises:

[0056] Step S11, reading a pre-trained general audio repair model from a general platform; the general platform is a platform constructed based on a first processor.

[0057] The application subject of the cross-platform model deployment method proposed in the embodiments of the present application can be a general platform or a third-party platform. It should be noted that the third-party platform is a platform that is communicatively connected with the general platform and the platform to be transplanted. The general platform is a platform built based on a first processor. Generally, the first processor is usually a CPU processor due to the strong compatibility of the CPU processor, but other types of processors can also be selected. The platform to be transplanted is a platform built based on a second processor, and the platform to be transplanted is different from the processor used by the general platform, for example, the second processor can be an NPU processor.

[0058] In the embodiments, when it is needed to deploy the general audio restoration model to the platform to be transplanted, the pre-trained general audio restoration model needs to be read from the general platform through the general platform or the third-party platform. It should be noted that the general audio restoration model is used to restore and restore the audio to improve the quality of the audio so that it can achieve an effect close to the master tape audio.

[0059] In step S12, the general model structure of the general audio restoration model is modified based on the target model structure required by the platform to be transplanted to derive a modified audio restoration model; the platform to be transplanted is a platform built based on a second processor; the model parameters of the modified audio restoration model are the same as the model parameters of the general audio restoration model.

[0060] In the embodiments, different platforms have different characteristics for deploying the same model, that is, different platforms have differences in model structure and model parameters for deploying the same model. Therefore, before deploying the general audio restoration model to the platform to be transplanted, the target model structure required by the platform to be transplanted needs to be determined, and then the general model structure of the general audio restoration model is modified based on the target model structure required by the platform to be transplanted to obtain a modified audio restoration model. It should be noted that the model structure of the modified audio restoration model is the target model structure, the model parameters of the modified audio restoration model are the model parameters of the general audio restoration model, and the model parameters are the parameters inside the model that can be learned and adjusted, such as model weight, model offset, etc.

[0061] According to one of the embodiments, the target model convolution structure required by the platform to be transplanted can be determined according to the calculation efficiency of different convolution kernels of the platform to be transplanted, and then the convolution structure in the general model structure of the general audio restoration model is modified based on the target model convolution structure required by the platform to be transplanted to obtain a modified audio restoration model. The structure parameters of the convolution structure include any one or a combination of several of the input channel number, the output channel number, the convolution kernel size, and the convolution step.

[0062] As Figure 3 shown, taking the general platform using CPU processor and the to-be-ported platform using NPU processor as an example, since the calculation efficiency of the NPU processor on the convolution kernel with the size of (1, 8) is higher than that on the convolution kernel with the size of (8, 1), the NPU processor is more suitable to use the convolution kernel with the size of (1, 8), so that the target model convolution structure required by the to-be-ported platform can be determined. Specifically, the target model convolution structure required by the to-be-ported platform is two-dimensional convolution Conv2d in the encoder Encoder, and according to Figure 3 Conv2d(2, 64, kernel_size=(1, 8), stride=(1, 4)) in the encoder Encoder, the structure parameters of the two-dimensional convolution include the input channel number 2, the output channel number 64, the convolution kernel size (1, 8) and the convolution stride (1, 4); the target model convolution structure required by the to-be-ported platform is two-dimensional transposed convolution ConvTranspose2d in the decoder Decoder, and the structure parameters of the two-dimensional transposed convolution include the input channel number 64, the output channel number 2, the convolution kernel size (1, 8) and the convolution stride (1, 4), corresponding to Figure 3 ConvTranspose2d(64, 2, kernel_size=(1, 8), stride=(1, 4)) in the decoder Decoder. Accordingly, the convolution structure in the CPU model structure of the general audio inpainting model can be modified according to the target model convolution structure required by the to-be-ported platform. Specifically, the convolution structure in the CPU model structure is two-dimensional convolution Conv2d in the encoder Encoder, and the structure parameters of the two-dimensional convolution include the input channel number 2, the output channel number 64, the convolution kernel size (8, 1) and the convolution stride (4, 1), corresponding to Figure 3 Conv2d(2, 64, kernel_size=(8, 1), stride=(4, 1)) in the encoder Encoder; the convolution structure in the CPU model structure is two-dimensional transposed convolution ConvTranspose2d in the decoder Decoder, and the structure parameters of the two-dimensional transposed convolution include the input channel number 64, the output channel number 2, the convolution kernel size (8, 1) and the convolution stride (4, 1), corresponding to Figure 3 ConvTranspose2d(64, 2, kernel_size=(8, 1), stride=(4, 1)) in the decoder Decoder.

[0063] In step S13, a target model parameter is determined based on the target model structure, and the model parameter of the modified audio inpainting model is reconstructed based on a difference comparison result between the target model parameter and the model parameter of the modified audio inpainting model, to obtain a target audio inpainting model.

[0064] In the embodiment, since the model structure is associated with the model parameter, the target model parameter can be determined based on the target model structure required by the to-be-ported platform, and then the target model parameter and the model parameter of the modified audio inpainting model are compared in difference to obtain a difference comparison result, and the model parameter of the modified audio inpainting model is reconstructed based on the difference comparison result, so as to obtain the target audio inpainting model.

[0065] According to one of the embodiments, the target model parameter is determined based on the structure parameter of the target model convolution structure required by the to-be-ported platform. Specifically, the model weight weight is determined according to the input channel number, the output channel number and the convolution kernel size in the structure parameter of the target model convolution structure, and the model offset bias is determined according to the output channel number in the structure parameter of the target model convolution structure.

[0066] As Figure 3 and Figure 4As shown, taking the NPU processor as an example, the target model convolution structure required by the to-be-ported platform is Conv2d(2, 64, kernel_size=(1, 8), stride=(1, 4)) in the encoder Encoder; correspondingly, the target model parameters required by the to-be-ported platform are encoder.0.0.weight torch.Size([64, 2, 1, 8]) and encoder.0.0.bias torch.Size(

[64] ) in the encoder Encoder, that is, the dimension size of the model weight tensor in the 0.0 version of the encoder is 4, and the first dimension is 64, the second dimension is 2, the third dimension is 1, and the fourth dimension is 8; the dimension size of the model bias tensor in the 0.0 version of the encoder is 1, and the first dimension is 64. Further, the target model convolution structure required by the to-be-ported platform is ConvTranspose2d(64, 2, kernel_size=(1, 8), stride=(1, 4)) in the decoder Decoder; correspondingly, the target model parameters required by the to-be-ported platform are decoder4.3.weight torch.Size([64, 2, 1, 8]) and decoder4.3.bias torch.Size([2]) in the decoder Decoder, that is, the dimension size of the model weight tensor in the 4.3 version of the decoder is 4, and the first dimension is 64, the second dimension is 2, the third dimension is 1, and the fourth dimension is 8; the dimension size of the model bias tensor in the 4.3 version of the decoder is 1, and the first dimension is 2.

[0067] The target model parameters and the model parameters of the modified audio restoration model are compared in difference to obtain a difference comparison result. Then, the model parameters of the modified audio restoration model that are different from the target model parameters are modified based on the target model parameters and the difference comparison result, and the model parameters of the modified audio restoration model that are not different from the target model parameters are retained, so as to obtain the target audio restoration model.

[0068] Specifically, if the difference comparison result indicates that there is a difference between the target model parameters and the model parameters of the modified audio restoration model, the to-be-adjusted model parameters that are different from the model parameters of the modified audio restoration model can be determined from the target model parameters, and the model parameters of the modified audio restoration model that are different from the target model parameters are modified using the to-be-adjusted model parameters, and the model parameters of the modified audio restoration model that are not different from the target model parameters are directly retained, so as to obtain the target audio restoration model.

[0069] Step S14, deploying the target audio inpainting model to the to-be-ported platform.

[0070] In this embodiment, after obtaining the target audio inpainting model with modified model structure and model parameters, the target audio inpainting model can be deployed to the to-be-ported platform based on the second processor. In this way, the step of retraining the model with modified model structure is avoided, and the efficiency of cross-platform deployment of the model is improved.

[0071] As can be seen, the present application modifies the general model structure of the general audio inpainting model according to the target model structure required by the to-be-ported platform to obtain a modified audio inpainting model, and then compares the target model parameters corresponding to the target model structure and the model parameters of the modified audio inpainting model, and reconstructs the model parameters of the modified audio inpainting model according to the difference comparison result, so as to deploy the target audio inpainting model obtained after reconstruction to the to-be-ported platform. In this way, cross-platform deployment of the audio inpainting model can be realized without retraining the audio inpainting model with modified model structure, which not only improves the efficiency of cross-platform model deployment, but also saves the resource cost consumed by retraining.

[0072] Based on the previous embodiment, the previous embodiment describes how to port the general audio inpainting model from the general platform to the to-be-ported platform. Next, this embodiment will describe in detail how to train the general audio inpainting model. Referring to FIG. 8, Figure 5 As shown in FIG. 8, the embodiment of the present application discloses a training process of a general audio inpainting model, which comprises the following steps:

[0073] Step S21, constructing a to-be-trained audio inpainting model based on an encoder and a decoder with convolutional layers, and reconstructing a reconstructed audio with the same sampling rate as the real master audio based on the real master audio.

[0074] In this embodiment, the to-be-trained audio inpainting model is constructed based on an autoencoder (AE). The autoencoder includes an encoder and a decoder with convolutional layers. The real master audio is obtained as training data, and a reconstructed audio with the same sampling rate as the real master audio is reconstructed based on the real master audio.

[0075] Specifically, as shown in FIG. 9, Figure 6As shown, the real master audio is down-sampled to obtain down-sampled audio; wherein the sampling rate of the down-sampled audio is less than the sampling rate of the real master audio. For example, the sampling rate of the real master audio is 192 kHz, and the sampling rate of the down-sampled audio obtained by down-sampling is 44.1 kHz. Then the down-sampled audio is up-sampled to obtain reconstructed audio with the same sampling rate as the real master audio. Since the loss of high-frequency spectrum will be caused in the process of down-sampling and up-sampling, although the reconstructed audio has the same sampling rate as the real master audio, the highest frequency in the reconstructed audio is less than that in the real master audio. As shown, Figure 7 and Figure 8 as shown, Figure 7 is a spectrum diagram of the real master audio, Figure 8 is a spectrum diagram of the reconstructed audio, it can be found that only low-frequency spectrum is retained in the reconstructed audio, and high-frequency spectrum is basically lost; and the high-frequency spectrum in the audio can be restored by the audio restoration model.

[0076] Step S22, input the reconstructed audio into the to-be-trained audio restoration model, to obtain hidden variables by dimension reduction of the reconstructed audio by using the encoder, and to obtain restored master audio by dimension increase of the hidden variables by using the decoder.

[0077] In this embodiment, as shown in Figure 6 , the reconstructed audio is taken as the input of the to-be-trained audio restoration model, to obtain hidden variables in the low-dimensional space by dimension reduction of the reconstructed audio by using the encoder in the to-be-trained audio restoration model, and to obtain restored master audio in the high-dimensional space by dimension increase of the hidden variables by using the decoder in the to-be-trained audio restoration model.

[0078] Step S23, determine the training loss between the real master audio and the restored master audio, and train the to-be-trained audio restoration model based on the training loss to obtain the pre-trained general audio restoration model.

[0079] In this embodiment, the preset loss function is used to calculate the training loss between the real master audio and the restored master audio, and the to-be-trained audio restoration model is back-propagated based on the training loss to optimize and train the to-be-trained audio restoration model, so as to obtain the pre-trained general audio restoration model.

[0080] Specifically, for the training loss between the real master audio and the repaired master audio, first, a time domain loss is determined based on the audio difference between the real master audio and the repaired master audio at different times; then a frequency domain loss is determined based on the short time Fourier transform (STFT) of the real master audio and the repaired master audio at different resolutions; and finally, the training loss between the real master audio and the repaired master audio is determined in combination with the time domain loss and the frequency domain loss. That is ; wherein, represents the training loss, represents the time domain loss, represents the frequency domain loss, represents a proportion factor of the time domain loss and the frequency domain loss.

[0081] Wherein, for the time domain loss, the audio difference between the real master audio and the repaired master audio at different times is calculated, and the audio difference at different times is accumulated to obtain the time domain loss between the real master audio and the repaired master audio. That is, ; wherein, represents the time domain loss, and t represents time, represents the real master audio, represents the repaired master audio, represents the first norm.

[0082] Wherein, for the frequency domain loss, different resolutions are determined based on different preset window lengths and preset frame shifts corresponding to each preset window length, respectively; for example, the preset window length can be 4096, 2048, 1024, etc., and the preset frame shift is generally set to one quarter of the preset window length, the preset frame shift corresponding to the preset window length 4096 can be 1024, the preset frame shift corresponding to the preset window length 2048 can be 512, and the preset frame shift corresponding to the preset window length 1024 can be 256. Then, the real master audio and the repaired master audio are subjected to short time Fourier transform based on the preset window length and the preset frame shift corresponding to the preset window length, respectively, to obtain corresponding transform values, and the target transform value is determined based on the transform values corresponding to the real master audio and the repaired master audio at the same resolution. Finally, the frequency domain loss is determined based on the target transform values of the real master audio and the repaired master audio at different resolutions. The formula involved in the frequency domain loss is as follows:

[0083] ;

[0084] Wherein, represents the frequency domain loss, M represents the total number of resolutions of different resolutions, STFT( ) represents a short time Fourier transform function, represents the second norm, represents the target transform value at the same resolution, and denote two intermediate parameters for calculating the target transform value at the same resolution.

[0085] It can be seen that, by adopting the time domain loss, the embodiment can constrain the consistency of the energy and distribution of the repaired master audio and the real master audio, thereby ensuring the consistency in the time domain. Further, by adopting the frequency domain loss, the embodiment can restore the high-frequency spectrum in the audio and repair the low-frequency spectrum in the audio, thereby enabling the audio repair model to learn towards the direction of restoring the master. Further, the embodiment determines the final training loss between the real master audio and the repaired master audio based on the time domain loss and the frequency domain loss, to optimize and train the model by using the final training loss, thereby ensuring the audio repair effect of the model. Further, the audio repair model is a model constructed based on an encoder and a decoder with convolutional layers, and therefore the model structure and the model parameters can be modified to realize the cross-platform deployment of the audio repair model without the need to retrain the audio repair model after modification of the model structure, thereby improving the efficiency of cross-platform model deployment.

[0086] Further, the embodiment of the present application further discloses an electronic device, Figure 9 is a structural diagram of an electronic device 10 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0087] Figure 9 The embodiment of the present application provides a structural schematic diagram of an electronic device 10. The electronic device 10 specifically can include at least one processor 11, at least one memory 12, a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16. The memory 12 is used to store a computer program, the computer program is loaded and executed by the processor 11 to realize the related steps in the cross-platform model deployment method disclosed in any of the preceding embodiments. In addition, the electronic device 10 in the embodiment specifically can be an electronic computer.

[0088] In the embodiment, the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and external devices, and the communication protocol followed by the communication interface 14 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 15 is used to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0089] In addition, the memory 12 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 121, a computer program 122, etc., and the storage mode can be temporary storage or permanent storage.

[0090] The operating system 121 is used to manage and control each hardware device on the electronic device 10 and the computer program 122, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including the computer program capable of completing the cross-platform model deployment method executed by the electronic device 10 disclosed in any of the preceding embodiments, the computer program 122 can further include a computer program capable of completing other specific work.

[0091] Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to implement the cross-platform model deployment method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0092] Further, the present application also discloses a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the cross-platform model deployment method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0093] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0094] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0095] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0096] Finally, it should be noted that, in the description of the application, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0097] The above provides a detailed description of the technical solutions of the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation modes and application ranges will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A cross-platform model deployment method, characterized in that, The method comprises the following steps: reading a pre-trained general audio restoration model from a general platform; the general platform is a platform constructed based on a first processor; modifying a general model structure of the general audio restoration model based on a target model structure required by a to-be-ported platform to derive a modified audio restoration model; the to-be-ported platform is a platform constructed based on a second processor; the model parameters of the modified audio restoration model are the same as the model parameters of the general audio restoration model; reconfiguring the model parameters of the modified audio restoration model based on a difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model; deploying the target audio restoration model to the to-be-ported platform; the step of modifying the general model structure of the general audio restoration model based on the target model structure required by the to-be-ported platform comprises the following steps: correspondingly modifying a convolution structure in the general model structure of the general audio restoration model based on a target model convolution structure required by the to-be-ported platform; the structure parameters of the convolution structure include any one or a combination of input channel number, output channel number, convolution kernel size and convolution step length; the step of reconfiguring the model parameters of the modified audio restoration model based on the difference comparison result between the target model parameters and the model parameters of the modified audio restoration model to obtain a target audio restoration model comprises the following steps: using the difference comparison result and based on the target model parameters, modifying the parameters of the model parameters of the modified audio restoration model that are different from the target model parameters, and retaining the parameters of the model parameters of the modified audio restoration model that are not different from the target model parameters, to obtain a target audio restoration model; before the step of reading a pre-trained general audio restoration model from a general platform, the method further comprises the following steps: constructing a to-be-trained audio restoration model based on an encoder and a decoder with convolution layers, and reconstructing a reconstructed audio with the same sampling rate as a real master audio based on the real master audio; inputting the reconstructed audio into the to-be-trained audio restoration model to obtain hidden variables by dimension reduction of the reconstructed audio using the encoder, and obtaining a restored master audio by dimension increase of the hidden variables using the decoder; determining a training loss between the real master audio and the restored master audio to train the to-be-trained audio restoration model to obtain the pre-trained general audio restoration model.

2. The cross-platform model deployment method of claim 1, wherein, the step of determining target model parameters based on the target model structure comprises the following steps: determining corresponding target model parameters based on the structure parameters of the target model convolution structure; wherein the target model parameters include at least one of model weights and model offsets.

3. The cross-platform model deployment method of claim 1, wherein, the step of reconstructing a reconstructed audio with the same sampling rate as a real master audio based on the real master audio comprises the following steps: downsample the real master audio to obtain downsampled audio, a sampling rate of the downsampled audio being less than a sampling rate of the real master audio; upsample the downsampled audio to obtain reconstructed audio having the same sampling rate as the real master audio; wherein a highest audio in the reconstructed audio is less than a highest audio in the real master audio.

4. The cross-platform model deployment method of claim 1, wherein, The determining the training loss between the real master audio and the repaired master audio comprises: determining a time domain loss based on an audio difference between the real master audio and the repaired master audio at different times; determining a frequency domain loss based on a short-time Fourier transform of the real master audio and the repaired master audio at different resolutions; determining the training loss between the real master audio and the repaired master audio based on the time domain loss and the frequency domain loss.

5. The cross-platform model deployment method of claim 4, wherein, The determining the frequency domain loss based on the short-time Fourier transform of the real master audio and the repaired master audio at different resolutions comprises: determining different resolutions based on different preset window lengths and preset frame shifts respectively corresponding to the preset window lengths; respectively performing short-time Fourier transform on the real master audio and the repaired master audio based on the preset window lengths and the preset frame shifts respectively corresponding to the preset window lengths to obtain corresponding transform values, and determining a target transform value based on the transform values respectively corresponding to the real master audio and the repaired master audio at the same resolution; determining the frequency domain loss based on the target transform values of the real master audio and the repaired master audio at different resolutions.

6. An electronic device, comprising: comprise: a memory for saving a computer program; a processor for executing the computer program to implement the cross-platform model deployment method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, a computer program for saving, the computer program being executed by a processor to implement the cross-platform model deployment method according to any one of claims 1 to 5.

Images