A model training method and device
By introducing an intermediate model into the variational autoencoder and optimizing the probability distribution of feature encoding, the problems of long training time and low generation quality of the variational autoencoder are solved, and more efficient and high-quality model training results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI XIYU JIZHI TECH CO LTD
- Filing Date
- 2025-05-27
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, variational autoencoders suffer from long training times, numerous training steps, and low audio quality in large language model training, making it difficult to improve the quality of synthesized audio.
By optimizing the variational autoencoder model and adding an intermediate model, the probability distribution of the generated feature encoding satisfies a non-standard normal distribution. The intermediate model is then used to convert it into a standard normal distribution. Finally, the model parameters are adjusted by comparison to improve the encoding and decoding performance.
It improves training efficiency and the quality of generated content, making the generated target content closer to real data, with more detail and higher stability.
Smart Images
Figure CN120596918B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model training technology, and in particular to a model training method and apparatus. Background Technology
[0002] In the field of large language model technology, in order for large language models to learn from training data, the training data needs to be encoded. How to encode the data directly affects not only the learning and training effect and efficiency of the large model, but also the quality of the generated content.
[0003] For example, in the field of speech synthesis technology, in order for a large language model to learn from and train speech data, audio data needs to be compressed and encoded, and then the compressed and encoded audio data is input into the large language model for training. Related technologies generally use the encoder in a Variational Auto-Encoder (VAE) to downsample the audio file by several times. For example, for a 44.1kHz audio file, every 4410 sampling points are compressed into one code, equivalent to compressing every 0.1 seconds of audio into one audio feature code. This audio feature code is then input into the large language model for training, and the large language model outputs a predicted audio feature code. The decoder in the VAE then upsamples the predicted audio feature code by several times, for example, decoding one predicted audio feature code to generate an audio waveform with 4410 sampling points.
[0004] In a variational autoencoder (VAE), the encoder and decoder are two invertible models trained simultaneously. While the VAE model has a simple structure, achieving good training results requires a long training time and numerous training steps. Furthermore, the probability distribution in the VAE model uses a standard normal distribution to fit the real audio features. Therefore, the audio quality generated by the trained decoder is generally low, with a limited upper limit, making it difficult to improve the quality of synthesized audio. Summary of the Invention
[0005] This invention provides a model training method and apparatus. By optimizing the variational autoencoder model and adding intermediate models, the encoding and decoding effects of the target file can be improved, thereby enhancing the quality of the generated target content.
[0006] According to one aspect of the present invention, a model training method is provided, comprising:
[0007] A first training file is obtained, and the first training file is input into the encoding model in the variational autoencoder. The first training file is encoded by the encoding model to generate a first feature code; wherein the probability distribution of the first feature code satisfies a non-first preset distribution mode.
[0008] Based on the first feature encoding and the first model or first function in the intermediate model, a second feature encoding is generated; wherein the probability distribution of the second feature encoding satisfies the first preset distribution method;
[0009] Based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder, a first generated file corresponding to the first training file is generated; wherein, the second model or second function in the intermediate model performs the opposite process to the first model or first function;
[0010] The first training file and the first generated file are compared, and the model parameters of the variational autoencoder and / or the intermediate model are adjusted based on the comparison results.
[0011] According to another aspect of the present invention, a model training apparatus is provided, comprising:
[0012] The first feature encoding generation module is used to acquire a first training file, input the first training file into the encoding model in the variational autoencoder, encode the first training file through the encoding model, and generate a first feature encoding; wherein the probability distribution of the first feature encoding satisfies a non-first preset distribution mode;
[0013] The second feature encoding generation module is used to generate a second feature encoding based on the first feature encoding and the first model or first function in the intermediate model; wherein the probability distribution of the second feature encoding satisfies the first preset distribution method;
[0014] The first generated file generation module is used to generate a first generated file corresponding to the first training file based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder; wherein the second model or second function in the intermediate model performs the opposite process to the first model or first function;
[0015] The model parameter adjustment module is used to compare the first training file and the first generated file, and adjust the model parameters of the variational autoencoder and / or the intermediate model based on the comparison results.
[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0017] At least one processor; and
[0018] A memory communicatively connected to the at least one processor; wherein,
[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model training method according to any embodiment of the present invention.
[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model training method described in any embodiment of the present invention.
[0021] The model training scheme of this invention involves obtaining a first training file, inputting the first training file into the encoding model of a variational autoencoder, encoding the first training file through the encoding model to generate a first feature code; wherein the probability distribution of the first feature code satisfies a non-first preset distribution pattern; generating a second feature code based on the first feature code and a first model or a first function in an intermediate model; wherein the probability distribution of the second feature code satisfies the first preset distribution pattern; generating a first generated file corresponding to the first training file based on the second feature code, a second model or a second function in the intermediate model, and a decoding model in the variational autoencoder; wherein the second model or the second function in the intermediate model performs the opposite process to the first model or the first function; comparing the first training file and the first generated file, and adjusting the model parameters of the variational autoencoder and / or the intermediate model based on the comparison result. Through the technical solution provided by this invention, by optimizing the variational autoencoder model and adding an intermediate model, the encoding and decoding effect of the target file can be improved, thereby improving the quality of the generated target content. Meanwhile, compared to existing technologies, the improved variational autoencoder does not require constraining file feature encoding to a standard normal distribution or other first-preset distribution. Instead, it only requires constraining to a non-standard normal distribution (i.e., distributions other than the standard normal distribution) or other non-first-preset distributions, which greatly reduces the difficulty of model training and improves training efficiency. Furthermore, since the improved variational autoencoder has fewer constraints, the generated target content can be closer to the real data. In addition, adding an intermediate model can make the feature encoding richer in feature information, resulting in more detailed target content and significantly improving the quality and stability of target content generation.
[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 A flowchart of a model training method provided in an embodiment of the present invention;
[0025] Figure 2 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present invention;
[0026] Figure 3 A schematic diagram of the structure of an electronic device for implementing the model training method of this embodiment of the invention. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0029] Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of the present invention. This embodiment is applicable to the training of models. The method can be executed by a model training device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:
[0030] S110. Obtain a first training file, input the first training file into the encoding model in the variational autoencoder, encode the first training file through the encoding model, and generate a first feature code; wherein, the probability distribution of the first feature code satisfies a non-first preset distribution mode.
[0031] In this embodiment of the invention, a first training file is obtained, wherein the first training file includes at least one of text, audio, video, and image types. For example, the first training file is an audio training file, an image training file, or a video training file; or, for example, the first training file is a combination of an audio training file and a text training file; or, for example, the first training file is a combination of an audio training file, a video training file, and a text training file. It should be noted that this embodiment of the invention does not limit the file type of the first training file. Furthermore, the first training file can be 100 or 1000, and this embodiment of the invention does not limit the number of first training files.
[0032] The first training file is input into the encoding model of the variational autoencoder. The encoding model of the variational autoencoder encodes the first training file to generate a first feature code. The probability distribution of the first feature code satisfies a non-first preset distribution pattern, that is, the probability distribution of the first feature code satisfies other preset distribution patterns or any distribution pattern besides the first preset distribution pattern. Optionally, the probability distribution of the first feature code satisfies a second preset distribution pattern, which has fewer constraints than the first preset distribution pattern. It is understood that the second preset distribution pattern is a non-first preset distribution pattern, and the second preset distribution pattern has fewer constraints than the first preset distribution pattern. For example, the first preset distribution pattern can be a standard normal distribution, i.e., a standard Gaussian distribution, then the second preset distribution pattern can be a more complex and diverse normal distribution, i.e., a Gaussian distribution, or other predefined distribution patterns. Optionally, the first preset distribution pattern is a standard normal distribution, and the second preset distribution pattern is a normal distribution. It is understandable that the second preset distribution method is not the first preset distribution method because the second preset distribution method has fewer constraints than the first preset distribution method, rather than because the second preset distribution method and the first preset distribution method have no overlap. Continuing with the previous example, the first preset distribution method is a standard normal distribution, and the second preset distribution method is a normal distribution. Therefore, it can be seen that the standard normal distribution of the first preset distribution method is a special case of the second preset distribution method - normal distribution with a mean of 0 and a standard deviation of 1.
[0033] In traditional variational autoencoders, when downsampling a file, the encoding model constrains the probability distribution of the output feature code to conform to a standard normal distribution (or a standard Gaussian distribution). That is, the encoding model uses a standard normal or standard Gaussian distribution function to fit the probability distribution of the feature code corresponding to the file, calculates the loss value, and continues until it converges to a preset range. However, in this embodiment of the invention, when downsampling the first training file based on the encoding model in the variational autoencoder, the probability distribution of the output first feature code is constrained to conform to a non-preset distribution pattern.
[0034] S120. Based on the first feature code and the first model or first function in the intermediate model, generate a second feature code; wherein the probability distribution of the second feature code satisfies the first preset distribution method.
[0035] In this embodiment of the invention, a first feature code is input into a first model or a first function in an intermediate model to obtain a second feature code output by the first model or the first function, wherein the probability distribution of the second feature code satisfies a first preset distribution pattern. It can be understood that the first model or the first function in the intermediate model converts a first feature code whose probability distribution does not satisfy the first preset distribution pattern into a second feature code whose probability distribution satisfies the first preset distribution pattern. For example, the first model or the first function in the intermediate model is a model or function used to convert a feature code whose probability distribution satisfies a normal distribution or other predefined distribution pattern into a feature code whose probability distribution satisfies a standard normal distribution.
[0036] S130. Based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder, a first generated file corresponding to the first training file is generated; wherein, the second model or second function in the intermediate model performs the opposite process to the first model or first function.
[0037] For example, the second feature encoding is input into the second model or second function in the intermediate model, causing the second model or second function to perform the reverse process of the first model or first function on the second feature encoding, obtaining the feature encoding output by the second model or second function (which can be called the target feature encoding). This target feature encoding is then input into the decoding model in the variational autoencoder, causing the decoding model in the variational autoencoder to decode the target feature encoding and generate a first generated file corresponding to the first training file. The first generated file is a prediction file generated by the variational autoencoder and the intermediate model. It can be understood that the second model or second function in the intermediate model converts the second feature encoding, whose probability distribution satisfies a first preset distribution, into a target feature encoding, whose probability distribution satisfies a non-first preset distribution. For example, the second model or second function in the intermediate model is a model or function used to convert feature encodings whose probability distribution satisfies a standard normal distribution into feature encodings whose probability distribution satisfies a more complex, diverse normal distribution or other predefined distributions. The output encoding of the second model or second function in the intermediate model is the input encoding of the decoding model in the variational autoencoder. By using a second model or function in the intermediate model, the probability distribution of the feature encoding in the decoding model of the input variational autoencoder can be made to satisfy a more complex and diverse normal distribution or other predefined distribution patterns. This results in richer file features contained in the feature encoding of the input decoding model. For example, if the first training file is an audio training file, the second model or function in the intermediate model transforms the audio feature encoding whose probability distribution satisfies a first preset distribution pattern into a target audio feature encoding that satisfies a more complex and diverse normal distribution or other predefined distribution patterns. This allows the target audio feature encoding input to the decoder to contain richer audio features, resulting in a more detailed and higher-quality audio waveform generated after decoding.
[0038] Optionally, the second model or second function can be a Flow Matching model. The Flow Matching model is a reversible probability density transformation model that progressively transforms a simple distribution into a complex target distribution through a series of reversible transformation functions. This process can be viewed as an iterative process of variable substitution, with each substitution following the variable transformation principle of the probability density function. The Flow Matching model enables an accurate mapping from a simple distribution to a complex distribution.
[0039] S140. Compare the first training file and the first generated file, and adjust the model parameters of the variational autoencoder and / or the intermediate model based on the comparison results.
[0040] In this embodiment of the invention, the first training file is compared with the first generated file, and the model parameters of the variational autoencoder (including encoder and / or decoder) and / or intermediate model (including first model or first function and / or second model or second function) are adjusted based on the comparison results. The loss value is continuously calculated based on the comparison results between the first training file and the first generated file through steps S110-S140, and steps S110-S140 are iteratively executed until the loss value converges to a preset range, thereby completing the training of the variational autoencoder and intermediate model.
[0041] The model training method of this invention involves obtaining a first training file, inputting the first training file into the encoding model of a variational autoencoder, encoding the first training file through the encoding model to generate a first feature code; wherein the probability distribution of the first feature code satisfies a non-first preset distribution pattern; generating a second feature code based on the first feature code and a first model or a first function in an intermediate model; wherein the probability distribution of the second feature code satisfies the first preset distribution pattern; generating a first generated file corresponding to the first training file based on the second feature code, a second model or a second function in the intermediate model, and a decoding model in the variational autoencoder; wherein the second model or the second function in the intermediate model performs the opposite process to the first model or the first function; comparing the first training file and the first generated file, and adjusting the model parameters of the variational autoencoder and / or the intermediate model based on the comparison result. Through the technical solution provided by this invention, by optimizing the variational autoencoder model and adding an intermediate model, the encoding and decoding effect of the target file can be improved, thereby improving the quality of the generated target content. Meanwhile, compared to existing technologies, the improved variational autoencoder does not require constraining file feature encoding to a standard normal distribution or other first-preset distribution. Instead, it only requires constraining to a non-standard normal distribution (i.e., distributions other than the standard normal distribution) or other non-first-preset distributions, which greatly reduces the difficulty of model training and improves training efficiency. Furthermore, since the improved variational autoencoder has fewer constraints, the generated target content can be closer to the real data. In addition, adding an intermediate model can make the feature encoding richer in feature information, resulting in more detailed target content and significantly improving the quality and stability of target content generation.
[0042] In some embodiments, after training the variational autoencoder and intermediate model through the above embodiments, the method further includes: inputting the first generative code output by the first language model into the trained second model or second function, and outputting a second generative code; wherein, the first language model is a pre-trained large language model; inputting the second generative code into the trained decoding model, and outputting the target generated content; wherein, the second model or second function and the decoding model are trained by the model training method provided in the above embodiments.
[0043] In this embodiment of the invention, a file to be synthesized is obtained and input into a first language model to obtain a first generated code output by the first language model, wherein the first language model is a pre-trained large language model. Then, the first generated code output by the first language model is input into a second model or a second function in an intermediate model trained using the model training method provided in the above embodiment to obtain a second generated code output by the second model or the second function. The second generated code is input into a decoding model in a variational autoencoder trained using the model training method provided in the above embodiment, enabling the decoding model to decode the second generated code and generate the target generated content. It is understood that the probability distribution of the output code of a large language model typically follows a standard normal distribution. By inputting the first generated code output by the first language model into the second model or the second function, the first generated code, whose probability distribution follows a standard normal distribution or other pre-defined distribution, is converted into a second generated code whose probability distribution follows a more complex and diverse normal distribution or other predefined distribution. This allows the feature code input into the decoder to contain richer feature information, resulting in the target generated content generated by the decoding model having more detailed information and higher quality.
[0044] In this embodiment of the invention, speech synthesis using a first speech model is used as an example for explanation. During speech synthesis, the encoding model in the improved variational autoencoder (VAE) that has been trained can be used to extract audio feature codes from pre-trained speech data. These codes are then input into a large language model for pre-training. Ultimately, the probability distribution of the predicted audio feature codes output by the large language model conforms to the normal distribution constraint or other predefined distribution patterns in the improved VAE. Only in this way can the decoding model in the improved VAE, after training, correctly decode and generate the corresponding audio waveform. However, retraining the large language model is very costly and time-consuming. Therefore, the optimization and improvement process for speech synthesis is as follows: The predicted audio feature codes (i.e., the first generated codes) output by the pre-trained large language model are input into the trained second model or second function. The probability distribution of the audio feature codes (i.e., the second generated codes) transformed by the second model or second function satisfies a more complex and diverse normal distribution or other predefined distribution patterns. Finally, the decoding model in the improved VAE, after training, decodes the audio.
[0045] Optionally, the first generated code output by the first language model is input into the second model or the second function in a streaming manner. Optionally, the second generated code is input into the trained decoding model in a streaming manner to output the target generated content. For example, taking speech synthesis through a first speech model as an example, the text content to be synthesized is input into the trained first language model in a streaming manner, outputting the first generated code. Then, the first generated code is input into the trained second model or the second function in a streaming manner. For example, every 100 text contents are taken as a streaming input packet and input into the first language model to obtain the first generated code. Then, the first generated codes corresponding to the 100 texts are streamed into the second model or the second function, without waiting to obtain the generated codes corresponding to all 1000 text contents and then inputting them all into the second model or the second function. The second model or the second function then outputs the second generated code corresponding to the streaming input packet in sequence. The second generated code is input into the trained decoding model in a streaming manner to output the target generated content. Through the above method, the time interval for obtaining the target generated content can be significantly shortened, and the generation efficiency of the target generated content can be improved.
[0046] Optionally, the first generated code output by the first language model is the probability distribution of the hidden layer of the first language model. It is understood that the first generated code output by the first language model is the probability distribution of the hidden layer in the first language model, and is not the existing final token. The final token is obtained by sampling based on the hidden layer probability distribution.
[0047] In some embodiments, a second training file is obtained, and the second training file is input into the trained encoding model to generate a third feature code; wherein the third feature code satisfies a non-first preset distribution pattern; the third feature code is input into a second language model to generate a third generated code; wherein the second language model is a large language model that needs to be pre-trained; the third generated code is input into the trained decoding model to generate a second generated file; the second training file and the second generated file are compared, and the model parameters of the second language model are adjusted based on the comparison result.
[0048] A second training file is obtained, which may include at least one of the following types: text, audio, video, and image. For example, the second training file may be an audio training file, an image training file, or a video training file; or it may be a combination of an audio training file and a text training file; or it may be a combination of an audio training file, a video training file, and a text training file. It should be noted that the file type of the second training file is not limited in this embodiment of the invention. Furthermore, there may be 100 or 1000 second training files, and the number of second training files is not limited in this embodiment of the invention.
[0049] The second training file is input into the encoding model of the variational autoencoder trained using the model training method provided in the above embodiments. The encoding model encodes the second training file to generate a third feature code. The probability distribution of the third feature code satisfies a non-first preset distribution, meaning it satisfies a preset distribution or any distribution other than the first preset distribution. The third feature code is input into the second language model (i.e., the untrained large language model) to generate a third generated code. The third generated code output by the second language model is then input into the decoding model trained using the model training method provided in the above embodiments to generate a second generated file. It is understood that the third generated code output by the second language model has a probability distribution that does not satisfy the first preset distribution, thus ensuring that the decoding model can correctly decode the third generated code when it is input into the trained decoding model. The second training file and the second generated file are compared, and the model parameters of the second language model are adjusted based on the comparison results. The loss value is continuously calculated based on the comparison results between the second training file and the second generated file, and the above steps are continuously iterated and executed until the loss value of the second language model converges to the preset range, thereby completing the training of the second language model.
[0050] In some embodiments, the fourth generative encoding output by the trained second language model is input into the trained decoding model to output the target generated content. The advantage of this setup is that it eliminates the need for a second model or second function in the intermediate model to convert the output encoding of the second language model. The probability distribution of the output encoding of the second language model can directly satisfy a non-first preset distribution, thereby effectively ensuring that the decoding model can correctly decode when decoding the output encoding of the second language model using the trained decoding model.
[0051] The file to be synthesized is obtained and input into the trained second language model. The fourth generated code output by the trained second language model is obtained, wherein the probability distribution of the fourth generated code does not satisfy a first preset distribution. Then, the fourth generated code output by the trained second language model is input into the decoding model of the variational autoencoder trained using the model training method provided in the above embodiment. The decoding model decodes the fourth generated code to generate the target generated content. It is understood that the trained second language model can directly output a fourth generated code with a probability distribution satisfying a more complex and diverse normal distribution or other predefined distributions. This allows the code input to the decoder to contain richer feature information, resulting in the target generated content generated by the decoding model having more detailed information and higher quality.
[0052] In this embodiment of the invention, speech synthesis using a trained second speech model is used as an example for explanation. During speech synthesis, the encoding model in the trained improved variational autoencoder can be used to extract audio features from the text data to be synthesized. Then, the audio feature encoding output by the encoding model is input into the trained second language model. The fourth generative encoding, whose probability distribution directly satisfies a normal distribution constraint or other predefined distribution, is input into the trained decoding model. Thus, the decoding model in the trained improved variational autoencoder correctly decodes the audio and generates the corresponding audio waveform.
[0053] Optionally, the fourth generated code output by the second language model is streamed into the decoding model. For example, taking speech synthesis using a first speech model, the text content to be synthesized is streamed into the trained second language model, which outputs the fourth generated code. This fourth generated code is then streamed into the trained decoding model. For instance, every 100 text characters are input as a single stream into the second language model to obtain the fourth generated code. The fourth generated codes corresponding to these 100 characters are then streamed into the decoding model, without waiting to obtain the generated codes corresponding to all 1000 text characters before inputting them all into the decoding model. The decoding model then sequentially outputs the fourth generated codes corresponding to the streamed input packets. This method significantly shortens the time interval for obtaining the target generated content, improving the generation efficiency of the target generated content.
[0054] Optionally, the fourth generated code output by the second language model is the probability distribution of the hidden layer of the second language model. It is understood that the fourth generated code output by the second language model is the probability distribution of the hidden layer in the second language model, and is not the existing final token. The final token is obtained by sampling based on the hidden layer probability distribution.
[0055] Figure 2 This is a schematic diagram of a model training device provided in an embodiment of the present invention. Figure 2 As shown, the device includes:
[0056] The first feature encoding generation module 210 is used to acquire a first training file, input the first training file into the encoding model in the variational autoencoder, encode the first training file through the encoding model, and generate a first feature encoding; wherein, the probability distribution of the first feature encoding satisfies a non-first preset distribution mode;
[0057] The second feature encoding generation module 220 is used to generate a second feature encoding based on the first feature encoding and the first model or first function in the intermediate model; wherein the probability distribution of the second feature encoding satisfies the first preset distribution method;
[0058] The first generated file generation module 230 is used to generate a first generated file corresponding to the first training file based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder; wherein the second model or second function in the intermediate model performs the opposite process to the first model or first function.
[0059] The model parameter adjustment module 240 is used to compare the first training file and the first generated file, and adjust the model parameters of the variational autoencoder and / or the intermediate model based on the comparison results.
[0060] Optional, also includes:
[0061] The second generative encoding output module is used to input the first generative encoding output by the first language model into the trained second model or second function, and output the second generative encoding; wherein, the first language model is a pre-trained large language model;
[0062] The first target content generation output module is used to input the second generation encoding into the trained decoding model and output the target content.
[0063] Optional, also includes:
[0064] The third feature encoding generation module is used to obtain the second training file, input the second training file into the trained encoding model, and generate the third feature encoding; wherein the third feature encoding satisfies a non-first preset distribution pattern;
[0065] The third generative encoding module is used to generate a third generative encoding based on the third feature encoding input to the second language model; wherein, the second language model is a large language model that needs to be pre-trained;
[0066] The second generated file generation module is used to input the third generated encoding into the trained decoding model to generate the second generated file;
[0067] The second language model training module is used to compare the second training file and the second generated file, and adjust the model parameters of the second language model based on the comparison results.
[0068] Optional, also includes:
[0069] The first target content generation output module is used to input the fourth generation encoding output by the trained second language model into the trained decoding model and output the target content.
[0070] Optionally, the probability distribution of the first feature encoding satisfies a second preset distribution method, which has fewer constraints than the first preset distribution method.
[0071] Optionally, the first preset distribution method is a standard normal distribution, and the second preset distribution method is a normal distribution.
[0072] Optionally, the first training file may include at least one of the following types: text, audio, video, and image.
[0073] Optionally, the second model or the second function may include a Flow Matching model.
[0074] Optionally, the output encoding of the language model is input to the second model or the second function, or the decoding model, in a streaming manner.
[0075] Optionally, the generated encoding output by the language model is the probability distribution of the hidden layer of the language model.
[0076] The model training apparatus provided in this embodiment of the invention can execute the model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0077] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0078] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0079] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0080] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as model training methods.
[0081] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.
[0082] In some embodiments, the model training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model training method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the model training method by any other suitable means (e.g., by means of firmware).
[0083] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.
[0084] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0085] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0087] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0088] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0089] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A model training method, characterized in that, include: A first training file is obtained, and the first training file is input into the encoding model in the variational autoencoder. The first training file is encoded by the encoding model to generate a first feature code. The probability distribution of the first feature code satisfies a non-first preset distribution. The first training file is an audio training file. Based on the first feature encoding and the first model or first function in the intermediate model, a second feature encoding is generated; wherein, the probability distribution of the second feature encoding satisfies the first preset distribution pattern; the non-first preset distribution pattern is the second preset distribution pattern, which has fewer constraints than the first preset distribution pattern; the first preset distribution pattern is a standard normal distribution, and the second preset distribution pattern is a normal distribution; Based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder, a first generated file corresponding to the first training file is generated; wherein, the second model or second function in the intermediate model performs the opposite process to the first model or first function; the second model or second function includes a Flow Matching model; The first training file and the first generated file are compared, and the model parameters of the variational autoencoder and / or the intermediate model are adjusted based on the comparison results. This includes, during speech synthesis, the following: The first generated code output by the first language model is input into the trained second model or second function to output a second generated code; wherein, the first language model is a pre-trained large language model; the first generated code output by the first language model is the probability distribution of the hidden layer of the first language model; The second generated encoding is input into the trained decoding model, which outputs the target generated content. This also includes: Obtain a second training file, input the second training file into the trained encoding model, and generate a third feature code; wherein the third feature code satisfies a non-first preset distribution pattern; The third feature encoding is input into the second language model to generate a third generated encoding; wherein, the second language model is a large language model that needs to be pre-trained. The third generated encoding is input into the trained decoding model to generate a second generated file; The second training file and the second generated file are compared, and the model parameters of the second language model are adjusted based on the comparison results.
2. The method according to claim 1, characterized in that, Also includes: The fourth generative encoding output from the trained second language model is input into the trained decoding model to output the target generated content.
3. The method according to claim 1 or 2, characterized in that, The output encoding of the first language model is input to the second model or the second function in a streaming manner, and the output encoding of the second language model is input to the decoding model in a streaming manner.
4. The method according to claim 2, characterized in that, The fourth generated encoding output by the second language model is the probability distribution of the hidden layer of the second language model.
5. A model training device, characterized in that, include: The first feature encoding generation module is used to acquire a first training file, input the first training file into the encoding model in the variational autoencoder, encode the first training file through the encoding model, and generate a first feature encoding; wherein, the probability distribution of the first feature encoding satisfies a non-first preset distribution mode; the first training file is an audio training file; The second feature encoding generation module is used to generate a second feature encoding based on the first feature encoding and a first model or first function in the intermediate model; wherein the probability distribution of the second feature encoding satisfies the first preset distribution pattern; the non-first preset distribution pattern is the second preset distribution pattern, and the second preset distribution pattern has fewer constraints than the first preset distribution pattern; the first preset distribution pattern is a standard normal distribution, and the second preset distribution pattern is a normal distribution; The first generated file generation module is used to generate a first generated file corresponding to the first training file based on the second feature encoding, the second model or second function in the intermediate model, and the decoding model in the variational autoencoder; wherein the second model or second function in the intermediate model performs the opposite process to the first model or first function; the second model or second function includes a Flow Matching model; The model parameter adjustment module is used to compare the first training file and the first generated file, and adjust the model parameters of the variational autoencoder and / or the intermediate model based on the comparison results. This includes, during speech synthesis, the following: The first generated code output by the first language model is input into the trained second model or second function to output a second generated code; wherein, the first language model is a pre-trained large language model; the first generated code output by the first language model is the probability distribution of the hidden layer of the first language model; The second generated encoding is input into the trained decoding model, which outputs the target generated content. This also includes: The third feature encoding generation module is used to obtain the second training file, input the second training file into the trained encoding model, and generate the third feature encoding; wherein the third feature encoding satisfies a non-first preset distribution pattern; The third generative encoding module is used to generate a third generative encoding based on the third feature encoding input to the second language model; wherein, the second language model is a large language model that needs to be pre-trained; The second generated file generation module is used to input the third generated encoding into the trained decoding model to generate the second generated file; The second language model training module is used to compare the second training file and the second generated file, and adjust the model parameters of the second language model based on the comparison results.