Language model training method and device

Abstract

A language model training method and apparatus (900). The method comprises: acquiring a training data set (401); respectively performing feature extraction on speech sample data and text sample data to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, wherein the first feature representation and the second feature representation belong to the same representation space (402); mapping the first feature representation and the second feature representation to a discrete coding space to obtain first discrete codes corresponding to the speech sample data and second discrete codes corresponding to the text sample data (403); and on the basis of the first discrete codes and the second discrete codes of a plurality of speech-text pairs, training an LLM to be trained (404). According to the method, speech sample data and text sample data which are not paired are expressed in the same representation space, so that multi-modal data of the speech sample data and the text sample data is used to train language models, thereby reducing the model training cost and improving the model training effect.

Description

A language model training method and device

[0001] This application claims priority to Chinese patent application number 202410332441.5, filed on March 21, 2024, entitled “A training method and device for a language model,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for training a language model. Background Art

[0003] Existing speech pre-training model frameworks, such as wav2vec 2.0 and HuBERT, are trained solely on unlabeled speech. According to neuroscience research, humans first pre-process speech and text using different regions of the cerebral cortex, and then extract high-level semantic information in the same region (the Wernicke-Geschwind area). Inspired by this discovery, we designed two pre-processing networks and a unified representation space to achieve joint pre-training of speech and text. This allows speech models to extract more high-level semantic information from text modalities.

[0004] Most of the existing speech-text joint pre-training model frameworks require a certain number of semantically identical speech-text pairs to guide the model to align and learn speech data and text data of different modalities. Otherwise, the model training effect will be poor, which undoubtedly increases the training cost of the model. Summary of the Invention

[0005] The embodiments of the present application provide a language model training method and device, which realize the expression of unpaired speech sample data and text sample data in the same representation space, and further realize the multimodal data training of the language model using speech sample data and text sample data. No speech-text pair data is required during the model training process, which reduces the model training cost and improves the model training effect.

[0006] In a first aspect, the present application provides a method for training a language model, comprising obtaining a training data set, the training data set comprising multiple speech-text pairs, the speech-text pairs comprising speech sample data and text sample data, the speech sample data and the text sample data having different semantics; performing feature extraction on the speech sample data and the text sample data respectively, to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, the first feature representation and the second feature representation belonging to the same representation space; mapping the first feature representation and the second feature representation to a discrete coding space, to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data respectively; and training the language model to be trained based on the first discrete codes and the second discrete codes of the multiple speech-text pairs.

[0007] The language model training method provided in this application expresses unpaired speech sample data and text sample data in the same representation space to obtain feature representations of the speech sample data and feature representations of the text sample data, then maps the feature representations of the speech sample data and the feature representations of the text sample data to a discrete coding space to obtain discretized speech coding and text coding, and finally uses the discretized speech coding and text coding to jointly train the language model, so that the language model can benefit from both speech and text data at the same time, thereby training a language model with better performance.

[0008] In one possible implementation, feature extraction is performed on speech sample data and text sample data respectively to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data. A specific implementation is: using the speech sample data as the input of a target speech unit generator, and outputting the first feature representation; using the text sample data as the input of a target text unit generator, and outputting the second feature representation; wherein the target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism.

[0009] The target speech unit generator and the target text unit generator are obtained by joint training iteration through the generative adversarial network (GAN) mechanism, so that the target text unit generator can extract the text features H from the text sample data. i And the speech feature H′ extracted by the target speech unit generator i They can belong to the same representation space, which facilitates the subsequent training of the language model.

[0010] In another possible implementation, each speech-text pair in a plurality of speech-text pairs has a label, which indicates whether the feature distributions of the speech sample data and the text sample data in the speech-text pair are the same; a specific implementation of the target speech unit generator and the target text unit generator being jointly iteratively trained based on the generative adversarial network mechanism is as follows: using the speech sample data as the input of the first speech unit generator, and outputting a first speech feature representation; using the text sample data as the input of the first text unit generator, and outputting a first text feature representation; using the first speech feature representation and the first text feature representation as the input of the discriminator, and outputting a first probability distribution that the first speech feature representation and the first text feature representation belong to the same representation space; based on the first probability distribution and the label, adjusting the weight parameters of the first speech unit generator and the weight parameters of the first text unit generator to obtain the target speech unit generator and the target text unit generator.

[0011] In this possible implementation, the text feature H is generated by treating the first speech unit generator and the first text unit generator as generators of a generative adversarial network. i and speech feature H′ i , and then generate the text feature H i and speech feature H′ i Input to the discriminator, and adjust the weight parameters of the first speech unit generator and the first text unit generator according to whether the discriminator's discrimination result is consistent with the label, and finally realize the speech feature H' output by the first speech unit generator i And the text feature H output by the first text unit generator i The feature distribution of i and text features H i belong to the same representation space.

[0012] It should be noted that the feature distribution of the speech sample data and text sample data in the speech-text pair indicated by the label of the speech-text pair does not refer to a specific semantic feature distribution, but to a broader feature distribution. For example, if the language model is a speech recognition model, the feature distribution refers to the grammatical / language feature distribution, which indicates whether the speech sample data and the text sample data are of the same language, or whether the grammar of the speech sample data and the text sample data is correct. For example, if the speech in a speech-text pair is Chinese speech and the text is English text, the label is 0, indicating that the feature distributions of the speech-text pair are different. If the speech in a speech-text pair is Chinese speech and the text is Chinese text, the label is 1, indicating that the feature distributions of the speech-text pair are the same. For another example, if the speech in a speech-text pair is Chinese speech with correct grammar and the text is grammatically incorrect text, such as "I love Shenzhen", the label is 0, indicating that the feature distributions of the speech-text pair are different. If the speech in a speech-text pair is Chinese speech with correct grammar and the text is grammatically incorrect text, such as "I love Shenzhen", the label is 1, indicating that the feature distributions of the speech-text pair are the same.

[0013] In another possible implementation, the first speech unit generator is trained based on the following steps: converting text sample data into a phoneme sequence; using the phoneme sequence as the input of the embedding network layer and outputting a second text feature representation; using the speech sample data as the input of the first speech unit generator to be trained and outputting a second speech feature representation; using the second speech feature representation and the second text feature representation as the input of the discriminator and outputting a second probability distribution of the second speech feature representation and the second text feature representation belonging to the same representation space; based on the second probability distribution and the label, adjusting the weight parameters of the first speech unit generator to be trained to obtain the first speech unit generator.

[0014] Exemplarily, the network structure of the first speech unit generator can be: a three-layer 1D convolutional neural network (CNN), with specific structural parameters of kernel size = 9, stride size = 3, and batchnorm1d layer dimension = 30. The implementation tool for converting text sample data into phoneme sequences can be a grapheme to phoneme conversion tool (G2P). The discriminator can be a 1D CNN, with specific structural parameters of kernel size = 8, stride size = 1, and batchnorm1d layer dimension = 30.

[0015] In another possible implementation, the first text unit generator is trained based on the following steps: fixing the weight parameters of the first speech unit generator, taking the speech sample data as the input of the first speech unit generator, and outputting the third speech feature representation; converting the text sample data into a phoneme sequence; taking the phoneme sequence as the input of the first text unit generator to be trained, and outputting the third text feature representation; taking the third speech feature representation and the third text feature representation as the input of the discriminator, and outputting the third probability distribution of the third speech feature representation and the third text feature representation belonging to the same representation space; based on the third probability distribution and the label, adjusting the weight parameters of the first text unit generator to be trained to obtain the first text unit generator.

[0016] Exemplarily, the network structure of the first text unit generator can be: three-layer 1D CNN, with specific structural parameters of kernel size = 9, stride size = 3, batchnorm1d layer dimension = 30, and the structure of the discriminator network is 1D CNN, with specific structural parameters of kernel size = 8, stride size = 1, batchnorm1d layer dimension = 30.

[0017] This application first uses the factor sequence converted from text sample data as the initial text unit to train a first speech unit generator, then fixes the first speech unit generator and trains the first text unit generator, and finally uses the first speech unit generator and the first text unit generator as generators of a generative adversarial network. The first speech unit generator and the first text unit generator are jointly trained with the discrimination results of the discriminator and the labels of the speech-text pairs to obtain a target speech unit generator and a target text unit generator that can map the extracted features to the same representation space.

[0018] In another possible implementation, a specific implementation of mapping the first feature representation and the second feature representation to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data is as follows: mapping the first feature representation to a first discrete coding space to obtain a first discrete code, the first discrete coding space includes identifiers of multiple first clusters, and the multiple first clusters are clustered based on multiple first feature representations corresponding to multiple speech sample data in multiple speech-text pairs; mapping the second feature representation to a second discrete coding space to obtain a second discrete code, the second discrete coding space includes identifiers of multiple second clusters, and the multiple second clusters are clustered based on multiple second feature representations corresponding to multiple text sample data in multiple speech-text pairs.

[0019] In this possible implementation, special clustering models are constructed for speech feature representation and text feature representation respectively, such as a speech clustering model and a text clustering model. The speech clustering model is used to map the speech feature representation to the speech discrete coding space to obtain speech discrete coding; the text clustering model is used to map the text feature representation to the text discrete coding space to obtain text discrete coding, which facilitates subsequent encoding by the encoding network of the specific modality.

[0020] In another possible implementation, mapping the first feature representation and the second feature representation to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively, is specifically implemented as follows: mapping the first feature representation and the second feature representation to the same discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, wherein the discrete coding space includes identifiers of multiple clusters, and the multiple clusters are clustered based on multiple first feature representations corresponding to multiple speech sample data and multiple second feature representations corresponding to multiple text sample data in multiple speech-text pairs.

[0021] That is to say, this application can also construct only one clustering model for speech feature representation and text feature representation, and use this clustering model to map the speech feature representation and text feature representation to the same discrete coding space to obtain speech discrete coding and text discrete coding, saving training resource overhead.

[0022] In another possible implementation, the above-mentioned language model includes a speech coding network, a text coding network and a first shared coding network; based on the first discrete codes and the second discrete codes of multiple speech-text pairs, a specific implementation of training the language model to be trained is: using the first discrete code as the input of the speech coding network, outputting the first speech precode; using the second discrete code as the input of the text coding network, outputting the first text precode; masking the first speech precode to obtain first masked data; masking the first text precode to obtain second masked data; using the first masked data as the input of the first shared coding network, outputting a first feature vector; predicting at least one masked code in the first masked data through the first feature vector, and adjusting the weight parameters of the speech coding network and the first shared coding network based on the prediction result; using the second masked data as the input of the first shared coding network, outputting a second feature vector; predicting at least one masked code in the second masked data through the second feature vector, and adjusting the weight parameters of the text coding network and the first shared coding network based on the prediction result.

[0023] In this possible implementation, the language model has encoding networks for specific modalities, such as a speech encoding network for discrete speech encoding and a text encoding network for discrete text encoding. This allows different masking strategies to be used for data input from different modalities, further reducing the semantic differences between data from different modalities. For example, a 15% masking strategy may be used for speech precoding, while a 20% masking strategy may be used for text precoding.

[0024] In another possible implementation, the above-mentioned language model includes a speech-text encoding network and a second shared encoding network; based on the first discrete encoding and the second discrete encoding of multiple speech-text pairs, a specific implementation of training the language model to be trained is: using the first discrete encoding and the second discrete encoding as inputs of the speech-text encoding network, respectively, and outputting the second speech precoding and the second text precoding; masking the second speech precoding and the second text precoding, respectively, to obtain third masked data and fourth masked data; using the third masked data as input of the second shared encoding network, and outputting a third feature vector; predicting at least one masked encoding in the third masked data through the third feature vector, and adjusting the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result; using the fourth masked data as input of the second shared encoding network, and outputting a fourth feature vector; predicting at least one masked encoding in the fourth masked data through the fourth feature vector, and adjusting the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result.

[0025] The language model to be trained in the present application can also set up a common speech-text coding network for speech discrete coding and text discrete coding. Both speech discrete coding and text discrete coding are encoded through a common speech-text coding network to obtain speech precoding and text precoding. There is no need to model encoders for speech and text separately, thereby reducing costs.

[0026] Optionally, the language model is a large language model (LLM).

[0027] In a second aspect, the present application provides a language model training device, comprising an acquisition module, a feature extraction module, a mapping module and a training module, wherein the acquisition module is used to acquire a training data set, the training data set comprising multiple speech-text pairs, the speech-text pairs comprising speech sample data and text sample data, and the speech sample data and the text sample data have different semantics; the feature extraction module is used to perform feature extraction on the speech sample data and the text sample data respectively, to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, and the first feature representation and the second feature representation belong to the same representation space; the mapping module is used to map the first feature representation and the second feature representation to a discrete coding space, to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data respectively; the training module is used to train the language model to be trained based on the first discrete code and the second discrete code of the multiple speech-text pairs.

[0028] In one possible implementation, the feature extraction module is specifically used to: use speech sample data as input to a target speech unit generator and output a first feature representation; use text sample data as input to a target text unit generator and output a second feature representation; wherein the target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism.

[0029] In another possible implementation, each speech-text pair in a plurality of speech-text pairs has a label, and the label indicates whether the feature distributions of the speech sample data and the text sample data in the speech-text pair are the same; the target speech unit generator and the target text unit generator are trained based on the following steps: taking the speech sample data as the input of the first speech unit generator, and outputting a first speech feature representation; taking the text sample data as the input of the first text unit generator, and outputting a first text feature representation; taking the first speech feature representation and the first text feature representation as the input of the discriminator, and outputting a first probability distribution that the first speech feature representation and the first text feature representation belong to the same representation space; based on the first probability distribution and the label, adjusting the weight parameters of the first speech unit generator and the weight parameters of the first text unit generator to obtain the target speech unit generator and the target text unit generator.

[0030] In another possible implementation, the first speech unit generator is trained based on the following steps: converting text sample data into a phoneme sequence; using the phoneme sequence as the input of the embedding network layer and outputting a second text feature representation; using the speech sample data as the input of the first speech unit generator to be trained and outputting a second speech feature representation; using the second speech feature representation and the second text feature representation as the input of the discriminator and outputting a second probability distribution of the second speech feature representation and the second text feature representation belonging to the same representation space; based on the second probability distribution and the label, adjusting the weight parameters of the first speech unit generator to be trained to obtain the first speech unit generator.

[0031] In another possible implementation, the first text unit generator is trained based on the following steps: fixing the weight parameters of the first speech unit generator, taking the speech sample data as the input of the first speech unit generator, and outputting the third speech feature representation; converting the text sample data into a phoneme sequence; taking the phoneme sequence as the input of the first text unit generator to be trained, and outputting the third text feature representation; taking the third speech feature representation and the third text feature representation as the input of the discriminator, and outputting the third probability distribution of the third speech feature representation and the third text feature representation belonging to the same representation space; based on the third probability distribution and the label, adjusting the weight parameters of the first text unit generator to be trained to obtain the first text unit generator.

[0032] In another possible implementation, the mapping module is specifically used to: map the first feature representation to a first discrete coding space to obtain a first discrete code, the first discrete coding space includes identifiers of multiple first clusters, and the multiple first clusters are clustered based on multiple first feature representations corresponding to multiple speech sample data in multiple speech-text pairs; map the second feature representation to a second discrete coding space to obtain a second discrete code, the second discrete coding space includes identifiers of multiple second clusters, and the multiple second clusters are clustered based on multiple second feature representations corresponding to multiple text sample data in multiple speech-text pairs.

[0033] In another possible implementation, the mapping module is specifically used to: map the first feature representation and the second feature representation to the same discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, wherein the discrete coding space includes identifiers of multiple clusters, and the multiple clusters are obtained by clustering multiple first feature representations corresponding to multiple speech sample data in multiple speech-text pairs and multiple second feature representations corresponding to multiple text sample data.

[0034] In another possible implementation, the above-mentioned language model includes a speech coding network, a text coding network and a first shared coding network; the training module is specifically used to: use the first discrete code as the input of the speech coding network, and output the first speech precode; use the second discrete code as the input of the text coding network, and output the first text precode; mask the first speech precode to obtain first mask data; mask the first text precode to obtain second mask data; use the first mask data as the input of the first shared coding network, and output a first feature vector; predict at least one masked code in the first mask data through the first feature vector, and adjust the weight parameters of the speech coding network and the first shared coding network based on the prediction result; use the second mask data as the input of the first shared coding network, and output a second feature vector; predict at least one masked code in the second mask data through the second feature vector, and adjust the weight parameters of the text coding network and the first shared coding network based on the prediction result.

[0035] In another possible implementation, a masking strategy for performing masking on the first discrete code is different from a masking strategy for performing masking on the second discrete code.

[0036] In another possible implementation, the above-mentioned language model includes a speech-text encoding network and a second shared encoding network; the training module is specifically used to: use the first discrete code and the second discrete code as inputs of the speech-text encoding network, and output a second speech precode and a second text precode; mask the second speech precode and the second text precode, respectively, to obtain third mask data and fourth mask data; use the third mask data as input of the second shared encoding network, and output a third feature vector; predict at least one masked code in the third mask data through the third feature vector, and adjust the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result; use the fourth mask data as input of the second shared encoding network, and output a fourth feature vector; predict at least one masked code in the fourth mask data through the fourth feature vector, and adjust the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result.

[0037] In a third aspect, an embodiment of the present application provides a computing device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0039] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product, which includes instructions that, when executed, cause a computer to execute the method described in the first aspect.

[0040] In a sixth aspect, an embodiment of the present application further provides a chip comprising at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 shows a schematic diagram of an artificial intelligence main body framework;

[0042] FIG2 is a system architecture diagram of a sample processing system provided in an embodiment of the present application;

[0043] FIG3 is a schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0044] FIG4 is a flow chart of a method for training a language model according to an embodiment of the present application;

[0045] FIG5 is a schematic diagram showing the structure of a GAN network for training the speech unit generator and the text unit generator mentioned in an embodiment of the present application;

[0046] FIG6 shows a schematic diagram of LLM training;

[0047] FIG7 shows a schematic diagram of another LLM training;

[0048] FIG8 is a schematic diagram showing the process of processing input data by the LLM mentioned in an embodiment of the present application;

[0049] FIG9 is a schematic diagram of the structure of a language model training device provided in an embodiment of the present application;

[0050] FIG10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The term "and / or" as used herein describes an association relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " in this document indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0052] In the specification and claims herein, the terms "first" and "second" are used to distinguish between different objects, rather than to describe a specific order of objects. For example, a first feature representation and a second feature representation are used to distinguish between different feature representations, rather than to describe a specific order of feature representations.

[0053] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0055] First, let's describe the overall workflow of the artificial intelligence system. Please refer to Figure 1. Figure 1 shows a schematic diagram of an artificial intelligence main framework. This main framework describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0056] The following explains the AI ​​framework from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general progression from intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, to intelligent execution and output. In this process, data undergoes the condensed evolution of "data-information-knowledge-wisdom." The "IT value chain" reflects the value AI brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem. From the perspective of the IT value chain, the AI ​​framework comprises an infrastructure layer, a data processing layer, a general capability layer (which implements common capabilities such as translation, text analysis, computer vision processing, speech recognition, and image recognition through data processing results), and finally, the topmost application layer, which encapsulates the overall AI solution, productizing intelligent information decision-making and enabling practical applications.

[0057] (1) Infrastructure:

[0058] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. Communication with the outside world occurs through sensors; computing power is provided by intelligent chips, including but not limited to hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs); the foundational platform includes distributed computing frameworks and networks, along with other related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the distributed computing system provided by the foundational platform for computation.

[0059] (2) Data

[0060] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0061] (3) Data processing

[0062] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0063] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0064] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0065] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0066] (4) General ability

[0067] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0068] (5) Smart products and industry applications

[0069] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0070] The language model training method and device provided in the embodiments of the present application are mainly used in the field of natural language processing, such as intelligent voice scenarios such as speech translation, semantic recognition, spoken language understanding and voice dialogue systems.

[0071] The training method and device of the language model provided in the embodiment of the present application adopt a fully self-supervised speech-text joint pre-training framework based on speech discretization, and map the speech and text of different modalities to a unified representation space through iterative training of the generative adversarial network GAN, so as to realize the simultaneous pre-training of the language model (such as the large language model LLM) using non-parallel speech and text data. It is realized that the unpaired speech sample data and text sample data are expressed in the same representation space, and then the multimodal data training language model using speech sample data and text sample data is realized. No speech-text pair data is required during the model training process, which reduces the model training cost and improves the model training effect. The trained language model can be applied to the various application fields mentioned above to realize functions such as speech translation, semantic recognition, spoken language understanding and speech dialogue, and the processing object of the trained language model can be a speech sample. As an example, in the field of smart terminals, the trained language model can be configured on the smart terminal to realize the speech translation function. As another example, in the field of autonomous driving, the trained language model can be configured on the autonomous driving vehicle to realize the speech dialogue function. The language models mentioned in the aforementioned examples can be trained using the language model training method provided in this application during the training phase, thereby improving the performance of the trained language models. It should be understood that the examples given here are only for the purpose of facilitating the understanding of the application scenarios of the embodiments of this application, and are not intended to be exhaustive of the application scenarios of the embodiments of this application.

[0072] In order to facilitate understanding of the present solution, the sample processing system provided in the embodiment of the present application is first introduced in conjunction with Figure 2. Referring to Figure 2, the data acquisition device 260 is used to collect training data. The training data in the embodiment of the present application includes: labeled training text pairs.

[0073] After collecting the training data, the data collection device 260 stores the training data in the database 230. The training device 220 generates the target model / rule 201 based on the training data maintained in the database 230. The following describes in more detail how the training device 220 obtains the target model / rule 201 based on the training data. The target model / rule 201 can implement the speech data processing functions provided in the embodiments of the present application, such as converting speech data into corresponding translated text.

[0074] In actual applications, the training data maintained in the database 230 may not all be collected by the data acquisition device 260, but may also be received from other devices. In addition, it should be noted that the training device 220 may not train the target model / rule 201 entirely based on the training data maintained by the database 230, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0075] The target model / rule 201 obtained through training with the training device 220 can be applied to different systems or devices, such as the execution device 210 in FIG2 . The execution device 210 can be a terminal, such as a mobile phone, a tablet computer, a laptop computer, augmented reality (AR), virtual reality (VR), a wearable device, an intelligent robot, an in-vehicle terminal, etc., or a server or cloud, etc.

[0076] In Figure 2 , execution device 210 is configured with an input / output (I / O) interface 212 for exchanging data with external devices. A user can input data into I / O interface 212 via client device 240. In this embodiment, this data may include user-entered speech. It should be noted that speech can be input in any manner, such as user-entered speech data, directly captured via a microphone, or user-entered text data, which is then converted into speech using a speech generation model.

[0077] The preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing based on the input data received by the I / O interface 212 (such as the voice input by the user). For example, in an embodiment of the present application, the preprocessing module 213 removes the silence at the beginning and end of the voice data and divides the voice signal after the silence is removed into frames, and the preprocessing module 214 extracts the features of each voice.

[0078] The calculation module 211 is used to perform calculation and other related processing on the data input from the pre-processing module 213 and the pre-processing module 214 according to the above-mentioned target model / rule 201.

[0079] When the execution device 210 preprocesses the input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. of the database storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 250.

[0080] Finally, the I / O interface 212 returns the processing results (such as speech recognition results, translation results, etc.) to the client device 240 and provides them to the user. It should be understood that corresponding to different natural language processing tasks, the target model / rule 201 is different, and its processing results are also different accordingly.

[0081] It is worth noting that the training device 220 can generate target models / rules 201 corresponding to different downstream systems. The corresponding target models / rules 201 can achieve the above-mentioned goals or complete the above-mentioned tasks, thereby providing the user with the desired results. It should be noted that the training device 220 can also generate corresponding preprocessing models for target models / rules 201 corresponding to different downstream systems, such as the corresponding preprocessing models in the preprocessing module 213 and / or the preprocessing module 214.

[0082] In the case shown in FIG2 , the user can manually specify the data to be input into the execution device 210 (for example, input a voice), for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data into the I / O interface 212 (for example, input a voice) and obtain the result. If the automatic data input of the client device 240 requires the user's authorization, the user can set the corresponding authority in the client device 240. The user can view the result output by the execution device 210 on the client device 240 (for example, the output result can be a translated text, an answer to a question, etc.), and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also serve as a data acquisition terminal, collecting the input data (voice data to be translated) input into the I / O interface 212 as shown in FIG2 and the translation result output by the target model / rule 201 as new training sample data, and storing it in the database 230.

[0083] It is worth noting that Figure 2 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in Figure 2, data storage system 250 is an external memory relative to execution device 210. In other cases, data storage system 250 can also be placed within execution device 210.

[0084] FIG3 is a schematic diagram of the hardware structure of a chip provided in an embodiment of the present application. The chip includes a neural network processor (Network Process Units, NPU) 30. The chip can be set in the execution device 210 shown in FIG2 to complete the calculation work of the calculation module 211. The chip can also be set in the training device 220 shown in FIG2 to complete the training work of the training device 220 and output the target model / rule 201.

[0085] The language model training method of the embodiment of the present application can be executed in the operation circuit 303 and / or the vector calculation unit 307 in the neural network processor 30, so as to obtain a high-performance speech model.

[0086] The following is a brief introduction to the various modules and units in the neural network processor 30.

[0087] The neural network processor 30 is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks to it. The core of the neural network processor 30 is the arithmetic circuit 303. When the neural network processor 30 is operating, the controller 304 in the neural network processor 30 can control the arithmetic circuit 303 to extract matrix data from the memory and perform multiplication operations.

[0088] In some implementations, the arithmetic circuit 303 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0089] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 302 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 301 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 308.

[0090] The vector calculation unit 307 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-fully connected layers (FC layers) in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0091] In some implementations, the vector calculation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0092] The unified memory 306 is used to store input data and output data.

[0093] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the Direct Memory Access Controller (DMAC) 305, the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0094] BIU stands for Bus Interface Unit, i.e., the bus interface unit 310, which is used to implement interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0095] An instruction fetch buffer 309 connected to the controller 304 for storing instructions used by the controller 304;

[0096] The controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.

[0097] Generally, the unified memory 306, the input memory 301, the weight memory 302, and the instruction fetch memory 309 can all be on-chip memories. The external memory of the NPU can be a memory external to the NPU, and the external memory can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM), or other readable and writable memory. It should be understood that the chip hardware structure shown in FIG3 is only an exemplary description, and the present application is not limited thereto.

[0098] To facilitate understanding of the solutions of the embodiments of the present application, the technical terms involved in this document are first explained below.

[0099] A cascaded system (also known as a pipelined system) breaks down a task into multiple steps, each using a different model. For example, a cascaded speech translation system first converts speech into text using a speech recognition system, and then uses a text translation system to translate the text.

[0100] An end-to-end system (E2E system) uses a single model to complete the entire task. For example, an E2E speech translation system directly generates a text translation based on speech input.

[0101] Paired data and unpaired data. For example, speech recognition uses paired data such as "speech-transcribed text." Each data sample is a pair consisting of a piece of speech data and the corresponding text, called (speech, transcribed text). Another example is translation tasks, where a piece of Chinese text and its corresponding English translation form a pair consisting of (Chinese text, English translation). Conversely, when speech and text are not paired in terms of content, this is called unpaired data.

[0102] Labeled data refers to data with the labels required for the target task. For example, for speech recognition applications, the data is the audio of the speech, and the label is the text corresponding to the speech. Conversely, if the audio does not provide corresponding text, the speech is unlabeled data.

[0103] Spoken Language Understanding (SLU) extracts user intent and entity information related to the intent from user voice input. For example, if a user says "What's the weather like in Shenzhen?", SLU extracts the intent as "weather query" and the location as Shenzhen.

[0104] Supervised learning / unsupervised learning. Supervised learning refers to the method of training a model using labeled data. The goal of training is to enable the model to predict the corresponding label after receiving the data. Conversely, if the training data is completely unlabeled, the learning process is unsupervised learning.

[0105] Representation and latent space. After a multi-layer neural network model receives input, the output of each layer, except for the layer that outputs label predictions, can be called the representation of this model in the latent space, referred to as latent representation or representation, also known as features. The latent space representation of the model is generally more advanced and abstract than the original input, and is easier to process subsequently.

[0106] Language model training in related technologies often presents various challenges, particularly in speech models. For example, in related technology 1, speech pre-training model frameworks, such as wav2vec 2.0 and HuBERT, are pre-trained models specifically for speech data. They utilize unlabeled speech audio for pre-training, and the resulting base models help improve the performance of downstream speech-related tasks.

[0107] However, this solution: the trained speech pre-training model does not contain much high-level information such as semantics, and cannot effectively utilize information from other modalities (such as text) to help the performance of the speech pre-training model, which limits the performance of the speech pre-training model in helping downstream tasks.

[0108] Related technology 2 involves joint speech and text pre-training, such as A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (SLAM) and enhanced speech pre-training with unpaired textual data (SpeechLM). This research path typically requires the use of a small amount of paired speech and text data to guide the model's alignment learning of different modalities, ensuring the ultimate effectiveness of the joint speech and text pre-training model.

[0109] However, this approach lacks sufficient research on the direct interface between text and speech data, as they are two different modalities. This can cause the outputs of the two encoders to be in different spaces, exposing the shared encoder to transfer interference and capacity dilution. Furthermore, this approach utilizes a small amount of supervised data, which increases the cost of speech pre-training.

[0110] In response to the above problems, an embodiment of the present application provides a language model training method, which enables unpaired speech sample data and text sample data to be expressed in the same representation space, thereby enabling multimodal data training of the language model using speech sample data and text sample data. No speech-text pair data is required during the model training process, thereby reducing the model training cost and improving the model training effect.

[0111] The language model training method provided in the embodiment of the present application can be applied to the pre-training of the speech model, especially the pre-training of the speech LLM model. The following takes the speech LLM as an example to introduce the language model training method provided in the embodiment of the present application.

[0112] The following detailed description of the language model training method provided in an embodiment of the present application is provided in conjunction with FIG4 . This method can be implemented by the training device 220 in FIG2 , utilizing unpaired speech-to-text data to train an LLM capable of extracting high-level semantic knowledge. As shown in FIG4 , the language model training method provided in an embodiment of the present application includes at least steps S401 to S404.

[0113] In step S401, a training data set is obtained.

[0114] The training device obtains a training data set from a database. The training data set includes multiple speech-text pairs. Each speech-text pair includes speech sample data and text sample data. The speech sample data and text sample data in each speech text are unpaired data, that is, the speech sample data and text sample data in the speech-text pair have different semantics.

[0115] For example, the training data set D = {(S1, T1), (S2, T2), (S3, T3), ..., (S i ,T i ),…}, where (S i ,T i ) represents a set of unpaired speech S i and text T i .

[0116] Paired data is often scarce, difficult to obtain, or expensive, while unpaired data is often easy to obtain. For example, English speech recognition models typically require speech-labeled data such as (English speech and its corresponding English transcript). This type of data is often scarce, difficult to obtain, or expensive. Unpaired data such as (English speech and its corresponding English transcript) is highly paired and relatively easy to obtain.

[0117] For example, in a speech translation model, the training dataset (S1, T1) contains the speech S1 of "How will the weather be tomorrow" and the text T2 of "I love you China." This means that the speech S1 and text T2 are unpaired data. However, such unpaired data is readily available and reduces the cost of acquiring the training dataset. For example, unpaired speech and text data can be obtained online as training datasets.

[0118] In step S402, feature extraction is performed on the speech sample data and the text sample data respectively to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data. The first feature representation and the second feature representation belong to the same representation space.

[0119] Speech sample data and text sample data of different modalities are represented in the same representation space so that they can be mapped to the same discrete coding space. The discretized speech and text encodings can then be used together to train the LLM, allowing the LLM backbone network to benefit from both speech sample data and text sample data, thus avoiding the situation where the speech sample data and text sample data are not output in a unified representation space, causing the backbone network to be disturbed by transfer interference and capacity dilution.

[0120] Exemplarily, a unit generator corresponding to the modality can be constructed for sample data of different modalities, such as a speech unit generator for speech sample data and a text unit generator for text sample data, to process the sample data of different modalities into corresponding discrete data representations. For example, the speech unit generator is used to encode multiple speech frames in the speech sample data to obtain a speech feature representation (i.e., a first feature representation). For another example, the text unit generator is used to encode discrete tokens in the text sample data to obtain a text feature representation (i.e., a second feature representation). The speech feature representation and the text feature representation belong to a unified representation space.

[0121] The speech unit generator and the text unit generator are trained, and the speech sample data and the text sample data are mapped into the same representation space using the trained speech unit generator and the text unit generator.

[0122] In one example, the speech unit generator and the text unit generator can be trained by generating adversarial networks. In order to facilitate the training of the GAN network, the speech-text pairs (S i ,T i ) is annotated with a label when it is constructed, which indicates whether the feature distributions of the speech sample data and the text sample data in the speech-text pair are the same.

[0123] It should be noted that the feature distribution of the speech sample data and text sample data in the speech-text pair indicated by the label does not refer to a specific semantic feature distribution, but a broader feature distribution. For example, if the language model is a speech recognition model, the feature distribution refers to the grammatical / language feature distribution, which indicates whether the speech sample data and text sample data are of the same language, or whether the grammar of the speech sample data and text sample data is correct. For example, if the speech in a speech-text pair is Chinese and the text is English, the label is 0, that is, Indicates that the language feature distribution of the speech-text pair is different; if the speech in a speech-text pair is Chinese speech and the text is Chinese text, the label is 1, indicating that the language feature distribution of the speech-text pair is the same; for another example, if the speech in a speech-text pair is Chinese speech with correct grammar and the text is grammatically incorrect text, such as "I love Shenzhen", the label is 0, indicating that the grammatical feature distribution of the speech-text pair is different; if the speech in a speech-text pair is Chinese speech with correct grammar and the text is grammatically correct text, such as "I love Shenzhen", the label is 1, indicating that the grammatical feature distribution of the speech-text pair is the same.

[0124] Figure 5 shows a schematic diagram of the structure of the GAN network for training the speech unit generator and text unit generator mentioned in the embodiment of this application. As shown in Figure 5, the GAN network includes a speech unit generator, a text unit generator and a discriminator, wherein the speech unit generator and the text unit generator serve as the generators of the GAN network, and the text unit generator is used to generate a text feature representation H according to the input text sample data. i The speech unit generator is used to generate speech feature representation H′ according to the input speech text data i , the discriminator is used to discriminate the text feature representation H i and speech feature representation H′ i The probability distribution of the same representation space, if the probability is greater than the set threshold, the judgment text feature representation H is output i and speech feature representation H′ i Belong to the same representation space, output True, otherwise judge the text feature representation H i and speech feature representation H′ i If they do not belong to the same representation space, the output is False. Then, according to the label and the discriminator’s judgment results, the speech unit generator, text unit generator and discriminator are adjusted. i and speech feature representation H′ i The training direction is to generate text feature representation H that the discriminator discriminates as being in the same representation space i and speech feature representation H′ i , and the training direction of the discriminator is to try to detect the text feature representation H i and speech feature representation H′ i The features that are not in the same representation space can be improved through alternating optimization training, and finally a speech unit generator and a text unit generator with very good effect are obtained, that is, a text feature representation H can be extracted. i and speech feature representation H′ i Text unit generator and speech unit generator in the same representation space.

[0125] In one example, the training process of the speech unit generator and the text unit generator can be roughly divided into three steps.

[0126] Step 1: Use the GAN network to train to obtain the first speech unit generator (which can be called SG1).

[0127] For example, converting text sample data into a phoneme sequence; using the phoneme sequence as the input of an embedding network layer (e.g., an Embedding network layer), and outputting a text feature representation H i ; Take the speech sample data as the input of the speech unit generator SG1 to be trained, and output the speech feature representation H′i ; Represent the text feature H i and speech feature representation H′ i As the input of the discriminator, the output text feature representation H i and speech feature representation H′ i The probability distribution of the same representation space is obtained; and then the weight parameters of SG1 are adjusted according to the discrimination result and label of the discriminator until the training end condition is reached (for example, the training end condition such as the set number of training times or training convergence is reached) to obtain SG1.

[0128] For example, the G2P tool is used to convert text into a phoneme sequence. For example, the converted phoneme sequence of "I love Shenzhen" is w o3 / ai4 / sh en1 / zh en4 / , where the symbol " / " is used as a syllable separator. At the same time, in order to cope with the large mismatch in length between the voice sample data and the text sample data (that is, the number of pre-processed voice frames is generally much larger than the phoneme sequence length of the corresponding text), we uniformly sample the obtained phoneme sequence upward (for example, a sampling rate of 2 means that each phoneme id in the sequence will be copied twice); therefore, for the text input T i , which will be converted and upsampled into a phoneme sequence, and then input into an Embedding network layer, and the final output text feature is recorded as H i ;

[0129] For the speech input sample S i , the output of the last layer of the neural network after being processed by the first speech unit generator to be trained is the output speech feature, then H′ i Indicates that. The specific operation can be: first remove S i The silence at the beginning and end is removed, and then the speech signal after removing the silence is divided into frames and the features of each frame are extracted using the pre-trained wave2vec2.0 pre-training model as the input of the first speech unit generator to be trained.

[0130] Due to T i and S i It is not paired data, in order to make the final learned H i and H′ i In order to make the distribution as similar as possible (i.e., share a unified representation space), the GAN mechanism is used for training. Specifically, the first speech unit generator is used as the generator of the GAN, and the discriminator of the GAN is designed to determine H′ i Are they of the same H i The distribution is similar (ie, True), otherwise False; after training with all the training data using the above training method, the first version of the speech unit generator (ie, the first speech unit generator) can be obtained, which is recorded as SG1.

[0131] Optionally, the network structure of the first speech unit generator can be: a three-layer 1D convolutional neural network (CNN), with specific structural parameters of kernel size = 9, stride size = 3, and batchnorm1d layer dimension = 30. The discriminator can be a 1D CNN, with specific structural parameters of kernel size = 8, stride size = 1, and batchnorm1d layer dimension = 30

[0132] Step 2: Fix the weight parameters of SG1 and train the first text unit generator.

[0133] Since the G2P simulated text unit generator used in the previous step is not the best text representation unit in actual use scenarios, the purpose of this step is to train a text unit generator based on a neural network. The specific process is to fix the speech unit generator SG1 obtained in the previous step, that is, for each speech input S i , using SG1 to obtain a fixed output feature H′ i ; and for text input T i , then based on the G2P result, it is first converted into a finer-grained pseudo-phoneme representation (for example, "I love Shenzhen", the converted phoneme sequence is w o3 / ai4 / sh en1 / zh en4 / , and then the finer-grained pseudo-phoneme representation is wo 3 / ai 4 / shen 1 / zhen 4 / ), and then input into the first text unit generator. Same as step 1, this time still uses the GAN mechanism for training, where the first text unit generator serves as the GAN generator, and the GAN discriminator is designed to use it to determine H i Is it the same as H′? i The distribution is similar (ie, True), otherwise False; after training with all the training data using the training method, the first version of the text unit generator (ie, the first text unit generator) neural network model can be obtained, which is recorded as TG1.

[0134] Optionally, the network structure of the first text unit generator can be: three-layer 1D CNN, with specific structural parameters of kernel size = 9, stride size = 3, and batchnorm1d layer dimension = 30; the structure of the discriminator network is 1D CNN, with specific structural parameters of kernel size = 8, stride size = 1, and batchnorm1d layer dimension = 30.

[0135] Step 3: Jointly train SG1 and TG1.

[0136] Using the labeled training dataset D = {(S1, T1), (S2, T2), (S3, T3), ..., (S i ,T i ),…}, SG1 and TG1 are jointly retrained in the hope of learning a better unified speech-text representation space.

[0137] For example, SG1 and TG1 serve as generators of the GAN network, and TG1 is used to generate text feature representation H according to the input text sample data. i and SG1 are used to generate speech feature representation H′ based on input speech text data i , the discriminator is used to discriminate the text feature representation H i and speech feature representation H′ i The probability distribution of the same representation space, if the probability is greater than the set threshold, the judgment text feature representation H is output i and speech feature representation H′ i Belong to the same representation space, output True, otherwise judge the text feature representation H i and speech feature representation H′ i If they do not belong to the same representation space, the output is False. Then, according to the label and the discriminator’s judgment results, the speech unit generator, text unit generator and discriminator are adjusted. i and speech feature representation H′ i The training direction is to generate text feature representation H that the discriminator discriminates as being in the same representation space i and speech feature representation H′ i , and the training direction of the discriminator is to try to detect the text feature representation H i and speech feature representation H′ i The features that are not in the same representation space can be improved through alternating optimization training, and finally the target speech unit generator and target text unit generator with very good effect are obtained, that is, the text feature representation H can be extracted. i and speech feature representation H′ i The target text unit generator and the target speech unit generator are in the same representation space.

[0138] In step S403, the first feature representation and the second feature representation are mapped to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively.

[0139] A trained target speech unit generator is used to extract features from speech sample data in a training dataset D to obtain all speech feature representations. A trained target single text generator is used to extract features from text sample data in the training dataset D to obtain all text feature representations. All or part of the speech feature representations and sample feature representations are then clustered to obtain multiple clusters, and the identity documents (IDs) of the multiple clusters constitute a discrete coding space. The speech feature representations extracted by the target speech unit generator are mapped to the ID of the cluster with the smallest distance therefrom to obtain discrete speech codes (speech codes) corresponding to the speech sample data input; and the text feature representations extracted by the target text unit generator are mapped to the ID of the cluster with the smallest distance therefrom to obtain discrete text codes (text codes) corresponding to the text sample data input.

[0140] In one example, the speech feature representation and the text feature representation can be uniformly clustered to obtain a unified discrete coding space, and then the semantic feature representation and the text feature representation can be represented using the unified discrete coding space to obtain speech discrete coding and text discrete coding. For example, all speech sample data in the training data set D are input into a trained target speech unit generator to obtain a set of all speech features P, and all text sample data in the training data set D are input into a trained target text unit generator to obtain a set of all text features Q. 30% of the features are randomly extracted from the union of the speech feature set P and the text feature set Q to train a Kmeans (number of clusters = 500) clustering model. Using the trained Kmeans clustering model, all speech features and text features can be mapped into discrete Kmeans cluster IDs, which are respectively called speech discrete coding and text discrete coding.

[0141] In another example, clustering can also be performed separately for speech feature representation and text feature representation to obtain a first discrete coding space specifically for speech and a second discrete coding space specifically for text. For example, all speech sample data in the training data set D are input into a trained target speech unit generator to obtain a set of all speech features P, and all text sample data in the training data set D are input into a trained target text unit generator to obtain a set of all text features Q. 30% of the features are randomly extracted from the speech feature set P to train a Kmeans (number of clusters = 500) clustering model. The Kmeans clustering model is used to cluster all speech features into multiple speech feature clusters. The speech feature representation extracted by the target speech unit generator is mapped to the ID of the cluster with the smallest distance to it, and the speech discrete coding corresponding to the speech sample data input is obtained. The Kmeans clustering model is used to cluster all text features into multiple text feature clusters. The text feature representation extracted by the target text unit generator is mapped to the ID of the cluster with the smallest distance to it, and the text discrete coding corresponding to the text sample data input is obtained.

[0142] In step S404, the LLM to be trained is trained based on the first discrete codes and the second discrete codes of the plurality of speech-text pairs.

[0143] Through the trained target speech unit generator, target text unit generator, and Kmeans clustering model, all speech sample data and text sample data can be converted into discrete speech discrete codes and text discrete codes. As can be seen from the above description, since the speech feature representation output by the target speech unit generator and the text feature representation output by the target text unit generator are represented in the same representation space, the discrete code finally mapped by the clustering model is also a unified discrete code, which is then used to train the subsequent unit encoder and the LLM backbone network (i.e., the shared encoder) so that it can learn high-order semantic knowledge of different modalities. The final trained LLM can better serve downstream tasks.

[0144] In one example, dedicated encoders can be set for discretely encoded data of different modalities. For example, a speech unit encoder can be set for the speech encoding corresponding to the speech sample data, and a text unit encoder can be set for the text encoding corresponding to the text sample data. The shared encoder is trained using the speech precode output by the speech unit encoder and the text precode output by the text unit encoder.

[0145] Figure 6 shows a training diagram of an LLM. As shown in Figure 6, the LLM is provided with a modal-specific encoder, namely a speech unit encoder for speech, which can also be called a speech encoding network, a text unit encoder for text, which can also be called a text encoding network, and a shared encoder. The output of the speech unit encoder and the output of the text unit encoder are respectively connected to the input of the shared encoder. That is, the output of the speech unit encoder serves as the input of the shared encoder, and the output of the text unit encoder serves as the input of the shared encoder.

[0146] Exemplarily, the RoBERTa network is used as the backbone network of LLM (i.e., a shared encoder), and the speech unit encoder and text unit encoder are respectively the Embedding layers. The input speech code and text code need to be pre-encoded by the speech unit encoder and text unit encoder respectively before being input into the backbone network. Then different masking strategies are used for the pre-encoding of different modalities. For example, a masking operation of 15% is used for speech pre-encoding, and a masking operation of 20% is used for text pre-encoding, thereby further reducing the semantic differences between different modalities. The masked speech mask data and text mask data are then input into the backbone network of LLM. The backbone network extracts the high-order semantic features of the speech, predicts the masked data based on the high-order semantic features of the speech, and then calculates the MLM loss (masked language modeling loss) based on the prediction results and the actual masked data. The weight parameters of the speech unit encoder and the LLM backbone network are then adjusted according to the MLM loss. The backbone network extracts high-order semantic features of the text, predicts the masked data based on the high-order semantic features of the text, and then calculates the MLM loss based on the prediction results and the actual masked data. The weight parameters of the text unit encoder and the LLM backbone network are then adjusted according to the MLM loss.

[0147] Figure 7 shows another LLM training diagram. As shown in Figure 7, the LLM is equipped with a unit encoder, which can be called a speech / text unit encoder, for pre-encoding discrete speech codes / discrete text codes; and a shared encoder. The speech / text unit encoder.

[0148] Exemplarily, since the speech discrete code and text discrete code already share the same discrete space, there is no need to model encoders for them separately. Using the same encoder can also make LLM have a better training effect. Therefore, the speech discrete code and text discrete code are input into a speech / text unit encoder, which outputs speech precoding and text precoding. Then, a certain proportion of masking operations are performed on the speech precoding and text precoding to obtain speech mask data and text mask data. The speech mask data and text mask data are input into the LLM backbone network. The backbone network extracts high-order semantic features of the speech, predicts the masked data based on the high-order semantic features of the speech, and then calculates the MLM loss based on the predicted results and the actual masked data. The weight parameters of the speech unit encoder and the LLM backbone network are then adjusted based on the MLM loss. The backbone network extracts high-order semantic features of the text, predicts the masked data based on the high-order semantic features of the text, and then calculates the MLM loss based on the predicted results and the actual masked data. The weight parameters of the text unit encoder and the LLM backbone network are then adjusted based on the MLM loss.

[0149] The language model training method provided in the embodiment of the present application, in the discretization stage, maps unpaired speech and text to a unified discrete representation space by utilizing a GAN network and iterative training technology, so that the input data of different modalities are aligned in representation, and then used to train the LLM. On the one hand, the LLM can simultaneously learn high-order semantic knowledge of speech and text, thereby achieving better performance in speech downstream tasks compared to a single speech modality LLM. On the other hand, it achieves completely unsupervised training without the need for paired data, thereby reducing training costs.

[0150] Figure 8 shows a schematic diagram of the processing process of the input data by the LLM mentioned in the embodiment of the present application. As shown in Figure 8, LLM includes a speech unit generator for speech input data and a text unit generator for text input data, as well as a speech unit encoder for pre-coding speech and a text unit generator for pre-coding text, and a shared encoder. Among them, the speech unit generator and the text unit generator serve as discretization modules (tokenizers) for mapping speech input data and text input data into a unified discrete representation space for discrete coding. The speech unit encoder and the text encoder serve as specific modal encoders for pre-coding speech discrete coding and text discrete coding, respectively. The shared encoder serves as the backbone network of the LLM for further feature extraction of speech precoding and text precoding to obtain a unified universal feature representation (universal representation).

[0151] During the model inference phase, a pre-trained LLM can extract a common representation for input data of different modalities, representing them in the same representation space. For example, when the input data is speech data, the input speech data is used as the input to the speech unit generator to obtain a speech discrete code. The speech discrete code output by the speech unit generator is used as the input to the speech unit encoder to obtain a speech precode. The speech precode is used as the input to the shared encoder, which outputs a speech feature vector containing high-order semantic knowledge. When the input data is text data, the input text data is used as the input to the text unit generator to obtain a text discrete code. The text discrete code output by the text unit generator is used as the input to the text unit encoder to obtain a text precode. The text precode is used as the input to the shared encoder, which outputs a text feature vector containing high-order semantic knowledge. The speech feature representation and the text feature representation belong to the same representation space. The downstream task model of the LLM implements downstream tasks based on the speech feature vector or the text feature vector.

[0152] Based on the same concept as the embodiment of the language model training method described above, the present application also provides a language model training device 900. The language model training device 900 can be deployed on any device, equipment, platform, or device cluster with computing capabilities to implement the language model training method provided in the embodiment of the present application, so as to train the language model using unpaired speech sample data and text sample data. The language model training device 900 includes units or modules for implementing each step in the language model training method shown in Figures 4-8.

[0153] FIG9 is a structural diagram of a training device for a language model provided by an embodiment of the present application. As shown in FIG9 , the training device 900 for a language model includes at least an acquisition module 901, a feature extraction module 902, a mapping module 903, and a training module 904, wherein the acquisition module 901 is used to acquire a training data set, the training data set including a plurality of speech-text pairs, the speech-text pairs including speech sample data and text sample data, and the speech sample data and the text sample data have different semantics; the feature extraction module 902 is used to perform feature extraction on the speech sample data and the text sample data respectively, to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, the first feature representation and the second feature representation belonging to the same representation space; the mapping module 903 is used to map the first feature representation and the second feature representation to a discrete coding space, to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data; the training module 904 is used to train the language model to be trained based on the first discrete code and the second discrete code of the plurality of speech-text pairs.

[0154] In one possible implementation, the feature extraction module 902 is specifically used to: use the speech sample data as the input of the target speech unit generator and output a first feature representation; use the text sample data as the input of the target text unit generator and output a second feature representation; wherein the target speech unit generator and the target text unit generator are obtained by joint iterative training based on the generative adversarial network mechanism.

[0155] In another possible implementation, each speech-text pair in a plurality of speech-text pairs has a label, and the label indicates whether the feature distributions of the speech sample data and the text sample data in the speech-text pair are the same; the target speech unit generator and the target text unit generator are trained based on the following steps: taking the speech sample data as the input of the first speech unit generator, and outputting a first speech feature representation; taking the text sample data as the input of the first text unit generator, and outputting a first text feature representation; taking the first speech feature representation and the first text feature representation as the input of the discriminator, and outputting a first probability distribution that the first speech feature representation and the first text feature representation belong to the same representation space; based on the first probability distribution and the label, adjusting the weight parameters of the first speech unit generator and the weight parameters of the first text unit generator to obtain the target speech unit generator and the target text unit generator.

[0156] In another possible implementation, the first speech unit generator is trained based on the following steps: converting text sample data into a phoneme sequence; using the phoneme sequence as the input of the embedding network layer and outputting a second text feature representation; using the speech sample data as the input of the first speech unit generator to be trained and outputting a second speech feature representation; using the second speech feature representation and the second text feature representation as the input of the discriminator and outputting a third probability distribution of the second speech feature representation and the second text feature representation belonging to the same representation space; based on the third probability distribution and the label, adjusting the weight parameters of the first speech unit generator to be trained to obtain the first speech unit generator.

[0157] In another possible implementation, the first text unit generator is trained based on the following steps: fixing the weight parameters of the first speech unit generator, taking the speech sample data as the input of the first speech unit generator, and outputting the third speech feature representation; converting the text sample data into a phoneme sequence; taking the phoneme sequence as the input of the first text unit generator to be trained, and outputting the third text feature representation; taking the third speech feature representation and the third text feature representation as the input of the discriminator, and outputting the third probability distribution of the third speech feature representation and the third text feature representation belonging to the same representation space; based on the third probability distribution and the label, adjusting the weight parameters of the first text unit generator to be trained to obtain the first text unit generator.

[0158] In another possible implementation, the mapping module 903 is specifically used to: map the first feature representation to a first discrete coding space to obtain a first discrete code, the first discrete coding space includes identifiers of multiple first clusters, and the multiple first clusters are clustered based on multiple first feature representations corresponding to multiple speech sample data in multiple speech-text pairs; map the second feature representation to a second discrete coding space to obtain a second discrete code, the second discrete coding space includes identifiers of multiple second clusters, and the multiple second clusters are clustered based on multiple second feature representations corresponding to multiple text sample data in multiple speech-text pairs.

[0159] In another possible implementation, the mapping module 903 is specifically used to: map the first feature representation and the second feature representation to the same discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, wherein the discrete coding space includes identifiers of multiple clusters, and the multiple clusters are obtained by clustering multiple first feature representations corresponding to multiple speech sample data in multiple speech-text pairs and multiple second feature representations corresponding to multiple text sample data.

[0160] In another possible implementation, the language model includes a speech coding network, a text coding network and a first shared coding network; the training module 904 is specifically used to: use the first discrete code as the input of the speech coding network, and output the first speech precode; use the second discrete code as the input of the text coding network, and output the first text precode; mask the first speech precode to obtain first mask data; mask the first text precode to obtain second mask data; use the first mask data as the input of the first shared coding network, and output a first feature vector; predict at least one masked code in the first mask data through the first feature vector, and adjust the weight parameters of the speech coding network and the first shared coding network based on the prediction result; use the second mask data as the input of the first shared coding network, and output a second feature vector; predict at least one masked code in the second mask data through the second feature vector, and adjust the weight parameters of the text coding network and the first shared coding network based on the prediction result.

[0161] In another possible implementation, a masking strategy for performing masking on the first discrete code is different from a masking strategy for performing masking on the second discrete code.

[0162] In another possible implementation, the language model includes a speech-text encoding network and a second shared encoding network; the training module 904 is specifically used to: use the first discrete code and the second discrete code as inputs of the speech-text encoding network, and output a second speech precode and a second text precode; perform masking on the second speech precode and the second text precode, respectively, to obtain third mask data and fourth mask data; use the third mask data as input of the second shared encoding network, and output a third feature vector; predict at least one masked code in the third mask data through the third feature vector, and adjust the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result; use the fourth mask data as input of the second shared encoding network, and output a fourth feature vector; predict at least one masked code in the fourth mask data through the fourth feature vector, and adjust the weight parameters of the speech-text encoding network and the second shared encoding network based on the prediction result.

[0163] The language model training device 900 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the language model training device 900 are respectively for implementing the corresponding processes of each method in Figures 4-8. For the sake of brevity, they will not be repeated here.

[0164] An embodiment of the present application also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is configured to execute the methods described in Figures 4-8.

[0165] FIG10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application.

[0166] As shown in Figure 10, the computing device 1000 includes at least one processor 1001, a memory 1002 and a communication interface 1003. The processor 1001, the memory 1002 and the communication interface 1003 are communicatively connected, and the communication connection can be achieved by a wired manner (such as a bus) or by a wireless manner. The communication interface 1003 is used to send and / or receive data sent by other devices; the memory 1002 stores computer instructions, and the processor 1001 executes the computer instructions to execute the language model training method in the aforementioned method embodiment to achieve the expression of unpaired speech sample data and text sample data in the same representation space, thereby realizing the multimodal data training of the language model using speech sample data and text sample data.

[0167] It should be understood that in the embodiment of the present application, the processor 1001 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0168] The memory 1002 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1001. The memory 1002 may also include a nonvolatile random access memory.

[0169] The memory 1002 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0170] It should be understood that the computing device 1000 according to the embodiment of the present application can execute the method shown in Figures 4-8 in the embodiment of the present application. The detailed description of the implementation of the method is given above and will not be repeated here for the sake of brevity.

[0171] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned method is implemented.

[0172] An embodiment of the present application provides a chip, which includes at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0173] An embodiment of the present application provides a computer program or a computer program product, which includes instructions. When the instructions are executed, the computer is caused to perform the above-mentioned method.

[0174] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0176] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A method for training a language model, characterized in that: include: Acquire a training data set, the training data set including a plurality of speech-text pairs, the speech-text pairs including speech sample data and text sample data, the speech sample data and the text sample data having different semantics; Performing feature extraction on the speech sample data and the text sample data respectively to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, wherein the first feature representation and the second feature representation belong to the same representation space; Mapping the first feature representation and the second feature representation to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively; The language model to be trained is trained based on the first discrete codes and the second discrete codes of the multiple speech-text pairs.

2. The method according to claim 1, characterized in that The performing feature extraction on the speech sample data and the text sample data respectively to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data includes: Using the speech sample data as input to a target speech unit generator, and outputting the first feature representation; Taking the text sample data as input of a target text unit generator and outputting the second feature representation; The target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism.

3. The method according to claim 2, characterized in that Each of the plurality of speech-text pairs has a label, wherein the label indicates whether the feature distributions of speech sample data and text sample data in the speech-text pair are the same; The target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism, including: Using the speech sample data as input to a first speech unit generator, and outputting the first speech feature representation; Taking the text sample data as input of a first text unit generator and outputting the first text feature representation; Taking the first speech feature representation and the first text feature representation as inputs of a discriminator, and outputting a first probability distribution of the first speech feature representation and the first text feature representation belonging to the same representation space; Based on the first probability distribution and the label, the weight parameters of the first speech unit generator and the weight parameters of the first text unit generator are adjusted to obtain the target speech unit generator and the target text unit generator.

4. The method according to claim 3, characterized in that The first speech unit generator is trained based on the following steps: Converting the text sample data into a phoneme sequence; Taking the phoneme sequence as input of an embedding network layer and outputting a second text feature representation; Using the speech sample data as input to a first speech unit generator to be trained, and outputting a second speech feature representation; Taking the second speech feature representation and the second text feature representation as inputs of the discriminator, and outputting a second probability distribution of the second speech feature representation and the second text feature representation belonging to the same representation space; Based on the second probability distribution and the label, the weight parameters of the first speech unit generator to be trained are adjusted to obtain the first speech unit generator.

5. The method according to claim 3 or 4, characterized in that The first text unit generator is trained based on the following steps: Fixing the weight parameters of the first speech unit generator, taking the speech sample data as input of the first speech unit generator, and outputting the third speech feature representation; Converting the text sample data into a phoneme sequence; Taking the phoneme sequence as input of a first text unit generator to be trained, and outputting a third text feature representation; Taking the third speech feature representation and the third text feature representation as inputs of the discriminator, and outputting a third probability distribution of the third speech feature representation and the third text feature representation belonging to the same representation space; Based on the third probability distribution and the label, the weight parameters of the first text unit generator to be trained are adjusted to obtain the first text unit generator.

6. The method according to any one of claims 1 to 5, characterized in that Mapping the first feature representation and the second feature representation to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively, includes: Mapping the first feature representation to a first discrete code space to obtain the first discrete code, wherein the first discrete code space includes identifiers of a plurality of first clusters, and the plurality of first clusters are obtained by clustering a plurality of first feature representations corresponding to a plurality of speech sample data in the plurality of speech-to-text pairs; The second feature representation is mapped to a second discrete coding space to obtain a second discrete code, wherein the second discrete coding space includes identifiers of multiple second clusters, and the multiple second clusters are obtained by clustering multiple second feature representations corresponding to multiple text sample data in the multiple speech-text pairs.

7. The method according to any one of claims 1 to 5, characterized in that Mapping the first feature representation and the second feature representation to a discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively, includes: The first feature representation and the second feature representation are mapped to the same discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, wherein the discrete coding space includes identifiers of multiple clusters, and the multiple clusters are obtained by clustering the multiple first feature representations corresponding to the multiple speech sample data and the multiple second feature representations corresponding to the multiple text sample data in the multiple speech-text pairs.

8. The method according to any one of claims 1 to 7, characterized in that The language model includes a speech encoding network, a text encoding network and a first shared encoding network; The training of the language model to be trained based on the first discrete codes and the second discrete codes of the plurality of speech-text pairs includes: Using the first discrete code as input to the speech coding network, and outputting a first speech precode; Using the second discrete code as input of the text encoding network, and outputting a first text precode; performing mask processing on the first speech precoding to obtain first mask data; performing mask processing on the first text precoding to obtain second masked data; Using the first mask data as input to the first shared encoding network, and outputting a first feature vector; Predicting at least one masked code in the first mask data using the first feature vector, and adjusting weight parameters of the speech coding network and the first shared coding network based on the prediction result; Using the second mask data as input to the first shared encoding network, and outputting a second feature vector; At least one masked code in the second mask data is predicted using the second feature vector, and weight parameters of the text encoding network and the first shared encoding network are adjusted based on the prediction result.

9. The method according to claim 8, characterized in that A masking strategy for performing masking processing on the first speech precoding is different from a masking strategy for performing masking processing on the first text precoding.

10. The method according to any one of claims 1 to 7, characterized in that The language model includes a speech-to-text encoding network and a second shared encoding network; The training of the language model to be trained based on the first discrete codes and the second discrete codes of the plurality of speech-text pairs includes: respectively taking the first discrete code and the second discrete code as inputs of the speech-to-text coding network, and outputting a second speech precode and a second text precode; performing mask processing on the second speech precoding and the second text precoding respectively to obtain third masked data and fourth masked data; Using the third mask data as input to the second shared encoding network, and outputting a third feature vector; Predicting at least one masked code in the third mask data using the third feature vector, and adjusting weight parameters of the speech-to-text encoding network and the second shared encoding network based on a result of the prediction; Using the fourth mask data as input to the second shared encoding network, and outputting a fourth feature vector; At least one masked code in the fourth mask data is predicted using the fourth feature vector, and weight parameters of the speech-to-text encoding network and the second shared encoding network are adjusted based on the prediction result.

11. The method according to any one of claims 1 to 10, characterized in that The language model is a large language model.

12. A language model training device, characterized in that: include: An acquisition module is used to acquire a training data set, wherein the training data set includes a plurality of speech-text pairs, wherein the speech-text pairs include speech sample data and text sample data, and the speech sample data and the text sample data have different semantics; a feature extraction module, configured to perform feature extraction on the speech sample data and the text sample data, respectively, to obtain a first feature representation corresponding to the speech sample data and a second feature representation corresponding to the text sample data, wherein the first feature representation and the second feature representation belong to the same representation space; A mapping module, configured to map the first feature representation and the second feature representation to a discrete code space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, respectively; A training module is used to train the language model to be trained based on the first discrete codes and the second discrete codes of the multiple speech-text pairs.

13. The device according to claim 12, characterized in that The feature extraction module is specifically used for: Using the speech sample data as input to a target speech unit generator, and outputting the first feature representation; Taking the text sample data as input of a target text unit generator and outputting the second feature representation; The target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism.

14. The device according to claim 13, characterized in that Each of the plurality of speech-text pairs has a label, wherein the label indicates whether the feature distributions of speech sample data and text sample data in the speech-text pair are the same; The target speech unit generator and the target text unit generator are obtained by joint iterative training based on a generative adversarial network mechanism, including: Using the speech sample data as input to a first speech unit generator, and outputting the first speech feature representation; Taking the text sample data as input of a first text unit generator and outputting the first text feature representation; Taking the first speech feature representation and the first text feature representation as inputs of a discriminator, and outputting a first probability distribution of the first speech feature representation and the first text feature representation belonging to the same representation space; Based on the first probability distribution and the label, the weight parameters of the first speech unit generator and the weight parameters of the first text unit generator are adjusted to obtain the target speech unit generator and the target text unit generator.

15. The device according to claim 14, characterized in that The first speech unit generator is trained based on the following steps: Converting the text sample data into a phoneme sequence; Taking the phoneme sequence as input of an embedding network layer and outputting a second text feature representation; Using the speech sample data as input to a first speech unit generator to be trained, and outputting a second speech feature representation; Taking the second speech feature representation and the second text feature representation as inputs of the discriminator, and outputting a second probability distribution of the second speech feature representation and the second text feature representation belonging to the same representation space; Based on the second probability distribution and the label, the weight parameters of the first speech unit generator to be trained are adjusted to obtain the first speech unit generator.

16. The device according to claim 14 or 15, characterized in that The first text unit generator is trained based on the following steps: Fixing the weight parameters of the first speech unit generator, taking the speech sample data as input of the first speech unit generator, and outputting the third speech feature representation; Converting the text sample data into a phoneme sequence; Taking the phoneme sequence as input of a first text unit generator to be trained, and outputting a third text feature representation; Taking the third speech feature representation and the third text feature representation as inputs of the discriminator, and outputting a third probability distribution of the third speech feature representation and the third text feature representation belonging to the same representation space; Based on the third probability distribution and the label, the weight parameters of the first text unit generator to be trained are adjusted to obtain the first text unit generator.

17. The device according to any one of claims 12 to 16, characterized in that The mapping module is specifically used for: Mapping the first feature representation to a first discrete code space to obtain the first discrete code, wherein the first discrete code space includes identifiers of a plurality of first clusters, and the plurality of first clusters are obtained by clustering a plurality of first feature representations corresponding to a plurality of speech sample data in the plurality of speech-to-text pairs; The second feature representation is mapped to a second discrete coding space to obtain a second discrete code, wherein the second discrete coding space includes identifiers of multiple second clusters, and the multiple second clusters are obtained by clustering multiple second feature representations corresponding to multiple text sample data in the multiple speech-text pairs.

18. The device according to any one of claims 12 to 16, characterized in that The mapping module is specifically used for: The first feature representation and the second feature representation are mapped to the same discrete coding space to obtain a first discrete code corresponding to the speech sample data and a second discrete code corresponding to the text sample data, wherein the discrete coding space includes identifiers of multiple clusters, and the multiple clusters are obtained by clustering the multiple first feature representations corresponding to the multiple speech sample data and the multiple second feature representations corresponding to the multiple text sample data in the multiple speech-text pairs.

19. The device according to any one of claims 12 to 18, characterized in that The language model includes a speech encoding network, a text encoding network and a first shared encoding network; The training module is specifically used for: Using the first discrete code as input to the speech coding network, and outputting a first speech precode; Using the second discrete code as input of the text encoding network, and outputting a first text precode; performing mask processing on the first speech precoding to obtain first mask data; performing mask processing on the first text precoding to obtain second masked data; Using the first mask data as input to the first shared encoding network, and outputting a first feature vector; Predicting at least one masked code in the first mask data using the first feature vector, and adjusting weight parameters of the speech coding network and the first shared coding network based on the prediction result; Using the second mask data as input to the first shared encoding network, and outputting a second feature vector; At least one masked code in the second mask data is predicted using the second feature vector, and weight parameters of the text encoding network and the first shared encoding network are adjusted based on the prediction result.

20. The device according to claim 19, characterized in that A masking strategy for performing masking processing on the first speech precoding is different from a masking strategy for performing masking processing on the first text precoding.

21. The device according to any one of claims 12 to 18, characterized in that The language model includes a speech-to-text encoding network and a second shared encoding network; The training module is specifically used for: respectively taking the first discrete code and the second discrete code as inputs of the speech-to-text coding network, and outputting a second speech precode and a second text precode; performing mask processing on the second speech precoding and the second text precoding respectively to obtain third masked data and fourth masked data; Using the third mask data as input to the second shared encoding network, and outputting a third feature vector; Predicting at least one masked code in the third mask data using the third feature vector, and adjusting weight parameters of the speech-to-text encoding network and the second shared encoding network based on a result of the prediction; Using the fourth mask data as input to the second shared encoding network, and outputting a fourth feature vector; At least one masked code in the fourth mask data is predicted using the fourth feature vector, and weight parameters of the speech-to-text encoding network and the second shared encoding network are adjusted based on the prediction result.

22. The device according to any one of claims 12 to 21, characterized in that The language model is a large language model.

23. A computing device comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 11 is implemented.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

25. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images