Adaptive hierarchical representation alignment training method and device for large voice model

Abstract

The invention provides an adaptive hierarchical representation alignment training method and device for a voice large model, and relates to the technical field of natural language processing. The method comprises the following steps: based on an internal voice adapter, according to a candidate voice large model, performing model training by using a source voice and a text cue word to obtain a first voice large model; based on a cross-modal semantic retrieval task, according to the source voice and the source voice transcription text, performing semantic retrieval capability screening on the first voice large model to obtain an optimal neural network hierarchy; based on the optimal neural network hierarchy, according to the source voice, the text cue word and the source voice transcription text, performing prediction through a first voice large model, and calculating model prediction loss; performing parameter optimization on the first voice large model according to the model prediction loss to obtain a second voice large model; according to the adaptive hierarchical representation alignment training method, the mapping relation between the source voice and the transcriptional text is fully utilized, and the adaptive hierarchical representation alignment training method is efficient and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an adaptive hierarchical representation alignment training method and device for a large speech model. Background Art

[0002] The current training method for large speech models mainly relies on paired speech and text data. The speech part of this data pair is usually in the form of natural language audio, and the text part is a text description associated with the semantics of the speech. The training process introduces a series of speech and language understanding tasks such as speech recognition tasks and speech annotation tasks, so that the model can predict the corresponding text semantic information when receiving speech input, thereby achieving implicit alignment of speech and text modalities.

[0003] The training paradigms of large speech models are mainly divided into two categories. One category uses the structural characteristics of the large model itself and mature training paradigms. Based on the existing text pre-trained large model, the model vocabulary is expanded to enable the model to recognize and process discrete speech units (such as phonemes, words, subwords, etc.) in the speech signal. In this process, the model not only models the text content, but also needs to convert the speech signal into discrete representation units during the training process and predict the text associated with it. This training method requires the use of a large amount of speech and text data for continued pre-training, while training the embedded representation of the expanded part of the large model vocabulary and the parameters inside the large model, so that the text large model can finally gain speech understanding capabilities.

[0004] Another type of training paradigm introduces a speech pre-training model, so that the speech input can first be converted into a continuous representation rich in semantic information through a dedicated speech understanding module (such as a speech pre-training model). The task of the speech pre-training model is to extract high-level, semantically rich features from the original speech signal, which can better represent the semantic content of the speech signal. The continuous representation is transformed into a representation space through a trainable adapter so that the converted speech representation can be understood by the large model. This training method uses only fewer speech-text data pairs than the previous one, so that the large text model can understand the speech signal input without training the original parameters of the large text model.

[0005] Although existing large speech models can initially complete speech-related understanding and generation tasks, there is still a significant performance gap compared to similar tasks in text scenarios (such as speech translation and machine translation). There is a monotonic mapping relationship between speech and text, and existing large speech model training technology does not explicitly utilize the close connection between speech and text, but only implicitly models it as a supervisory signal for text generation, failing to further narrow the modal gap between speech and text, hindering the in-depth use of speech data in complex application scenarios. Existing research lacks exploration of the internal understanding mechanism of speech models, which not only limits the in-depth understanding of the working principles of the models, but also hinders the ability to further optimize and improve these models.

[0006] In the prior art, there is a lack of an efficient and accurate adaptive hierarchical representation alignment training method that fully utilizes the mapping relationship between source speech and transcribed text. Summary of the invention

[0007] In order to solve the technical problems existing in the prior art, such as the difference between the source speech and the transcribed text modalities, the difference between the performance of similar tasks under different modalities, and the limitation of understanding information of different modalities, the embodiment of the present invention provides a method and device for adaptive hierarchical representation alignment training of a large speech model. The technical solution is as follows:

[0008] On the one hand, a method for adaptive hierarchical representation alignment training of a large speech model is provided, the method being implemented by an adaptive hierarchical representation alignment training device, the method comprising:

[0009] Acquire source speech and source speech target text; input the source speech into the candidate speech large model for text transcription to obtain the source speech transcription text;

[0010] Based on the internal speech adapter, according to the candidate speech large model, the source speech and the text prompt word are used to perform model training to obtain a first speech large model;

[0011] Based on the cross-modal semantic retrieval task, the semantic retrieval capability of multiple neural network layers of the first large speech model is screened according to the source speech and the source speech transcription text to obtain the optimal neural network layer for semantic representation alignment;

[0012] Based on the optimal neural network level, the source speech and the text prompt word are input into the first speech large model for text prediction to obtain speech representation, speech attention weight matrix and predicted text;

[0013] Based on the optimal neural network level, the source speech transcription text and the text prompt words are input into the first speech macro model to extract text representation, and obtain the optimal text representation and text attention weight matrix;

[0014] A loss function is calculated based on the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain a model prediction loss;

[0015] According to the model prediction loss, the first large speech model is optimized in parameters to obtain a second large speech model.

[0016] On the other hand, a device for adaptive hierarchical representation alignment training of a large speech model is provided, and the device is applied to an adaptive hierarchical representation alignment training method of a large speech model, and the device comprises:

[0017] A data acquisition module is used to acquire source speech and source speech target text; the source speech is input into the candidate speech large model for text transcription to obtain the source speech transcription text;

[0018] A model training module, configured to perform model training based on an internal speech adapter and the candidate speech large model using the source speech and the text prompt words to obtain a first speech large model;

[0019] A cross-modal semantic retrieval module, for performing semantic retrieval capability screening on multiple neural network layers of the first large speech model based on the cross-modal semantic retrieval task and according to the source speech and the source speech transcription text, to obtain the optimal neural network layer for semantic representation alignment;

[0020] A text prediction module, configured to input the source speech and the text prompt word into the first speech large model for text prediction based on the optimal neural network layer, and obtain speech representation, speech attention weight matrix and predicted text;

[0021] A text representation extraction module, for inputting the source speech transcription text and the text prompt words into the first speech macro model to extract text representation based on the optimal neural network layer, and obtaining an optimal text representation and a text attention weight matrix;

[0022] A loss calculation module calculates a loss function according to the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain a model prediction loss;

[0023] The model optimization module is used to optimize the parameters of the first large speech model according to the model prediction loss to obtain the second large speech model.

[0024] On the other hand, an adaptive hierarchical representation alignment training device is provided, which comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the adaptive hierarchical representation alignment training methods for the large speech model described above is implemented.

[0025] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned adaptive hierarchical representation alignment training methods for a large speech model.

[0026] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0027] The present invention proposes an adaptive hierarchical representation alignment training method for a large speech model. By introducing an internal speech adapter and an adaptive hierarchical optimization selection strategy, the semantic understanding and representation alignment capabilities of the large speech model in cross-modal speech-text tasks are significantly improved, and the performance of the model is enhanced. The adaptive hierarchical optimization selection strategy allows the model to automatically select the optimal layer for training according to the actual semantic understanding ability, avoiding resource waste, accelerating the convergence speed, and improving the training efficiency. By constructing the first representation loss, the difference between the speech representation and the text representation is directly minimized, ensuring a high degree of alignment between the two at the semantic level, and improving the model's understanding and robustness for complex semantic structures. By utilizing the natural mapping relationship between the source speech and the transcribed text, by splicing the speech and text prompt words into the model, the deep fusion of the two modal representations is promoted, the semantic parsing ability is enhanced, and the modal gap between speech and text is effectively narrowed. The present invention is an efficient and accurate adaptive hierarchical representation alignment training method that fully utilizes the mapping relationship between the source speech and the transcribed text. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of an adaptive hierarchical representation alignment training method for a large speech model provided by an embodiment of the present invention;

[0030] Figure 2 is a structural schematic diagram of an internal speech adapter provided by an embodiment of the present invention;

[0031] Figure 3It is a block diagram of an adaptive hierarchical representation alignment training device for a large speech model provided by an embodiment of the present invention;

[0032] Figure 4 It is a structural schematic diagram of an adaptive hierarchical representation alignment training device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0034] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0035] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0036] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0037] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0038] The embodiment of the present invention provides an adaptive hierarchical representation alignment training method for a large speech model, which can be implemented by an adaptive hierarchical representation alignment training device, which can be a terminal or a server. Figure 1 The flowchart of the adaptive hierarchical representation alignment training method for a large speech model is shown. The processing flow of the method may include the following steps:

[0039] S1. Obtain source speech and target text of the source speech; input the source speech into the candidate speech large model for text transcription to obtain the source speech transcription text.

[0040] In a feasible implementation manner, in the present invention, source speech refers to the original speech data at the input end, and speech data refers to the sound signal generated by human vocal cord vibrations recognized and processed by a computer system, which is a carrier of natural language. Each type of speech itself represents a corresponding type of natural language, such as Chinese speech, English speech, etc.

[0041] The source speech transcription text refers to the text data of the first natural language type corresponding to the source speech, and the natural language generally refers to the spoken language that has evolved naturally and culturally, such as Chinese, English, Japanese, German, etc. The first natural language can be implemented as any natural language. Taking the first natural language as Chinese as an example, the text data of the first natural language type can be "Hello!".

[0042] The target text refers to the text data that is expected to be generated or converted after the source speech or the source speech transcription text is input into the speech big model. The specific form and content of the target text depends on the task and application scenario of the speech big model. The second natural language type of the target text can be the same as the first natural language type of the source speech transcription text, or it can be different from the first natural language type of the source speech transcription text.

[0043] Acquire speech and transcribed text data of a first natural language type and target text data of a second natural language type in the speech domain from a database.

[0044] Alternatively, any speech data of the first natural language type and target text data of the second natural language are obtained from the database, and the speech data are transcribed using a professional speech recognition model to obtain corresponding source speech transcribed text data of the first natural language type.

[0045] In addition, due to the differences in pronunciation habits and pronunciation methods in different regions, the same text natural language scenario corresponds to different speech natural language scenarios, such as Cantonese, Minnan dialect, Mandarin and other speech natural languages ​​corresponding to the Chinese text natural language. If the natural language type of the source speech is a non-written language form, the source speech transcription text should be similar to the natural language type of the source speech or be in the same language system. For example, the source speech is Minnan dialect, and Minnan dialect and Chinese belong to the Chinese branch of the Sino-Tibetan language family, then the source speech transcription text can be a Chinese text. For example, the first natural language of the source speech is Chinese speech, and the first natural language of the source speech transcription text is Chinese, then the trained speech large model can realize the recognition, processing and analysis tasks of Chinese speech.

[0046] S2. Based on the internal speech adapter, according to the candidate speech large model, the source speech and the text prompt words are used to perform model training to obtain the first speech large model.

[0047] Optionally, based on the internal speech adapter, according to the candidate speech large model, the source speech and the text prompt words are used to perform model training to obtain the first speech large model, including:

[0048] Inserting the internal speech adapter into the candidate speech big model to obtain an improved candidate language big model;

[0049] Using the source speech and the text prompt words, the improved candidate speech model is trained to obtain the first speech model;

[0050] The language type of the text prompt is the same as the source speech transcription text.

[0051] In a feasible implementation, in the present invention, the candidate speech model includes a candidate speech encoder, a candidate feature adapter, a candidate text generation model, etc. Among them, the candidate speech encoder is responsible for signal processing and feature extraction of the source speech; the candidate feature adapter is used to convert the features extracted by the speech encoder; the candidate text generation model is responsible for understanding the speech representation after the feature adapter and generating text. When the source speech is input into the candidate speech model, the starting position and the ending position of the speech input will be recorded for the subsequent training of the internal speech adapter.

[0052] The training of the candidate speech model is generally a speech recognition task, that is, the source speech and text prompt words are input, and the model is required to output the transcribed text corresponding to the source speech, and the first speech model is obtained after training. The text prompt word refers to the text instruction input of the speech model, which is the same natural language type as the source speech transcribed text, that is, the first natural language type of the text prompt word is the same as the first natural language type of the source speech transcribed text. The instruction guides the speech model to complete a series of recognition, processing, analysis and text generation tasks of the source speech.

[0053] The structure of the internal speech adapter includes two linear mapping layers, a nonlinear activation function, a residual connection and a mean square error regularization module.

[0054] In one possible implementation, the internal speech adapter structure is as follows: Figure 2 As shown, the internal speech is inserted after each neural network layer, and its scope is only for the speech representation part in the concatenated representation.

[0055] The input representation is first reduced in dimension by the first linear mapping layer, and then obtained through a nonlinear activation function to obtain the nonlinear mapping capability of a high-dimensional space; it is increased in dimension by the second linear mapping layer; the input representation is added to the representation after dimensionality increase through a residual connection, and then output through a root mean square normalization module. According to the starting position and the ending position in the source speech encoding process, the speech representation of the corresponding position is obtained, and the part of the speech representation is converted, and the converted representation replaces the previous position, and the replaced representation continues to be input into the next layer of the candidate speech model.

[0056] S3. Based on the cross-modal semantic retrieval task, the semantic retrieval capabilities of multiple neural network layers of the first large speech model are screened according to the source speech and the source speech transcribed text to obtain the optimal neural network layer for semantic representation alignment.

[0057] Optionally, based on the cross-modal semantic retrieval task, multiple neural network layers of the first speech model are screened for semantic retrieval capabilities according to the source speech and the source speech transcription text to obtain the optimal neural network layer for semantic representation alignment, including:

[0058] According to the source speech, extract data representations at multiple neural network levels through the first speech large model to obtain a speech representation matrix;

[0059] According to the source speech transcription text, multiple neural network layer data representation extraction is performed through the first speech large model to obtain a text representation matrix;

[0060] Constructing a cross-modal semantic retrieval task based on the Wasserstein metric of optimal transfer theory;

[0061] Based on the cross-modal semantic retrieval task, similarity is calculated based on the speech representation matrix and the text representation matrix to obtain a similarity matrix;

[0062] Based on the retrieval indicators and the similarity matrix, the semantic retrieval capabilities of multiple neural network layers of the first speech model are screened to obtain the optimal neural network layer for semantic representation alignment; the retrieval indicators include the top K recall rate and the average reciprocal ranking.

[0063] In a feasible implementation manner, the present invention utilizes the pairing relationship between the source speech and the source speech transcription text, randomly selects a number of data, and considers that the paired data are positive examples, and the rest are negative examples.

[0064] The source speech is input into the first speech model, the speech representation of each layer after the internal speech adapter is extracted, the source speech transcribed text is input into the first speech model, the text representation of each layer is extracted, the representation is length compressed, and then the similarity between all the speech representations and text representations of each layer is calculated to obtain a similarity matrix.

[0065] Considering that common indicators such as cosine similarity only reflect the correlation of global semantic information between speech and text, but cannot reflect the correlation of fine-grained semantic information, the present invention adopts a loss based on optimal transmission theory and uses Wasserstein distance to measure the fine-grained semantic similarity between speech and text.

[0066] Wasserstein distance, also known as bulldozer distance, is a measure of the distance between two discrete probability distributions. It evaluates the minimum cost required to redistribute from one probability distribution to another. It is often used in style transfer in the image field and training in generative adversarial networks. The goal of optimal transmission is to find a mapping from X to Y that minimizes the "transportation cost" from μ to ν. The mathematical expression is shown in the following equations (1), (2), and (3):

[0067] (1);

[0068] (2);

[0069] (3);

[0070] Here, two probability distributions μ and ν are given in space X and Y respectively, which represent two different material distributions or resource distributions. The length of distribution μ is n and the length of ν is m, corresponding to the distribution space X is n d-dimensional vectors, and space Y is m d-dimensional vectors. Transmission Planning ,represent arrive The transmission quality, cost matrix ,represent The corresponding space arrive The corresponding space The final problem can be described as finding an optimal transmission plan Z that minimizes the overall transmission cost W.

[0071] Define X and Y as the speech representation space and text representation space, and assume that distribution u and distribution v are uniformly distributed, then the Wasserstein distance between the speech representation and the text representation can be calculated. This distance measures the impact of all elements in the speech / text representation on the distance and can reflect fine-grained semantic alignment information. The smaller the distance, the more aligned the semantic information of the two representations is, and the greater the distance, the more distant the semantic information of the two representations is.

[0072] The present invention adopts a series of retrieval indicators such as the top K recall rate, average reciprocal ranking, etc. to measure the semantic retrieval capabilities of each layer, and obtains the final semantic representation alignment optimal layer through multi-indicator weighing.

[0073] S4. Based on the optimal neural network layer, the source speech and text prompt words are input into the first large speech model for text prediction to obtain speech representation, speech attention weight matrix and predicted text.

[0074] Optionally, based on the optimal neural network level, the source speech and the text prompt word are input into the first speech large model for text prediction to obtain speech representation, speech attention weight matrix and predicted text, including:

[0075] The source voice and the text prompt word are concatenated to obtain first input data;

[0076] Inputting the first input data into a speech encoder for representation extraction to obtain a concatenated speech representation;

[0077] Inputting the concatenated speech representation into the internal speech adapter for dimension conversion to obtain the speech representation;

[0078] Inputting the speech representation into the feature adapter for adaptation to obtain an adapted speech representation;

[0079] Based on the optimal neural network layer, the adapted speech representation is input into the text generation model for text prediction to obtain the predicted text and the speech attention weight matrix.

[0080] In a feasible implementation manner, in the present invention, the speech representation includes the representation after passing through the optimal layer and its internal speech adapter and the attention representation in the middle of the optimal layer.

[0081] The attention representation in the middle of the optimal layer is obtained by calculation from the optimal layer of the first speech model. The number of layers in the optimal layer is , the input representation of the optimal layer is , the overall attention representation in the middle of the optimal layer The calculation formulas are as follows (4), (5), (6):

[0082] (4);

[0083] (5);

[0084] (6)

[0085] Among them, RMSNorm represents the root mean square normalization module, are two different linear mapping matrices, Represents a linear mapping matrix The hidden dimension of Represents the first Position to The degree of attention paid to each location; Represents the vector of the i-th position after the input representation is transformed by linear mapping; similarly, Represents the vector at the jth position after the input representation is transformed by another linear mapping.

[0086] For the speech attention weight matrix in the middle of the optimal layer, by recording the starting position of the speech representation and end position And the starting position of the target text after the first speech model segmentation and end position , we can confirm the corresponding speech attention weight matrix, that is, the overall attention representation matrix in the middle of the optimal layer The first dimension of to Position, pair matrix The second dimension of to Position, thus obtaining the speech attention weight matrix in the middle of the optimal layer .

[0087] The construction of text prompt words is related to the candidate speech task and the target text. The downstream speech task can be a speech translation task or a speech question-answering task. Taking the speech translation task as an example, the text prompt word can be "Please help me translate this speech into English."

[0088] The natural language type of the predicted text may be the same as or different from the first natural language type of the text prompt word.

[0089] If the source voice input is "Hello" in Mandarin, the first natural language type is the Chinese text prompt "Please help me translate this voice into English", and the second natural language type is the English target text "Hello", after training, the first voice model can generate the first predicted text "Hello" in English. The first language model can complete the cross-language and cross-modal generation task. If the source voice input is "Hello" in Mandarin, the first natural language type is the Chinese text prompt "Please give the Chinese synonym expression of this voice", and the second natural language type is the Chinese target text "Hello", after training, the first voice model can generate the Chinese "Hello". The first language model can complete the cross-modal generation task.

[0090] S5. Based on the optimal neural network layer, the source speech transcription text and the text prompt words are input into the first speech model for text representation extraction to obtain the optimal text representation and text attention weight matrix.

[0091] Optionally, based on the optimal neural network layer, the source speech transcription text and the text prompt words are input into the first speech model for text representation extraction to obtain the optimal text representation and the text attention weight matrix, including:

[0092] The source speech transcription text and the text prompt word are spliced ​​to obtain second input data;

[0093] Based on the optimal neural network layer and according to the second input data, text representation extraction is performed through the text generation model to obtain the optimal text representation and the text attention weight matrix.

[0094] In a feasible implementation, the source speech transcription text and the text prompt words are concatenated and input into the first speech model to obtain the text representation of the optimal layer, including the representation after the optimal layer and the attention representation in the middle of the optimal layer.

[0095] The calculation of the overall attention representation matrix of the text in the optimal layer is similar to the calculation of the overall attention representation matrix of the speech in the optimal layer. For the text attention weight matrix in the optimal layer, the starting position of the representation of the source speech transcription text input is recorded. and end position And the starting position of the target text after the first speech model segmentation and end position , we can confirm the corresponding text attention weight matrix, that is, the overall attention representation matrix in the middle of the optimal layer The first dimension of to Position, pair matrix The second dimension of to Position, thus obtaining the text attention weight matrix in the middle of the optimal layer .

[0096] S6. Calculate the loss function based on the target text, speech representation, speech attention weight matrix, predicted text, optimal text representation and text attention weight matrix to obtain the model prediction loss.

[0097] Optionally, a loss function is calculated according to the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain the model prediction loss, including:

[0098] Based on the Wasserstein metric of optimal transfer theory, the optimal transfer loss function is constructed according to the speech attention weight matrix and the text attention weight matrix;

[0099] Based on the optimal transmission loss function, the loss function is calculated according to the speech representation and the optimal text representation to obtain a first loss;

[0100] Based on the cross entropy loss function, the loss function is calculated according to the predicted text and the target text to obtain the second loss.

[0101] In a feasible implementation, the present invention takes into account the length difference between speech representation and text representation and cannot directly use the L2 norm as the model prediction loss. Therefore, Wasserstein Distance based on optimal transmission theory is used as the optimization target.

[0102] Different from the use of this loss in existing work, the existing loss based on Wasserstein Distance has an assumption, that is, it assumes that the distribution of speech representation and text representation in length is uniform, or it is an empirically assumed distribution. Such a distribution will have certain limitations, that is, it cannot well reflect the high-level distribution mapping relationship between speech and text. Any assumed distribution cannot be adapted to all situations.

[0103] Therefore, the present invention proposes a loss calculation method guided by model characteristics. The speech attention weight matrix and the text attention weight matrix After transposition and then softmax transformation, we can get the distribution u and distribution v of the optimal layer, and the speech representation and text representation extracted from this layer can be used to obtain space X and space Y. Based on the above method, we can use the characteristics of the model's different attention to different samples to dynamically guide the calculation of the optimal transmission loss, making the loss calculation more accurate, so as to improve the optimization accuracy and further improve the model performance; the first loss is the optimal transmission loss guided by the model characteristics, and the second loss includes at least one of the cross entropy loss and mean square error loss.

[0104] S7. Optimize the parameters of the first large speech model according to the model prediction loss to obtain the second large speech model.

[0105] In a feasible implementation, the training method proposed in the present invention can be applied to the training of large speech models under various architectures, enhance the speech comprehension ability of large speech models, and fully explore the rich knowledge and generalization ability of large text models in text scenarios. Explain the internal working mechanism of the model at the level of multimodal information understanding. In addition, the trained large speech model can be applied to a variety of downstream speech tasks. For example: when communicating across languages, users can input the speech to be translated into the large speech model, give text instructions to require the large speech model to translate in the direction of a specific target language, and the large speech model can generate a translated text in a given language.

[0106] The present invention proposes an adaptive hierarchical representation alignment training method for a large speech model. By introducing an internal speech adapter and an adaptive hierarchical optimization selection strategy, the semantic understanding and representation alignment capabilities of the large speech model in cross-modal speech-text tasks are significantly improved, and the performance of the model is enhanced. The adaptive hierarchical optimization selection strategy allows the model to automatically select the optimal layer for training according to the actual semantic understanding ability, avoiding resource waste, accelerating the convergence speed, and improving the training efficiency. By constructing the first representation loss, the difference between the speech representation and the text representation is directly minimized, ensuring a high degree of alignment between the two at the semantic level, and improving the model's understanding and robustness for complex semantic structures. By utilizing the natural mapping relationship between the source speech and the transcribed text, by splicing the speech and text prompt words into the model, the deep fusion of the two modal representations is promoted, the semantic parsing ability is enhanced, and the modal gap between speech and text is effectively narrowed. The present invention is an efficient and accurate adaptive hierarchical representation alignment training method that fully utilizes the mapping relationship between the source speech and the transcribed text.

[0107] Figure 3 The block diagram of a device for adaptive hierarchical representation alignment training of a large speech model according to an exemplary embodiment is shown. The device is used for an adaptive hierarchical representation alignment training method of a large speech model. Figure 3 The device includes a data acquisition module 310, a model training module 320, a cross-modal semantic retrieval module 330, a text prediction module 340, a text representation extraction module 350, a loss calculation module 360 ​​and a model optimization module 370. Among them:

[0108] The data acquisition module 310 is used to acquire the source speech and the target text of the source speech; the source speech is input into the candidate speech large model for text transcription to obtain the source speech transcription text;

[0109] A model training module 320 is used to perform model training based on the internal speech adapter and the candidate speech large model using the source speech and the text prompt words to obtain a first speech large model;

[0110] A cross-modal semantic retrieval module 330 is used to perform semantic retrieval capability screening on multiple neural network layers of the first speech model based on the cross-modal semantic retrieval task and the source speech and the source speech transcription text, so as to obtain the optimal neural network layer for semantic representation alignment;

[0111] A text prediction module 340 is used to input the source speech and the text prompt words into the first speech model for text prediction based on the optimal neural network layer, and obtain speech representation, speech attention weight matrix and predicted text;

[0112] A text representation extraction module 350 is used to input the source speech transcription text and the text prompt words into the first speech model to extract the text representation based on the optimal neural network layer, and obtain the optimal text representation and the text attention weight matrix;

[0113] A loss calculation module 360 ​​calculates a loss function based on the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain a model prediction loss;

[0114] The model optimization module 370 is used to optimize the parameters of the first large speech model according to the model prediction loss to obtain the second large speech model.

[0115] Optionally, the model training module 320 is further used to:

[0116] Inserting the internal speech adapter into the candidate speech big model to obtain an improved candidate language big model;

[0117] Using the source speech and the text prompt words, the improved candidate speech model is trained to obtain the first speech model;

[0118] The language type of the text prompt is the same as the source speech transcription text.

[0119] The structure of the internal speech adapter includes two linear mapping layers, a nonlinear activation function, a residual connection and a mean square error regularization module.

[0120] Optionally, the cross-modal semantic retrieval module 330 is further used to:

[0121] According to the source speech, extracting data representations at multiple neural network levels through the first speech large model to obtain a speech representation matrix;

[0122] According to the source speech transcription text, multiple neural network layer data representation extraction is performed through the first speech large model to obtain a text representation matrix;

[0123] Constructing a cross-modal semantic retrieval task based on the Wasserstein metric of optimal transfer theory;

[0124] Based on the cross-modal semantic retrieval task, similarity is calculated based on the speech representation matrix and the text representation matrix to obtain a similarity matrix;

[0125] Based on the retrieval indicators and the similarity matrix, the semantic retrieval capabilities of multiple neural network layers of the first speech model are screened to obtain the optimal neural network layer for semantic representation alignment; the retrieval indicators include the top K recall rate and the average reciprocal ranking.

[0126] Optionally, the text prediction module 340 is further configured to:

[0127] The source voice and the text prompt word are concatenated to obtain first input data;

[0128] Inputting the first input data into a speech encoder for representation extraction to obtain a concatenated speech representation;

[0129] Inputting the concatenated speech representation into the internal speech adapter for dimension conversion to obtain the speech representation;

[0130] Inputting the speech representation into the feature adapter for adaptation to obtain an adapted speech representation;

[0131] Based on the optimal neural network layer, the adapted speech representation is input into the text generation model for text prediction to obtain the predicted text and the speech attention weight matrix.

[0132] Optionally, the text representation extraction module 350 is further configured to:

[0133] The source speech transcription text and the text prompt word are spliced ​​to obtain second input data;

[0134] Based on the optimal neural network layer and according to the second input data, text representation extraction is performed through the text generation model to obtain the optimal text representation and the text attention weight matrix.

[0135] Optionally, the loss calculation module 360 ​​is further used to:

[0136] Based on the Wasserstein metric of optimal transfer theory, the optimal transfer loss function is constructed according to the speech attention weight matrix and the text attention weight matrix;

[0137] Based on the optimal transmission loss function, the loss function is calculated according to the speech representation and the optimal text representation to obtain a first loss;

[0138] Based on the cross entropy loss function, the loss function is calculated according to the predicted text and the target text to obtain the second loss.

[0139] The present invention proposes an adaptive hierarchical representation alignment training method for a large speech model. By introducing an internal speech adapter and an adaptive hierarchical optimization selection strategy, the semantic understanding and representation alignment capabilities of the large speech model in cross-modal speech-text tasks are significantly improved, and the performance of the model is enhanced. The adaptive hierarchical optimization selection strategy allows the model to automatically select the optimal layer for training according to the actual semantic understanding ability, avoiding resource waste, accelerating the convergence speed, and improving the training efficiency. By constructing the first representation loss, the difference between the speech representation and the text representation is directly minimized, ensuring a high degree of alignment between the two at the semantic level, and improving the model's understanding and robustness for complex semantic structures. By utilizing the natural mapping relationship between the source speech and the transcribed text, by splicing the speech and text prompt words into the model, the deep fusion of the two modal representations is promoted, the semantic parsing ability is enhanced, and the modal gap between speech and text is effectively narrowed. The present invention is an efficient and accurate adaptive hierarchical representation alignment training method that fully utilizes the mapping relationship between the source speech and the transcribed text.

[0140] Figure 4 is a schematic diagram of the structure of an adaptive hierarchical representation alignment training device provided by an embodiment of the present invention, such as Figure 4 As shown, the adaptive hierarchical representation alignment training device may include the above Figure 3 The adaptive hierarchical representation alignment training device for the large speech model shown. Optionally, the adaptive hierarchical representation alignment training device 410 may include a first processor 2001.

[0141] Optionally, the adaptive hierarchical representation alignment training device 410 may further include a memory 2002 and a transceiver 2003 .

[0142] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0143] Combine the following Figure 4 The components of the adaptive hierarchical representation alignment training device 410 are specifically introduced as follows:

[0144] The first processor 2001 is the control center of the adaptive hierarchical representation alignment training device 410, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0145] Optionally, the first processor 2001 may perform various functions of the adaptive hierarchical representation alignment training device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0146] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0147] In a specific implementation, as an embodiment, the adaptive hierarchical representation alignment training device 410 may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0148] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0149] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0150] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0151] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0152] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and align the interface circuit of the training device 410 with the adaptive hierarchical representation ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0153] It should be noted that Figure 4 The structure of the adaptive hierarchical representation alignment training device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0154] In addition, the technical effects of the adaptive hierarchical representation alignment training device 410 can refer to the technical effects of the adaptive hierarchical representation alignment training method for the large speech model described in the above method embodiment, and will not be repeated here.

[0155] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0156] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0157] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0158] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0159] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0160] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0167] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An adaptive hierarchical representation alignment training method for a large speech model, characterized in that: The method comprises: Acquire source speech and source speech target text; input the source speech into the candidate speech large model for text transcription to obtain the source speech transcription text; Based on the internal speech adapter, according to the candidate speech large model, the source speech and the text prompt word are used to perform model training to obtain a first speech large model; Based on the cross-modal semantic retrieval task, the semantic retrieval capability of multiple neural network layers of the first large speech model is screened according to the source speech and the source speech transcription text to obtain the optimal neural network layer for semantic representation alignment; Based on the optimal neural network level, the source speech and the text prompt word are input into the first speech large model for text prediction to obtain speech representation, speech attention weight matrix and predicted text; Based on the optimal neural network level, the source speech transcription text and the text prompt words are input into the first speech macro model to extract text representation, and obtain the optimal text representation and text attention weight matrix; A loss function is calculated based on the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain a model prediction loss; According to the model prediction loss, the first large speech model is optimized in parameters to obtain a second large speech model.

2. The adaptive hierarchical representation alignment training method for a large speech model according to claim 1, characterized in that: The method of performing model training based on the candidate speech large model using the source speech and the text prompt words based on the internal speech adapter to obtain the first speech large model includes: Inserting the internal speech adapter into the candidate speech large model to obtain an improved candidate language large model; Using the source speech and the text prompt word, the improved candidate speech large model is trained to obtain a first speech large model; The language type of the text prompt word is the same as that of the source speech transcription text.

3. The adaptive hierarchical representation alignment training method for a large speech model according to claim 2, characterized in that: The structure of the internal speech adapter includes two linear mapping layers, a nonlinear activation function, a residual connection and a mean square error regularization module.

4. The adaptive hierarchical representation alignment training method for a large speech model according to claim 1, characterized in that: The cross-modal semantic retrieval task is based on the source speech and the source speech transcription text, and the semantic retrieval capability of multiple neural network layers of the first speech large model is screened to obtain the optimal neural network layer for semantic representation alignment, including: According to the source speech, extract data representations at multiple neural network levels through the first speech large model to obtain a speech representation matrix; According to the source speech transcription text, extract data representations at multiple neural network levels through the first speech large model to obtain a text representation matrix; Constructing a cross-modal semantic retrieval task based on the Wasserstein metric of optimal transfer theory; Based on the cross-modal semantic retrieval task, similarity calculation is performed according to the speech representation matrix and the text representation matrix to obtain a similarity matrix; Based on the retrieval indicators and according to the similarity matrix, the semantic retrieval capabilities of multiple neural network layers of the first large speech model are screened to obtain the optimal neural network layer for semantic representation alignment; the retrieval indicators include the top K recall rate and the average reciprocal ranking.

5. The adaptive hierarchical representation alignment training method for a large speech model according to claim 1, characterized in that: The method of inputting the source speech and the text prompt word into the first speech large model for text prediction based on the optimal neural network level to obtain speech representation, speech attention weight matrix and predicted text includes: splicing the source voice and the text prompt word to obtain first input data; Inputting the first input data into a speech encoder for representation extraction to obtain a concatenated speech representation; Inputting the concatenated speech representation into an internal speech adapter for dimensional conversion to obtain a speech representation; Inputting the speech representation into a feature adapter for adaptation to obtain an adapted speech representation; Based on the optimal neural network level, the adapted speech representation is input into a text generation model for text prediction to obtain predicted text and a speech attention weight matrix.

6. The adaptive hierarchical representation alignment training method for a large speech model according to claim 1, characterized in that: The method of inputting the source speech transcription text and the text prompt words into the first speech model to extract text representation based on the optimal neural network level to obtain the optimal text representation and the text attention weight matrix includes: splicing the source speech transcription text and the text prompt word to obtain second input data; Based on the optimal neural network level and according to the second input data, text representation extraction is performed through a text generation model to obtain an optimal text representation and a text attention weight matrix.

7. The adaptive hierarchical representation alignment training method for a large speech model according to claim 1, characterized in that: The step of calculating the loss function according to the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain the model prediction loss includes: Based on the Wasserstein metric of optimal transmission theory, an optimal transmission loss function is constructed according to the speech attention weight matrix and the text attention weight matrix; Based on the optimal transmission loss function, a loss function is calculated according to the speech representation and the optimal text representation to obtain a first loss; Based on the cross entropy loss function, a loss function is calculated according to the predicted text and the target text to obtain a second loss.

8. An adaptive hierarchical representation alignment training device for a large speech model, the adaptive hierarchical representation alignment training device for a large speech model is used to implement the adaptive hierarchical representation alignment training method for a large speech model as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A data acquisition module is used to acquire source speech and source speech target text; the source speech is input into the candidate speech large model for text transcription to obtain the source speech transcription text; A model training module, configured to perform model training based on an internal speech adapter and the candidate speech large model using the source speech and the text prompt words to obtain a first speech large model; A cross-modal semantic retrieval module, for performing semantic retrieval capability screening on multiple neural network layers of the first large speech model based on the cross-modal semantic retrieval task and according to the source speech and the source speech transcription text, to obtain the optimal neural network layer for semantic representation alignment; A text prediction module, configured to input the source speech and the text prompt word into the first speech large model for text prediction based on the optimal neural network layer, and obtain speech representation, speech attention weight matrix and predicted text; A text representation extraction module, for inputting the source speech transcription text and the text prompt words into the first speech macro model to extract text representation based on the optimal neural network layer, and obtaining an optimal text representation and a text attention weight matrix; A loss calculation module calculates a loss function according to the target text, the speech representation, the speech attention weight matrix, the predicted text, the optimal text representation and the text attention weight matrix to obtain a model prediction loss; The model optimization module is used to optimize the parameters of the first large speech model according to the model prediction loss to obtain the second large speech model.

9. An adaptive hierarchical representation alignment training device, characterized in that: The adaptive hierarchical representation alignment training device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images