Dialect recognition model training method and device and dialect recognition method

Abstract

The invention discloses a training method and device of a dialect recognition model and a dialect recognition method. The model training method comprises the following steps: training a first initial model comprising a feature extraction module, a voice encoder and a natural language large model by using a first dialect sample set with a text label to obtain a first dialect recognition model; training a second initial model containing a feature extraction module, a voice encoder and a voice decoder in the first dialect recognition model by using a text result of a first dialect sample set predicted by the first dialect recognition model to obtain a second dialect recognition model; and finally, training a second dialect recognition model by using the second dialect sample set without text annotation and a text result of the second dialect sample set predicted by the first dialect recognition model to obtain a target dialect recognition model. The technical problems that a large number of samples are needed in traditional dialect recognition model training, the data quality is difficult to guarantee, and the labeling cost is high are solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and more specifically, to a method and device for training a dialect recognition model, and a dialect recognition method. Background Art

[0002] In the current field of speech recognition technology, dialect recognition remains a complex and challenging task. As a country with an exceptionally rich variety of dialects, China's widespread distribution and significant differences place extremely high demands on the generalization capabilities of speech recognition systems. Traditional automatic speech recognition systems, particularly those based on Gaussian mixture models and hidden Markov models, encounter bottlenecks when handling dialect recognition. These methods often rely on manually designed feature engineering, such as extracting spectral and prosodic features from speech. This not only consumes a significant amount of R&D resources, but also limits their adaptability to diverse dialects due to their reliance on expert knowledge.

[0003] With the development of deep learning technology, models such as convolutional neural networks and recurrent neural networks have achieved remarkable success in speech recognition. They can automatically learn and extract speech features, improving the accuracy of dialect recognition. However, training deep learning models requires large amounts of labeled data, which poses a significant challenge for dialect recognition. The diverse populations of speakers of dialects, with varying ages, genders, and educational backgrounds, makes data collection particularly challenging. This is especially true for dialects with smaller populations, where building comprehensive, high-quality training datasets is challenging. This data scarcity directly impacts model performance, making it difficult for models to fully learn from the diversity of dialects and limiting recognition effectiveness. Furthermore, inconsistency in data annotation is a key factor affecting the performance of dialect recognition systems. Since there are no standardized grammar, vocabulary, and pronunciation rules for dialects, the annotation process lacks unified guidance, making errors and inconsistencies prone to occur. This undoubtedly increases the difficulty for models that rely on data-driven learning.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for training a dialect recognition model, and a dialect recognition method, so as to at least solve the technical problems that traditional dialect recognition model training requires a large number of samples, data quality is difficult to ensure, and the labeling cost is high.

[0006] According to one aspect of an embodiment of the present application, a method for training a dialect recognition model is provided, comprising: obtaining a first dialect sample set with text annotations and a second dialect sample set without text annotations; training a first initial model using the first dialect sample set to obtain a first dialect recognition model, wherein the first initial model includes at least: a feature extraction module, a speech encoder, and a large natural language model; training a second initial model using the first dialect sample set and the first dialect recognition model based on the predicted text results of the first dialect sample set to obtain a second dialect recognition model, wherein the second initial model includes: the feature extraction module, speech encoder, and speech decoder in the first dialect recognition model; training the second dialect recognition model using the second dialect sample set and the first dialect recognition model based on the predicted text results of the second dialect sample set to obtain a target dialect recognition model.

[0007] Optionally, the method also includes: obtaining multiple sets of voice data obtained by multiple objects in multiple dimensions reading texts on multiple topics in the target dialect, wherein the dimensions include at least one of the following: age, gender, occupation, and place of origin; dividing the multiple sets of voice data into a first voice data set and a second voice data set, wherein the amount of voice data in the first voice data set is less than the amount of voice data in the second voice data set; performing text annotation on the voice data in the first voice data set, and using the annotated first voice data set as the first dialect sample set; and directly using the second voice data set as the second dialect sample set.

[0008] Optionally, the method also includes: before dividing the multiple groups of voice data into a first voice data set and a second voice data set, performing a first preprocessing operation on the multiple groups of voice data respectively, wherein the first preprocessing operation includes at least one of the following: adjusting the audio sampling rate, deleting voice data with a duration less than a first preset threshold, and segmenting voice data with a duration greater than a second preset threshold; after text annotation of the voice data in the first voice data set, performing a second preprocessing operation on the annotated first voice data set, wherein the second preprocessing operation includes at least one of the following: checking the consistency of the annotation text, and aligning the audio with the annotated text.

[0009] Optionally, the first initial model includes: a feature extraction module, a speech encoder, a speech adapter and a natural language large model connected in sequence; the speech encoder includes: a starting module, a position encoding module and multiple encoding blocks connected in sequence; the starting module includes: two convolutional layers and a Gaussian error linear unit activation function; the encoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; the speech adapter includes: a downsampling module, a perception layer, a rectified linear unit activation function and a perception layer connected in sequence; wherein, the feature extraction module is used to extract the Mel spectrum features corresponding to the input speech data; the speech encoder is used to encode the Mel spectrum features into a first feature representation; the speech adapter is used to map the first feature representation to the embedding space corresponding to the natural language large model to obtain a second feature representation; the natural language large model is used to analyze the second feature representation based on a preset prompt word to obtain a predicted text result corresponding to the input speech data.

[0010] Optionally, the first dialect sample set includes a training set and a test set, and the step of training the first initial model using the first dialect sample set includes: iteratively training the first initial model using the training set, and in each training batch, inputting multiple groups of sample data corresponding to the training batch into the first initial model in sequence to obtain multiple groups of first predicted text results output by the first initial model; constructing a first loss function based on the multiple groups of first predicted text results and the text annotation information corresponding to the multiple groups of sample data, wherein the type of the first loss function is a cross-entropy loss function; and adjusting the model parameters of the feature extraction module, speech encoder, speech adapter and natural language large model based on the first loss function.

[0011] Optionally, the second initial model includes: a feature extraction module and a speech encoder and a speech decoder in the first dialect recognition model connected in sequence; the speech decoder includes: a position encoding module and multiple decoding blocks connected in sequence; the decoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; wherein the speech decoder is used to decode the first feature representation into a predicted text result corresponding to the input speech data.

[0012] Optionally, the step of training the second initial model using the first dialect sample set and the first dialect recognition model for the predicted text results of the first dialect sample set includes: using the first dialect recognition model to analyze multiple groups of sample data in the training set respectively to obtain multiple groups of second predicted text results; using the training set to iteratively train the second initial model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second initial model in sequence to obtain multiple groups of third predicted text results output by the second initial model; constructing a second loss function based on the multiple groups of third predicted text results and the text annotation information corresponding to the multiple groups of sample data corresponding to the training batch, and constructing a third loss function based on the multiple groups of third predicted text results and the multiple groups of second predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the second loss function is a cross-entropy loss function, and the type of the third loss function is a relative entropy loss function; adjusting the model parameters of the speech decoder based on the second loss function and the third loss function.

[0013] Optionally, the step of training the second dialect recognition model using the second dialect sample set and the predicted text results of the second dialect sample set by the first dialect recognition model includes: using the first dialect recognition model to analyze multiple groups of sample data in the second dialect sample set respectively to obtain multiple groups of fourth predicted text results; using the second dialect sample set to iteratively train the second dialect recognition model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second dialect recognition model in sequence to obtain multiple groups of fifth predicted text results output by the second dialect recognition model; constructing a fourth loss function based on the multiple groups of fifth predicted text results and the multiple groups of fourth predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the fourth loss function is a relative entropy loss function; and adjusting the model parameters of the speech decoder based on the fourth loss function.

[0014] Optionally, the method further includes: testing the target dialect recognition model using a test set to determine the model performance of the target dialect recognition model.

[0015] According to another aspect of an embodiment of the present application, a dialect recognition method is also provided, including: obtaining dialect speech data to be recognized; analyzing the dialect speech data using a target dialect recognition model to obtain target text data corresponding to the dialect speech data, wherein the target dialect recognition model is trained using the training method of the above-mentioned dialect recognition model.

[0016] According to another aspect of an embodiment of the present application, a training device for a dialect recognition model is also provided, including: an acquisition module for acquiring a first dialect sample set with text annotations and a second dialect sample set without text annotations; a first training module for training a first initial model using the first dialect sample set to obtain a first dialect recognition model, wherein the first initial model at least includes: a feature extraction module, a speech encoder and a large natural language model; a second training module for training a second initial model using the first dialect sample set and the first dialect recognition model to predict the text results of the first dialect sample set to obtain a second dialect recognition model, wherein the second initial model includes: the feature extraction module and the speech encoder and the speech decoder in the first dialect recognition model; a third training module for training the second dialect recognition model using the second dialect sample set and the first dialect recognition model to predict the text results of the second dialect sample set to obtain a target dialect recognition model.

[0017] According to another aspect of an embodiment of the present application, a computer program product is further provided, comprising: a computer program, wherein when the computer program is executed by a processor, the training method of the dialect recognition model or the dialect recognition method described above is implemented.

[0018] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned dialect recognition model training method or dialect recognition method through the computer program.

[0019] In an embodiment of the present application, in the first stage of training, the first initial model is trained using the annotated first dialect sample set to obtain a first dialect recognition model, which not only enhances the model's learning ability for annotated dialect data, but also significantly improves the model's recognition accuracy by integrating the semantic understanding ability of the large natural language model; in the second stage of training, the first dialect recognition model is used to predict the first dialect sample set to obtain the predicted text results of the first dialect sample set, and the predicted text results are used to train the second initial model to obtain a second dialect recognition model, thereby realizing knowledge transfer between high-performance models and low-resource models. This migration not only maintains strong recognition capabilities but also significantly reduces the number of model parameters, thereby accelerating the model's inference speed. In the third stage of training, the first dialect recognition model is used to predict the second dialect sample set without text annotations. The generated predicted text serves as a weak supervisory signal to train the second dialect recognition model, resulting in the target dialect recognition model. This allows the model to learn on unlabeled data, effectively expanding the scope of training data and further improving the model's generalization performance. Through three stages of training, a high-performance dialect recognition model can be trained on less labeled data, and unlabeled data can be effectively used for subsequent training, significantly reducing the cost of collecting dialect data. This solves the technical problems of traditional dialect recognition model training, which require a large number of samples, have difficulty ensuring data quality, and have high annotation costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 1 is a flow chart of an optional method for training a dialect recognition model according to an embodiment of the present application;

[0022] Figure 2 is a schematic structural diagram of an optional first initial model according to an embodiment of the present application;

[0023] Figure 3 is a schematic structural diagram of an optional speech encoder according to an embodiment of the present application;

[0024] Figure 4 is a schematic structural diagram of an optional first initial model according to an embodiment of the present application;

[0025] Figure 5 is a schematic structural diagram of an optional speech decoder according to an embodiment of the present application;

[0026] Figure 6is a flowchart of an optional dialect recognition method according to an embodiment of the present application;

[0027] Figure 7 1 is a schematic diagram of the structure of a training device for an optional dialect recognition model according to an embodiment of the present application;

[0028] Figure 8 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0031] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0032] ASR (Automatic Speech Recognition_Large): Automatic speech recognition is a technology that converts spoken language into text. This technology is widely used in scenarios such as telephone speech recognition, voice assistants, real-time subtitle generation, and speech-to-text transcription and translation. ASR systems typically include steps such as speech signal processing, feature extraction, pattern matching, and speech recognition. In the application, the ASR system is being expanded to integrate with a large language model (LLM) to improve its performance in dialect recognition tasks.

[0033] To adapt to dialect recognition, the ASR system here is specially designed as "ASR_LLM," which means it incorporates the semantic understanding capabilities of a large language model. In the first stage, ASR_LLM leverages the speech encoder, speech adapter, and LLM itself to achieve deep learning and understanding of dialect characteristics, thereby improving recognition accuracy. Unlike traditional ASR systems that rely solely on speech features and acoustic models, ASR_LLM incorporates semantic information to better capture the meaning and context of dialects, improving recognition results.

[0034] Large Language Models (LLMs) are deep learning models with a large number of parameters and computational power, specifically designed for understanding, generating, and processing natural language text. Large Language Models (LLMs) are capable of learning the complex structure of language, including vocabulary, grammar, semantics, and even cultural context. Consequently, they are widely used in areas such as machine translation, text generation, and question-answering systems.

[0035] Relative entropy loss function: also known as Kullback-Leibler (KL) divergence, is a statistic that measures the difference between two probability distributions. In machine learning and deep learning, especially natural language processing and speech recognition, relative entropy loss function is often used as a supplement to supervised learning loss functions or as a learning objective in semi-supervised / unsupervised learning scenarios. Relative entropy is always greater than or equal to 0. It is 0 if and only if the two distributions are identical. In machine learning, relative entropy can be used to evaluate the difference between a model's predicted distribution and its true distribution, thereby optimizing model parameters.

[0036] Example 1

[0037] According to an embodiment of the present application, a method for training a dialect recognition model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1 is a flow chart of a method for training a dialect recognition model according to an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps:

[0039] Step S102, obtaining a first dialect sample set with text annotations and a second dialect sample set without text annotations;

[0040] Step S104: training a first initial model using the first dialect sample set to obtain a first dialect recognition model, wherein the first initial model includes at least: a feature extraction module, a speech encoder, and a natural language large model;

[0041] Step S106: training a second initial model using the first dialect sample set and the first dialect recognition model for the predicted text results of the first dialect sample set to obtain a second dialect recognition model, wherein the second initial model includes: a feature extraction module, a speech encoder, and a speech decoder in the first dialect recognition model;

[0042] Step S108 : training the second dialect recognition model using the second dialect sample set and the predicted text results of the first dialect recognition model on the second dialect sample set to obtain a target dialect recognition model.

[0043] The following describes the various steps of the dialect recognition model training method in conjunction with a specific implementation process.

[0044] As an optional implementation, the first dialect sample set and the second dialect sample set can be obtained in the following manner: obtaining multiple groups of voice data obtained by multiple objects in multiple dimensions reading texts on multiple topics in the target dialect, wherein the dimensions include at least one of the following: age, gender, occupation, and place of origin; dividing the multiple groups of voice data into a first voice data set and a second voice data set, wherein the number of voice data in the first voice data set is less than the number of voice data in the second voice data set; performing text annotation on the voice data in the first voice data set, and using the annotated first voice data set as the first dialect sample set; and directly using the second voice data set as the second dialect sample set.

[0045] Among them, the multiple groups of collected voice data involved can be the dialect voice data of people in different districts, counties and towns in the region, and the occupations include teachers, workers, farmers, students, medical staff and civil servants. The age of the readers ranges from 5 to 60 years old, with about half male and half female. In order to ensure the wide coverage and diversity of the collected voice data, the collected content can include daily conversations, storytelling, news broadcasts and other themes. Among them, daily conversations are carefully set up for parent-child exchanges and couples in family scenes, colleague discussions and business communications in work scenes, and friends gatherings and chats in social scenes. Communication in different scenarios.

[0046] By performing text annotation on a small number of speech datasets and using the annotated speech datasets as the first dialect sample set, we can effectively reduce the dependence of traditional dialect recognition methods on large amounts of high-quality dialect data, thereby lowering the high cost of data preparation.

[0047] In order to further ensure the data quality of the final model training and thereby improve the recognition performance and generalization ability of the model, relevant preprocessing operations are introduced to process the collected voice data, which can be specifically achieved in the following ways: before dividing the multiple groups of voice data into a first voice data set and a second voice data set, the method also includes: performing a first preprocessing operation on the multiple groups of voice data respectively, wherein the first preprocessing operation includes at least one of the following: adjusting the audio sampling rate, deleting voice data with a duration less than a first preset threshold, and segmenting voice data with a duration greater than a second preset threshold; after performing text annotation on the voice data in the first voice data set, the method also includes: performing a second preprocessing operation on the annotated first voice data set, wherein the second preprocessing operation includes at least one of the following: checking the consistency of the annotation text, and aligning the audio with the annotation text.

[0048] Specifically, the first preprocessing operation may include the following operations: adjustment of the speech sampling frequency, deletion of part of the speech data, and segmentation of part of the speech data. Among them, the adjustment of the speech sampling frequency: taking into account that the multiple groups of dialect speech data collected may come from different devices, resulting in inconsistent sampling rates. In order to ensure the consistency of all data and the training effect of the model, the sampling rate needs to be adjusted. During the adjustment process, not only the sampling rate should be unified to a preset fixed value (such as 16kHz), but also the impact of the sampling rate conversion on the audio quality needs to be considered. A high-quality resampling algorithm, such as the resample function in the sox tool, or super-resolution technology based on deep learning, can be used to reduce distortion during the resampling process; deletion of part of the speech data: the duration of some speech data is too short and may not fully reflect the speaker's voice characteristics, resulting in incomplete feature extraction and affecting the model training effect. A reasonable threshold (such as 0.5 seconds) can be set to filter data that is too short. At the same time, you can attempt to analyze and restore the context of these short speech segments to determine whether they have training value and avoid losing valuable information. Partial speech data segmentation: Long speech segments not only increase the computational burden of data processing but may also make it difficult for the model to capture the semantic information of the entire segment during learning, affecting recognition performance. Set a reasonable duration threshold (such as 30 seconds) and segment speech data that exceeds this threshold. Consider semantic integrity during segmentation, and incorporate voice activity detection (VAD) technology to ensure that each segment contains a complete speech unit to avoid semantic fragmentation.

[0049] Specifically, the second preprocessing step can include the following operations: checking the consistency of the annotated text, aligning the audio with the annotated text, speech enhancement and denoising, normalizing the speech standards, and multimodal data fusion. Regarding the consistency check of the annotated text, dialects can differ significantly in pronunciation and grammar. Therefore, during the annotation process, the annotation of words in the same dialect should be consistent. Dedicated annotation consistency checking tools can be developed to automatically compare words and sentences using statistical methods or LLMs to detect and correct inconsistent annotations, such as the use of homophones or multiple spellings of dialect words. Aligning the audio with the annotated text: Ensuring that the audio data and the corresponding annotated text are fully aligned on the timeline is key to training the recognition model. Alignment methods based on dynamic time warping or initial alignment using an ASR model, followed by manual verification, can be used to ensure that the start and end points of each speech segment match the annotated text to avoid recognition errors caused by temporal shifts. Speech enhancement and denoising: Due to the diversity of acquisition environments, raw speech data may contain background noise or poor recording quality. Speech enhancement and denoising techniques, such as spectral subtraction, Wiener filters, or deep learning-based denoising methods, can be introduced to improve the clarity of speech data and enhance the robustness of the model; speech standardization: In order to eliminate individual differences between different speakers, such as volume and speaking speed, speech standardization techniques can be adopted, such as normalizing the volume and adjusting the speaking speed, to ensure that the model treats all data fairly during training; multimodal data fusion: Considering the complexity of dialect recognition, it is possible to explore the fusion of speech data with other modal data (such as the speaker's facial expressions and gestures), and use multimodal information to improve the recognition performance of the model, especially when dealing with ambiguous pronunciation or heavy accents.

[0050] After obtaining the processed sample data, you can use the following methods to train the sample data. The specific training includes the following steps:

[0051] Step S1: a first dialect sample set includes a training set and a test set, and a first initial model is trained using the first dialect sample set to obtain a first dialect recognition model;

[0052] Step S2, training a second initial model using the first dialect sample set and the first dialect recognition model for the predicted text results of the first dialect sample set to obtain a second dialect recognition model;

[0053] Step S3: training the second dialect recognition model using the second dialect sample set and the predicted text results of the first dialect recognition model on the second dialect sample set to obtain a target dialect recognition model.

[0054] As an optional implementation, the first initial model includes: a feature extraction module, a speech encoder, a speech adapter and a natural language large model connected in sequence; the speech encoder includes: a starting module, a position encoding module and multiple encoding blocks connected in sequence; the starting module includes: two convolutional layers and a Gaussian error linear unit activation function; the encoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; the speech adapter includes: a downsampling module, a perception layer, a rectified linear unit activation function and a perception layer connected in sequence; wherein, the feature extraction module is used to extract the Mel spectrum features corresponding to the input speech data; the speech encoder is used to encode the Mel spectrum features into a first feature representation; the speech adapter is used to map the first feature representation to the embedding space corresponding to the natural language large model to obtain a second feature representation; the natural language large model is used to analyze the second feature representation based on a preset prompt word to obtain a predicted text result corresponding to the input speech data.

[0055] in, Figure 2 A schematic diagram of the structure of the first initial model is shown in FIG. Figure 2 As shown, the first initial model is specifically composed of a feature extraction module, a speech encoder, a speech adapter and a natural language large model connected in sequence. Specifically, Figure 3 Schematic diagram of the structure of the speech encoder is shown in FIG. Figure 3 As shown, the encoder includes a starting module, a position encoding module and multiple encoding blocks connected in sequence.

[0056] The feature extraction module utilizes Mel-spectrogram features, an acoustic feature widely used in speech recognition that can effectively capture the spectral characteristics of speech. The multi-head self-attention mechanism in the speech encoder allows the model to focus on different parts of the speech signal, enhancing its understanding of sentence structure. The speech adapter aligns the encoded features with the embedding space of the large natural language model through downsampling and perception layers. This step is crucial for dialect recognition because it ensures that dialect features can be correctly interpreted by the natural language model. The introduction of the large natural language model enables the model to predict text based on context, greatly improving recognition accuracy. This series of technical features, working together through a carefully designed model structure, solves problems such as inaccurate feature extraction and weak context understanding in dialect recognition, achieving high-quality dialect recognition.

[0057] Specifically, in an optional technical solution provided in the above step S1, the first initial model is trained using the training set and the test set in the first dialect sample set. This can be done in the following way: the first initial model is iteratively trained using the training set, and in each training batch, multiple groups of sample data corresponding to the training batch are sequentially input into the first initial model to obtain multiple groups of first predicted text results output by the first initial model; a first loss function is constructed based on the multiple groups of first predicted text results and the text annotation information corresponding to the multiple groups of sample data, wherein the type of the first loss function is a cross-entropy loss function; and the model parameters of the feature extraction module, speech encoder, speech adapter and natural language large model are adjusted based on the first loss function.

[0058] During the training process, the trans-entropy loss function is used to quantify the discrepancy between the model's predictions and the actual annotated text. This technical feature continuously optimizes model parameters to minimize the gap between predicted and true text, thereby improving the model's recognition accuracy. The joint training of the feature extraction module, speech encoder, speech adapter, and large natural language model enables the model to learn the acoustic characteristics, semantic structure, and language habits of dialects at multiple levels, overcoming the recognition limitations caused by single-module training and achieving comprehensive optimization of the model for dialect recognition tasks.

[0059] In the specific implementation, the feature extraction module is first used to convert the input speech into Mel spectrum features calculated by fixed window length and stride. The obtained Mel spectrum features are passed to the speech encoder for further processing. The speech encoder is as follows Figure 3 As shown in the figure. The speech encoder uses a small inception module to process the input representation. This module consists of two convolutional layers, each of which sets a specific filter width and uses a Gaussian error linear unit activation function. Since the Transformer model cannot directly perceive the order of the input sequence, position encoding is introduced in the speech encoder. Position encoding adds position information to each input vector to help the model understand the order relationship in the sequence. The core of the speech encoder is composed of multiple stacked Transformer encoding blocks, each of which contains a multi-head self-attention mechanism, a feedforward neural network, a layer normalization module, and a residual connection module. The multi-head self-attention mechanism can learn information about data in different dimensions under the current data and explore more signal features. The final result is calculated by summarizing the attention values calculated by different heads.

[0060] Among them, the self-attention calculation mechanism is as follows:

[0061] Perform a linear transformation on the input sequence X to obtain three homologous sequences Q, K, and V. The calculation formula is as follows:

[0062] Q=X·W Q

[0063] K=X·W K

[0064] V=X·W V

[0065] Among them, W Q , W K , W V It is the weight matrix corresponding to Q, K, and V.

[0066] Calculate the attention value a corresponding to positions i and j ij , where a ij It can be calculated in the following way:

[0067]

[0068] The attention value is calculated by taking the dot product of Q and K. The result of this dot product represents the similarity between the Q vector and the K vector. Then, this dot product result is divided by To normalize. k By W K The output dimension is determined by, assuming W K The output dimension of is 64, then d k =64. This normalization process ensures that the range of attention values is within the appropriate range. This normalization can be seen as scaling the attention value to prevent the attention value from being too large or too small. By dividing The attention value can be adapted to Q and K vectors of different dimensions, thereby improving the learning and generalization capabilities of the model.

[0069] Calculate the attention weight w for each position i to all unknown i , and normalize the weights, n is the length of the sequence.

[0070]

[0071] Apply the attention weights to the sequence V to obtain the weighted average output, and summarize the final output to obtain the first feature representation corresponding to the Mel spectrum feature encoding, denoted as V′.

[0072] The first feature representation is mapped to the embedding space corresponding to the large natural language model to obtain the second feature representation, denoted as V". Specifically, the speech adapter first downsamples the feature representation output by the speech encoder to reduce the sequence length, and then passes through two perception layers. The ReLU activation function is used in the middle of the perception layer to generate the processed feature representation denoted as V". The large natural language model is used to analyze the second feature representation based on the preset prompt words to obtain the predicted text result corresponding to the speech data.

[0073] After obtaining the predicted text results corresponding to the first dialect sample set, the predicted text results can be used as sample data to train the second initial model. Specifically, the structure of the second initial model can be as follows: the feature extraction module and the speech encoder and speech decoder in the first dialect recognition model are connected in sequence; the speech decoder includes: a position encoding module and multiple decoding blocks connected in sequence; the decoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; wherein the speech decoder is used to decode the first feature representation into the predicted text result corresponding to the input speech data.

[0074] Among them, the structural diagram of the second initial model is as follows Figure 4 As shown in the figure, the second initial model introduces a speech decoder for decoding based on the structure of the first initial model, thereby obtaining the recognition results of the small-parameter automatic speech recognition model. The decoder gradually generates the target text based on the feature representation provided by the encoder and the previously generated output words. The decoding block of the decoder uses the same transformer block as the encoder. The specific structure of the speech decoder is shown in the figure. Figure 5 shown.

[0075] The design of the second initial model places particular emphasis on the structure and role of the speech decoder, which is responsible for converting the encoded first feature representation back into readable text form. The positional encoding module adds a unique identifier to each position in the sequence, helping the model understand the order of the sequence. The multi-head self-attention mechanism of the decoding block ensures that the decoding process can take into account the influence of context, which is crucial for restoring the natural language structure of the dialect. This series of designs, including but not limited to the modules and mechanisms mentioned above, jointly improves the model's decoding capabilities and text generation quality, solves common decoding errors and text coherence issues in dialect recognition, and achieves smoother and more accurate dialect recognition results.

[0076] Specifically, in an optional technical solution provided in the above-mentioned step S2, training the second initial model using the predicted text result set corresponding to the obtained first dialect sample set may include the following steps: using the first dialect recognition model to analyze multiple groups of sample data in the training set respectively to obtain multiple groups of second predicted text results; using the training set to iteratively train the second initial model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second initial model in sequence to obtain multiple groups of third predicted text results output by the second initial model; constructing a second loss function based on the multiple groups of third predicted text results and the text annotation information corresponding to the multiple groups of sample data corresponding to the training batch, and constructing a third loss function based on the multiple groups of third predicted text results and the multiple groups of second predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the second loss function is a cross-entropy loss function, and the type of the third loss function is a relative entropy loss function; adjusting the model parameters of the speech decoder based on the second loss function and the third loss function.

[0077] During the training of the second initial model, the combined use of the second and third loss functions not only quantifies the discrepancy between the model's predictions and the actual annotations, but also assesses the similarity between the model's predictions and those of the first dialect recognition model. This dual-loss strategy, including but not limited to cross-entropy and relative entropy losses, effectively facilitates the model's learning of dialect features and reconstruction of natural language structure, resolving potential prediction bias issues encountered during the initial stages of model training.

[0078] By transferring the learning capabilities of the first dialect recognition model, the performance of the second dialect recognition model composed of a speech encoder and decoder has been significantly improved. However, due to the high cost of obtaining training datasets with text labels, the data during the training process is difficult to meet the dialect recognition needs of various scenarios. Therefore, it is considered to train the second dialect recognition model to obtain the target dialect recognition model.

[0079] In an optional technical solution provided in the above-mentioned step S3, the step of training the second dialect recognition model using the second dialect sample set and the predicted text results of the second dialect sample set by the first dialect recognition model includes: using the first dialect recognition model to analyze multiple groups of sample data in the second dialect sample set respectively to obtain multiple groups of fourth predicted text results; using the second dialect sample set to iteratively train the second dialect recognition model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second dialect recognition model in sequence to obtain multiple groups of fifth predicted text results output by the second dialect recognition model; constructing a fourth loss function based on the multiple groups of fifth predicted text results and the multiple groups of fourth predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the fourth loss function is a relative entropy loss function; and adjusting the model parameters of the speech decoder based on the fourth loss function.

[0080] In this process, the first dialect recognition model is first used for prediction: It analyzes multiple sets of sample data from the second dialect sample set using the first dialect recognition model. Specifically, it analyzes unlabeled dialect speech using the first dialect recognition model to generate multiple sets of fourth predicted text results. The key to this stage lies in the powerful semantic understanding capabilities of the first dialect recognition model, which, due to its integration with a large natural language model, generates relatively accurate text predictions.

[0081] Secondly, the second dialect recognition model is iteratively trained. Specifically, the training can be performed through the following steps: the second dialect sample set can be divided into batches. In each training batch, multiple groups of sample data from the batch are sequentially input into the second dialect recognition model. Through the model's feature extraction, encoding, and decoding modules, multiple groups of fifth predicted text results are output by the second dialect recognition model. Based on the fifth predicted text result of the second dialect recognition model and the fourth predicted text result predicted by the first dialect recognition model for the same batch of sample data, a fourth loss function is constructed. The fourth loss function can be expressed by the following formula:

[0082]

[0083] Where N is the number of tokens in the sequence; M is the size of the vocabulary; y ij is the distribution of target labels; p ij is the probability distribution of the model output. The fourth loss function is set to the relative entropy loss function (i.e., KL divergence), a statistical method that measures the difference between two probability distributions. It is used to determine the similarity between the model's output text probability distribution and the weak supervisory signal (the fourth predicted text result). By minimizing the relative entropy loss function, the model is encouraged to generate predictions that are closer to the weak supervisory signal.

[0084] Finally, according to the calculation result of the fourth loss function, the model parameters of the speech decoder are adjusted through the back propagation algorithm.

[0085] After the above steps, the final target dialect recognition model is obtained. In order to evaluate the performance of the model, a test set can also be introduced to test the target dialect recognition model. The first dialect sample set obtained is divided into a training set and a test set. The training set is introduced to train the model to obtain the target dialect recognition model. The introduction of the test set can test the performance of the target dialect recognition model on unseen data, and can objectively evaluate the generalization ability and recognition accuracy of the model. This technical feature, including but not limited to the use of an independent test set for performance evaluation, ensures the reliability of the model in practical applications, solves the possible overfitting problem of the model, and provides a strong guarantee for the online deployment of the dialect recognition system.

[0086] Through the above steps, in the first stage of training, the first initial model is trained using the annotated first dialect sample set to obtain the first dialect recognition model. This not only enhances the model's learning ability for annotated dialect data, but also significantly improves the model's recognition accuracy by integrating the semantic understanding ability of the large natural language model. In the second stage of training, the first dialect recognition model is used to predict the first dialect sample set to obtain the predicted text results of the first dialect sample set, and the predicted text results are used to train the second initial model to obtain the second dialect recognition model, thereby realizing knowledge transfer between high-performance models and low-resource models. This migration not only maintains strong recognition capabilities, but also significantly reduces the number of model parameters, thereby accelerating the model's reasoning speed; in the third stage of training, the first dialect recognition model is used to predict the second dialect sample set without text annotations, and the generated predicted text results are used as weak supervision signals to train the second dialect recognition model to obtain the target dialect recognition model, so that the model can learn on unlabeled data, effectively expanding the scope of training data and further improving the generalization performance of the model. Through three stages of training, it is possible to ensure that a high-performance dialect recognition model can be trained on less labeled data, and effectively use unlabeled data for subsequent training, which greatly reduces the cost of collecting dialect data.

[0087] Example 2

[0088] According to an embodiment of the present application, a dialect recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0089] Figure 6 FIG. 1 is a flow chart of a dialect recognition method according to an embodiment of the present application, such as Figure 6 As shown, the specific steps include:

[0090] Step S602, obtaining dialect speech data to be recognized;

[0091] Step S604: Analyze the dialect speech data using the target dialect recognition model to obtain target text data corresponding to the dialect speech data, wherein the target dialect recognition model is trained using the above-mentioned dialect recognition model training method.

[0092] The following describes each step of the dialect recognition method in conjunction with a specific implementation process.

[0093] Specifically, suppose a user calls the customer service hotline and begins asking a question in Sichuan dialect. The system first captures this voice data and then analyzes it using the target dialect recognition model. Within the model, a feature extraction module rapidly parses the speech signal, extracting key mel-spectrogram features. The speech encoder converts these features into high-level representations, and the speech adapter further maps them into the embedding space of a large natural language model. Finally, an optimized speech decoder, based on weak supervision signals and knowledge from the dialect recognition model, converts the dialect speech into target text data—the text content of the user's question.

[0094] In the above process, the training method of the dialect recognition model corresponds one-to-one to the method steps in Implementation 1 of this application, and will not be repeated here.

[0095] By utilizing a rigorously trained target dialect recognition model, dialect speech data can be quickly and accurately converted into target text data. The technical solutions provided in the embodiments of this application, including but not limited to the model training and application processes, solve the common problems of slow recognition speed and low accuracy in dialect recognition, and provide efficient and convenient voice services for dialect users. In practical applications, whether in the fields of speech recognition, speech transcription, or intelligent customer service, they can significantly improve user experience and promote the protection and dissemination of dialect culture.

[0096] Example 3

[0097] According to an embodiment of the present application, a training device for a dialect recognition model for implementing the training method of the dialect recognition model in Example 1 is also provided. Figure 7 As shown, the training device of the dialect recognition model includes at least: an acquisition module 71, a first training module 72, a second training module 73 and a third training module 74, wherein:

[0098] An acquisition module 71 acquires a first dialect sample set with text annotations and a second dialect sample set without text annotations;

[0099] A first training module 72 trains a first initial model using a first dialect sample set to obtain a first dialect recognition model, wherein the first initial model includes at least: a feature extraction module, a speech encoder, and a large natural language model;

[0100] A second training module 73 trains a second initial model using the first dialect sample set and the predicted text results of the first dialect recognition model on the first dialect sample set to obtain a second dialect recognition model, wherein the second initial model includes: a feature extraction module and a speech encoder and a speech decoder in the first dialect recognition model;

[0101] The third training module 74 trains the second dialect recognition model using the second dialect sample set and the predicted text results of the first dialect recognition model on the second dialect sample set to obtain a target dialect recognition model.

[0102] The functions of each module of the training device for the dialect recognition model are described below in conjunction with a specific implementation process.

[0103] As an optional implementation, the acquisition module also includes, when specifically acquiring voice data: acquiring multiple groups of voice data obtained by multiple objects in multiple dimensions reading texts on multiple topics in the target dialect, wherein the dimensions include at least one of the following: age, gender, occupation, and place of origin; dividing the multiple groups of voice data into a first voice data set and a second voice data set, wherein the amount of voice data in the first voice data set is less than the amount of voice data in the second voice data set; performing text annotation on the voice data in the first voice data set, and using the annotated first voice data set as the first dialect sample set; and directly using the second voice data set as the second dialect sample set.

[0104] As an optional implementation, after acquiring the dialect sample set, the acquisition module can also perform the following preprocessing operations on the data: before dividing multiple groups of speech data into a first speech data set and a second speech data set, performing a first preprocessing operation on each of the multiple groups of speech data, wherein the first preprocessing operation includes at least one of the following: adjusting the audio sampling rate, deleting speech data with a duration less than a first preset threshold, and segmenting speech data with a duration greater than a second preset threshold; after performing text annotation on the speech data in the first speech data set, performing a second preprocessing operation on the annotated first speech data set, wherein the second preprocessing operation includes at least one of the following: checking the consistency of the annotation text, and aligning the audio with the annotated text.

[0105] As an optional implementation, the first initial model trained by the first training model module includes: a feature extraction module, a speech encoder, a speech adapter and a natural language large model connected in sequence; the speech encoder includes: a starting module, a position encoding module and multiple encoding blocks connected in sequence; the starting module includes: two convolutional layers and a Gaussian error linear unit activation function; the encoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; the speech adapter includes: a downsampling module, a perception layer, a rectified linear unit activation function and a perception layer connected in sequence; wherein, the feature extraction module is used to extract the Mel spectrum features corresponding to the input speech data; the speech encoder is used to encode the Mel spectrum features into a first feature representation; the speech adapter is used to map the first feature representation to the embedding space corresponding to the natural language large model to obtain a second feature representation; the natural language large model is used to analyze the second feature representation based on a preset prompt word to obtain a predicted text result corresponding to the input speech data.

[0106] As an optional implementation, the first dialect sample set includes a training set and a test set. The first training module uses the first dialect sample set to train the first initial model, which may specifically include the following steps: iteratively training the first initial model using the training set, and in each training batch, inputting multiple groups of sample data corresponding to the training batch into the first initial model in sequence to obtain multiple groups of first predicted text results output by the first initial model; constructing a first loss function based on the multiple groups of first predicted text results and the text annotation information corresponding to the multiple groups of sample data, wherein the type of the first loss function is a cross-entropy loss function; and adjusting the model parameters of the feature extraction module, speech encoder, speech adapter and natural language large model based on the first loss function.

[0107] As an optional implementation, the second initial model in the second training module includes: a feature extraction module and a speech encoder and a speech decoder in the first dialect recognition model connected in sequence; the speech decoder includes: a position encoding module and multiple decoding blocks connected in sequence; the decoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; wherein the speech decoder is used to decode the first feature representation into a predicted text result corresponding to the input speech data.

[0108] As an optional implementation, the second training module uses the first dialect sample set and the first dialect recognition model to train the second initial model based on the predicted text results of the first dialect sample set, including: using the first dialect recognition model to analyze multiple groups of sample data in the training set separately to obtain multiple groups of second predicted text results; using the training set to iteratively train the second initial model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second initial model in turn to obtain multiple groups of third predicted text results output by the second initial model; constructing a second loss function based on the text annotation information corresponding to the multiple groups of third predicted text results and the multiple groups of sample data corresponding to the training batch, and constructing a third loss function based on the multiple groups of third predicted text results and the multiple groups of second predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the second loss function is a cross-entropy loss function, and the type of the third loss function is a relative entropy loss function; adjusting the model parameters of the speech decoder based on the second loss function and the third loss function.

[0109] As an optional implementation, the third training module uses the second dialect sample set and the first dialect recognition model to train the second dialect recognition model based on the predicted text results of the second dialect sample set, including: using the first dialect recognition model to analyze multiple groups of sample data in the second dialect sample set separately to obtain multiple groups of fourth predicted text results; using the second dialect sample set to iteratively train the second dialect recognition model, and in each training batch, inputting the multiple groups of sample data corresponding to the training batch into the second dialect recognition model in turn to obtain multiple groups of fifth predicted text results output by the second dialect recognition model; constructing a fourth loss function based on the multiple groups of fifth predicted text results and the multiple groups of fourth predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the fourth loss function is a relative entropy loss function; and adjusting the model parameters of the speech decoder based on the fourth loss function.

[0110] As an optional implementation, the training device for the dialect recognition model may further include a testing module, which may use a test set in the first dialect sample set to test the target dialect recognition model to determine the model performance of the target dialect recognition model.

[0111] It should be noted that each module in the training device of the dialect recognition model in the embodiment of the present application corresponds one-to-one to each implementation step of the training method of the dialect recognition model in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated here.

[0112] Example 4

[0113] According to an embodiment of the present application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the training method of the dialect recognition model in Example 1 or the dialect recognition method in Example 2.

[0114] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the training method of the dialect recognition model in Example 1 or the dialect recognition method in Example 2 by running the computer program.

[0115] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein when the computer program is running, the training method of the dialect recognition model in Example 1 or the dialect recognition method in Example 2 is executed.

[0116] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the training method of the dialect recognition model in Example 1 or the dialect recognition method in Example 2 through the computer program.

[0117] Specifically, when the computer program is running, the following steps of the training method for the dialect recognition model in Example 1 can be executed: obtaining a first dialect sample set with text annotations and a second dialect sample set without text annotations; using the first dialect sample set to train a first initial model to obtain a first dialect recognition model, wherein the first initial model at least includes: a feature extraction module, a speech encoder and a large natural language model; using the first dialect sample set and the first dialect recognition model to predict the text results of the first dialect sample set to train a second initial model to obtain a second dialect recognition model, wherein the second initial model includes: the feature extraction module and speech encoder, speech decoder in the first dialect recognition model; using the second dialect sample set and the first dialect recognition model to predict the text results of the second dialect sample set to train the second dialect recognition model to obtain a target dialect recognition model.

[0118] Specifically, when the computer program is running, the following steps of the dialect recognition method in Example 2 can also be executed: obtaining the dialect speech data to be recognized; analyzing the dialect speech data using the target dialect recognition model to obtain target text data corresponding to the dialect speech data, wherein the target dialect recognition model is trained using the training method of the above-mentioned dialect recognition model.

[0119] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 8FIG1 shows a hardware structure block diagram of an electronic device for implementing a training method for a dialect recognition model. Figure 8 As shown, the electronic device 80 may include one or more (802a, 802b, ..., 802n are used to illustrate) processors 802 (the processor 802 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 804 for storing data, and a transmission device 806 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components than shown, or with Figure 8 Different configurations shown.

[0120] It should be noted that the one or more processors 802 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the electronic device 80. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0121] The memory 804 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the training method of the dialect recognition model in the embodiment of the present application. The processor 802 executes various functional applications and data processing by running the software programs and modules stored in the memory 804, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 804 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 804 may further include a memory remotely located relative to the processor 802, and these remote memories may be connected to the electronic device 80 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The transmission device 806 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the electronic device 80. In one embodiment, the transmission device 806 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 806 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0123] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 80 .

[0124] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0125] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0127] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0128] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0130] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for training a dialect recognition model, characterized in that: include: Obtaining a first dialect sample set with text annotations and a second dialect sample set without text annotations; Training a first initial model using the first dialect sample set to obtain a first dialect recognition model, wherein the first initial model includes at least: a feature extraction module, a speech encoder, and a large natural language model; Training a second initial model using the first dialect sample set and the predicted text results of the first dialect recognition model on the first dialect sample set to obtain a second dialect recognition model, wherein the second initial model includes: a feature extraction module, a speech encoder, and a speech decoder in the first dialect recognition model; The second dialect recognition model is trained using the second dialect sample set and the predicted text result of the first dialect recognition model on the second dialect sample set to obtain a target dialect recognition model.

2. The method according to claim 1, characterized in that Obtaining a first dialect sample set with text annotations and a second dialect sample set without text annotations, including: Acquiring multiple sets of voice data obtained by multiple subjects in multiple dimensions reading aloud texts on multiple topics in a target dialect, wherein the dimensions include at least one of the following: age, gender, occupation, and place of origin; Dividing the plurality of groups of speech data into a first speech data set and a second speech data set, wherein the amount of speech data in the first speech data set is less than the amount of speech data in the second speech data set; Performing text annotation on the speech data in the first speech dataset, and using the annotated first speech dataset as a first dialect sample set; The second speech data set is directly used as the second dialect sample set.

3. The method according to claim 2, characterized in that Before dividing the plurality of groups of speech data into the first speech data set and the second speech data set, the method further includes: performing a first preprocessing operation on each of the plurality of groups of speech data, wherein the first preprocessing operation includes at least one of the following: adjusting the audio sampling rate, deleting speech data having a duration less than a first preset threshold, and segmenting speech data having a duration greater than a second preset threshold; After performing text annotation on the speech data in the first speech data set, the method further includes: performing a second preprocessing operation on the annotated first speech data set, wherein the second preprocessing operation includes at least one of the following: checking the consistency of the annotated text and aligning the audio with the annotated text.

4. The method according to claim 1, wherein The first initial model includes: a feature extraction module, a speech encoder, a speech adapter and a natural language large model connected in sequence; the speech encoder includes: a starting module, a position encoding module and multiple encoding blocks connected in sequence; the starting module includes: two convolutional layers and a Gaussian error linear unit activation function; the encoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module and a residual connection module; the speech adapter includes: a downsampling module, a perception layer, a rectified linear unit activation function and a perception layer connected in sequence; wherein, The feature extraction module is used to extract the Mel spectrum features corresponding to the input speech data; The speech encoder is used to encode the Mel-spectrogram feature into a first feature representation; The speech adapter is configured to map the first feature representation to an embedding space corresponding to the large natural language model to obtain a second feature representation; The natural language large model is used to analyze the second feature representation based on a preset prompt word to obtain a predicted text result corresponding to the input voice data.

5. The method according to claim 4, characterized in that The first dialect sample set includes a training set and a test set, and training the first initial model using the first dialect sample set includes: Iteratively training the first initial model using the training set, and in each training batch, sequentially inputting multiple groups of sample data corresponding to the training batch into the first initial model to obtain multiple groups of first predicted text results output by the first initial model; Constructing a first loss function based on the multiple groups of first predicted text results and the text annotation information corresponding to the multiple groups of sample data, wherein the type of the first loss function is a cross-entropy loss function; Adjust the model parameters of the feature extraction module, the speech encoder, the speech adapter and the natural language model according to the first loss function.

6. The method according to claim 4, characterized in that The second initial model includes: the feature extraction module, speech encoder, and speech decoder of the first dialect recognition model connected in sequence; the speech decoder includes: a position encoding module and multiple decoding blocks connected in sequence; the decoding block includes: a multi-head self-attention module, a feedforward neural network, a layer normalization module, and a residual connection module; wherein, The speech decoder is used to decode the first feature representation into a predicted text result corresponding to the input speech data.

7. The method according to claim 6, characterized in that The first dialect sample set includes a training set and a test set, and the second initial model is trained using the first dialect sample set and a text prediction result of the first dialect recognition model on the first dialect sample set, including: Analyzing the plurality of sample data sets in the training set respectively using the first dialect recognition model to obtain a plurality of second predicted text results; Iteratively training the second initial model using the training set, and in each training batch, sequentially inputting multiple groups of sample data corresponding to the training batch into the second initial model to obtain multiple groups of third predicted text results output by the second initial model; Constructing a second loss function based on the multiple groups of third predicted text results and the text annotation information corresponding to the multiple groups of sample data corresponding to the training batch, and constructing a third loss function based on the multiple groups of third predicted text results and the multiple groups of second predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the second loss function is a cross-entropy loss function, and the type of the third loss function is a relative entropy loss function; Adjust the model parameters of the speech decoder according to the second loss function and the third loss function.

8. The method according to claim 6, characterized in that Training the second dialect recognition model using the second dialect sample set and a text prediction result of the first dialect recognition model on the second dialect sample set includes: Analyzing the plurality of groups of sample data in the second dialect sample set using the first dialect recognition model to obtain a plurality of groups of fourth predicted text results; Iteratively training the second dialect recognition model using the second dialect sample set, wherein in each training batch, multiple groups of sample data corresponding to the training batch are sequentially input into the second dialect recognition model to obtain multiple groups of fifth predicted text results output by the second dialect recognition model; Constructing a fourth loss function based on the multiple groups of fifth predicted text results and the multiple groups of fourth predicted text results corresponding to the multiple groups of sample data corresponding to the training batch, wherein the type of the fourth loss function is a relative entropy loss function; Adjusting model parameters of the speech decoder according to the fourth loss function.

9. The method according to claim 8, characterized in that The first dialect sample set includes a training set and a test set. After obtaining the target dialect recognition model, the method further includes: The target dialect recognition model is tested using the test set to determine the model performance of the target dialect recognition model.

10. A dialect recognition method, characterized in that: include: Obtaining dialect speech data to be recognized; The dialect speech data is analyzed using a target dialect recognition model to obtain target text data corresponding to the dialect speech data, wherein the target dialect recognition model is trained using the dialect recognition model training method described in any one of claims 1 to 9.

11. A training device for a dialect recognition model, characterized in that: include: An acquisition module, configured to acquire a first dialect sample set with text annotations and a second dialect sample set without text annotations; A first training module is configured to train a first initial model using the first dialect sample set to obtain a first dialect recognition model, wherein the first initial model includes at least: a feature extraction module, a speech encoder, and a large natural language model; a second training module, configured to train a second initial model using the first dialect sample set and the predicted text results of the first dialect recognition model on the first dialect sample set, to obtain a second dialect recognition model, wherein the second initial model includes: a feature extraction module, a speech encoder, and a speech decoder in the first dialect recognition model; The third training module is used to train the second dialect recognition model using the second dialect sample set and the predicted text results of the first dialect recognition model on the second dialect sample set to obtain a target dialect recognition model.

12. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, it implements the dialect recognition model training method described in any one of claims 1 to 9 or the dialect recognition method described in claim 10.

13. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the dialect recognition model training method described in any one of claims 1 to 9 or the dialect recognition method described in claim 10 through the computer program.

Images