Complex scene speech recognition optimization method based on transfer learning
By optimizing feature fusion through compact bilinear pooling and transfer learning, and combining long short-term memory networks and domain adversarial neural networks, the problems of high computational cost and low recognition efficiency in speech recognition under complex scenarios are solved, achieving efficient and accurate speech recognition results.
Patent Information
- Application Number
- CN202511716477.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies for speech recognition in complex scenarios suffer from problems such as high computational costs, insufficient feature description capabilities, difficulty in dialect recognition, significant accent interference, and low recognition efficiency.
We employ compact bilinear pooling and tensor sketch projection to optimize feature fusion, combine a transfer learning framework and a long short-term memory network to build a classifier, eliminate dialect segment interference through sliding window segmentation and dynamic decoding, construct a domain adversarial neural network to adapt to accents, and establish a speaker-specific dialect feature library.
It reduces computational costs, improves feature description capabilities, enhances the accuracy and generalization ability of health status classification, and improves the accuracy and efficiency of speech recognition in complex scenarios.
Smart Images
Figure CN121506115A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech recognition, specifically referring to an optimization method for speech recognition in complex scenarios based on transfer learning. Background Technology
[0002] A method for optimizing speech recognition in complex scenarios based on transfer learning refers to a method that optimizes speech recognition in complex scenarios by using transfer learning techniques.
[0003] Among existing approximate solutions, such as CN111986661B's deep neural network speech recognition method based on speech enhancement in complex environments, this solution addresses the technical problem of demanding high-quality speech signals in existing Chinese speech recognition algorithms. This solution first constructs a multi-environment mixed speech dataset and designs a preprocessing pipeline for speech enhancement. The preprocessing pipeline includes boosting high-frequency components, windowing and framing, transforming to the frequency domain using FFT, extracting power for each frequency band using a gamma-pass filter bank, subtracting noise components by calculating low-pass power, adjusting the original spectrum, and transforming back to the time domain using IFFT to generate enhanced clear speech. Then, a convolutional neural network is used to extract local correlations from the enhanced speech, combined with the connectionist temporal classification CTC loss function, to construct a 2-gram language model. This provides a linguistic-level probabilistic constraint for pinyin sequences, achieving the technical effect of existing speech recognition algorithms being sensitive to complex environmental factors such as noise and demanding high speech quality. However, it still suffers from the technical problem of traditional feature fusion methods directly performing outer product calculations to generate extremely high-dimensional features, resulting in high computational costs and insufficient feature description capabilities.
[0004] Secondly, for example, CN112151030B, a method and device for speech recognition in complex scenarios based on multimodality, addresses the technical problem that traditional speech-text recognition technologies based on single channels and single modalities such as audio signals, lip images, and facial electromyography cannot meet the actual needs of engineering applications. It transforms the user's lip image into a unified start signal, simultaneously collects audio, lip video, and facial electromyography signals, and adopts a deep fusion model based on Transformer and cross-modal attention. It uses the Transformer encoder to extract common information separately, and then introduces a cross-modal attention mechanism to allow the features of the three modalities of audio, vision, and electromyography to query, weight, and complement each other. Combined with an n-gram language model, it decodes the fused multimodal information into the final text. This achieves efficient, accurate, and robust speech recognition in complex scenarios such as vocal cord damage, high noise, high closure, and high privacy requirements. However, it still has technical problems such as the small scale of dialect speech annotation data leading to easy overfitting of the model and limited performance, and the difficulty of capturing complex implicit health information in speech due to the reliance on manually designed low-level features in traditional methods.
[0005] In addition, existing technologies still face technical challenges such as difficulty in recognizing pure and mixed segments in cross-dialect mixed speech, significant interference from dialect words, and the susceptibility of dialect-accented Mandarin recognition to the influence of accents. Furthermore, they also suffer from technical problems such as difficulty in determining the dialect affiliation of ambiguous segments due to the individualized dialect habits of speakers, resulting in low recognition efficiency. Summary of the Invention
[0006] To address the above issues and overcome the shortcomings of existing technologies, this invention provides an optimization method for speech recognition in complex scenarios based on transfer learning. Addressing the problem that traditional feature fusion methods directly perform outer product calculations, resulting in extremely high-dimensional features, leading to high computational costs and insufficient feature description capabilities, this solution introduces compact bilinear pooling technology and optimizes based on tensor sketch projection. This fuses features from different layers of different speech recognition networks, avoiding the burden of high-dimensional computation while capturing the multiplicative interactions between features, reducing computational costs and dimensionality overhead, and improving feature description capabilities. Furthermore, addressing the problem that the small scale of dialect speech annotation data leads to model overfitting and limited performance, and that traditional methods rely on manually designed low-level features to fail to capture complex, implicit health status information in speech, this solution employs a transfer learning framework. It reuses the general acoustic principles of pre-trained speech recognition networks and combines a recurrent neural network based on long short-term memory to construct a classifier, reducing reliance on target task annotation data and effectively capturing health-related information in speech. This solution addresses the challenges of complex, implicit information related to health status, improving the accuracy and generalization of health status classification. For example, it tackles the difficulties in recognizing pure and mixed segments in cross-dialect mixed speech, significant interference from dialectal words, and the susceptibility of dialect-accented Mandarin recognition to accent influence. The solution employs sliding window segmentation and dialect category probability determination to differentiate segment types. It then optimizes mixed segment recognition by combining dynamic decoding, semantic verification rearrangement, and a dialect confusion correction table. Furthermore, it constructs a domain adversarial neural network to adapt standard Mandarin to speech with dialectal accents, effectively eliminating dialectal word and accent interference and improving speech recognition accuracy in complex scenarios. Finally, it addresses the issue of difficulty in determining the dialect affiliation of ambiguous segments and low recognition efficiency due to speakers' personalized dialect habits. This solution builds a dedicated dialect feature library for each speaker, prioritizing the matching of mixed segments to this library and dynamically updating it. Combined with the local loading mechanism of the cloud platform's dialect speech recognition model package, it improves the accuracy and efficiency of dialect determination for ambiguous segments and adapts to speakers' personalized dialect habits.
[0007] The technical solution adopted by this invention is as follows: This invention provides a method for optimizing speech recognition in complex scenes based on transfer learning, which includes the following steps:
[0008] Step S1: Abnormal state recognition, used to indirectly infer the speaker's state by utilizing changes in the speech signal;
[0009] Step S2: Cloud-based construction, specifically, constructing dialect speech recognition model packages and storing all types of dialect speech recognition model packages on the cloud platform, and setting up a dialect feature library composed of multiple dialect types for each speaker;
[0010] Step S3: Eliminate dialect word interference to achieve complex speech recognition tasks that mix dialects and Mandarin;
[0011] Step S4: Eliminate dialect accent interference, used to analyze the meaning of Mandarin speech with dialect accents.
[0012] Furthermore, in step S1, the abnormal state identification specifically includes the following steps:
[0013] Step S11: Pre-trained feature extraction, used to obtain multi-level effective features of speech. Specifically, at least two speech recognition networks that have completed pre-training are selected, and the speech to be recognized is sequentially input into each of the speech recognition networks to obtain feature outputs from different layers of different speech recognition networks.
[0014] Step S12: Feature Projection and Integration. Specifically, compact bilinear pooling is used, based on tensor sketch projection, to fuse features from different layers of the different speech recognition networks. This avoids the high-dimensional burden caused by directly calculating the outer product. By randomly initializing the mapping vector and symbol vector, the features are projected into a low-dimensional space, and convolution is calculated using Fourier transform to generate a fused compact feature set. The formula used is as follows: ;
[0015] In the formula, Represents a low-dimensional projection function. The index subscript representing the feature. This represents the features of different layers in the different speech recognition networks. This represents a randomly initialized mapping vector. This represents a randomly initialized symbol vector used to control the symbol weights of the feature projection. Indicates traversing the first... Features The index of an element. express The total dimension Indicates traversal All elements, only retain The middle is equal to The elements are then multiplied by their corresponding symbol vectors. We obtain the cumulative sum after traversal, and finally get... The Middle The value of each element, Indicates the feature size after dimensionality reduction; ;
[0016] In the formula, Represents a compact feature set, This represents the total number of features in different layers of different speech recognition networks. Represents the Fast Fourier Transform. This represents the inverse fast Fourier transform. This represents element-wise Hadamard product multiplication;
[0017] Step S13: Classifier construction and optimization to achieve state awareness. Specifically, a recurrent neural network based on a long short-term memory layer is constructed, and a special state dataset is collected and labeled. The special states include eating, drunkenness, illness, and exercise. The special state dataset consists of voice data of the special states.
[0018] Step S14: State classification, used to achieve special state detection. Specifically, the recurrent neural network is used to perform special state classification training on the special state dataset. The trained recurrent neural network takes the compact feature set as input and outputs the state classification result.
[0019] Furthermore, in step S3, the elimination of dialect segment interference specifically includes the following steps:
[0020] Step S31: Segmentation, specifically, the speech to be recognized is processed by sliding window to obtain speech segments, the dialect category probability distribution of each speech segment is calculated and output by a neural network, a probability threshold is set, and the speech segment type is determined. If the maximum value of the dialect category probability of the speech segment exceeds the probability threshold, it is determined to be a pure segment; otherwise, it is determined to be a mixed segment.
[0021] Step S32: Dynamic decoding, adjusting the decoding strategy according to the type of the speech segment. The specific operation steps are as follows:
[0022] Step S321: If the speech segment type is a pure segment, call the corresponding dialect speech recognition model package for decoding;
[0023] Step S322: If the speech segment type is a mixed segment, the specific operation is as follows:
[0024] Step S3221: Semantic verification and rearrangement, specifically, using weighted fusion decoding technology, the weights are determined based on the probability distributions of the R dialects with the highest probability distributions in the dialect speech recognition model package that has been loaded locally, and a dialect confusion correction table is constructed and used to replace the syllables of the speech segments to optimize the decoding results, resulting in R candidate texts after optimization;
[0025] Step S3222: Construct a context window, specifically by concatenating the R candidate texts of the current mixed segment with the identified context text segments before and after it to form R complete candidate segments;
[0026] Step S3223: Semantic consistency scoring, specifically, using semantic understanding technology to analyze the semantic representation of all candidate segments to obtain a semantic coherence score, and selecting the candidate segment with the highest semantic coherence score as the final recognition result, i.e., the text of the mixed segments;
[0027] Step S3224: Add the dialect corresponding to the candidate segment with the highest semantic coherence score to the dialect feature library of the current speaker, and load the corresponding dialect speech recognition model package on the cloud platform to the local machine.
[0028] Furthermore, in step S4, the elimination of dialect accent interference specifically includes the following steps:
[0029] Step S41: Divide the domains, specifically, take standard Mandarin as the source domain and dialect accents as the target domain, define standard Mandarin speech data as source domain data, and define speech data with dialect accents as target domain data;
[0030] Step S42: Domain adversarial training, used to achieve speech recognition with accents, specifically involves constructing three sub-networks of the adversarial neural network, with the following specific steps:
[0031] Step S421: A convolutional neural network is used as a feature extractor to learn domain-invariant features. The feature extractor extracts domain-invariant features from the two-dimensional Mel spectrum features of the speech to be recognized.
[0032] Step S422: Use a multilayer perceptron as a domain classifier. The domain classifier and the feature extractor form an adversarial relationship. The domain classifier performs binary classification based on the state classification result and the domain-invariant features to determine whether the speech to be recognized belongs to the source domain or the target domain.
[0033] Step S423: Using a bidirectional gated recurrent unit network as a label predictor, the domain-invariant features are analyzed according to the speech time series, and the text corresponding to the speech to be recognized is finally predicted and output. The character error rate and word error rate are used as evaluation indicators to correct the feature extractor.
[0034] The beneficial effects achieved by the present invention using the above solution are as follows:
[0035] (1) To address the technical problem that traditional feature fusion methods directly perform outer product calculations to generate extremely high-dimensional features, resulting in high computational costs and insufficient feature description capabilities, this solution introduces compact bilinear pooling technology and optimizes based on tensor sketch projection to fuse features from different layers of different speech recognition networks. This avoids the burden of high-dimensional computation while capturing the multiplicative interaction relationships between features, reducing computational costs and dimensional overhead, and improving feature description capabilities.
[0036] (2) In view of the technical problems that the small scale of dialect speech annotation data leads to the model being prone to overfitting and limited performance, and that traditional methods rely on manual design of low-level features and are difficult to capture complex implicit health status information in speech, this solution adopts a transfer learning framework, reuses the general acoustic laws of pre-trained speech recognition networks, and combines a recurrent neural network based on long short-term memory network to build a classifier, reducing the dependence on target task annotation data, effectively capturing complex implicit information related to health status in speech, and improving the accuracy and generalization ability of health status classification;
[0037] (3) In response to the technical problems of difficulty in recognizing pure segments and mixed segments in cross-dialect mixed speech, obvious interference from dialect words, and the easy influence of accent on the recognition of Mandarin with dialect accent, this solution distinguishes segment types by sliding window segmentation and dialect category probability determination, optimizes the recognition of mixed segments by combining dynamic decoding, semantic verification and rearrangement, and dialect confusion correction table, and constructs a domain adversarial neural network to achieve the adaptation of the source domain (standard Mandarin) and the target domain (speech with dialect accent), effectively eliminating interference from dialect words and accent, and improving the accuracy of speech recognition in complex scenarios;
[0038] (4) In response to the technical problem that it is difficult to determine the dialect attribution of ambiguous segments and the recognition efficiency is low due to the personalization of the speaker's dialect habits, this solution builds an exclusive dialect feature library for each speaker, prioritizes matching the mixed segments to the library and dynamically supplements and updates it, and combines the local loading mechanism of the dialect speech recognition model package of the cloud platform to improve the accuracy and recognition efficiency of the dialect determination of ambiguous segments and adapt to the speaker's personalized dialect habits. Attached Figure Description
[0039] Figure 1 A flowchart illustrating the complex scene speech recognition optimization method based on transfer learning provided by this invention;
[0040] Figure 2 A flowchart illustrating step S1 provided by the present invention;
[0041] Figure 3 A flowchart illustrating step S3 provided by the present invention;
[0042] Figure 4 This is a flowchart illustrating step S4 of the present invention.
[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0045] Example 1, see Figures 1 to 4 The present invention provides a method for optimizing speech recognition in complex scenes based on transfer learning, which includes the following steps:
[0046] Step S1: Abnormal state recognition, used to indirectly infer the speaker's state by utilizing changes in the speech signal;
[0047] Step S2: Cloud-based construction, specifically, constructing dialect speech recognition model packages and storing all types of dialect speech recognition model packages on the cloud platform, and setting up a dialect feature library composed of multiple dialect types for each speaker;
[0048] Step S3: Eliminate dialect word interference to achieve complex speech recognition tasks that mix dialects and Mandarin;
[0049] Step S4: Eliminate dialect accent interference, used to analyze the meaning of Mandarin speech with dialect accents.
[0050] Example 2, see Figures 1 to 2 This embodiment is based on the above embodiment. In step S1, the abnormal state identification specifically includes the following steps:
[0051] Step S11: Pre-trained feature extraction, used to obtain multi-level effective features of speech. Specifically, at least two speech recognition networks that have completed pre-training are selected, and the speech to be recognized is sequentially input into each of the speech recognition networks to obtain feature outputs from different layers of different speech recognition networks.
[0052] Step S12: Feature Projection and Integration. Specifically, compact bilinear pooling is used, based on tensor sketch projection, to fuse features from different layers of the different speech recognition networks. This avoids the high-dimensional burden caused by directly calculating the outer product. By randomly initializing the mapping vector and symbol vector, the features are projected into a low-dimensional space, and convolution is calculated using Fourier transform to generate a fused compact feature set. The formula used is as follows: ;
[0053] In the formula, Represents a low-dimensional projection function. The index subscript representing the feature. This represents the features of different layers in the different speech recognition networks. This represents a randomly initialized mapping vector. This represents a randomly initialized symbol vector used to control the symbol weights of the feature projection. Indicates traversing the first... Features The index of an element. express The total dimension Indicates traversal All elements, only retain The middle is equal to The elements are then multiplied by their corresponding symbol vectors. We obtain the cumulative sum after traversal, and finally get... The Middle The value of each element, Indicates the feature size after dimensionality reduction; ;
[0054] In the formula, Represents a compact feature set, This represents the total number of features in different layers of different speech recognition networks. Represents the Fast Fourier Transform. This represents the inverse fast Fourier transform. This represents element-wise Hadamard product multiplication;
[0055] Step S13: Classifier construction and optimization to achieve state awareness. Specifically, a recurrent neural network based on a long short-term memory network is constructed, and a special state dataset is collected and labeled. The special states include eating, drunkenness, illness, and exercise. The special state dataset consists of voice data of the special states.
[0056] Step S14: State classification, used to achieve special state detection. Specifically, the recurrent neural network is used to perform special state classification training on the special state dataset. The trained recurrent neural network takes the compact feature set as input and outputs the state classification result.
[0057] Example 3, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S3, the elimination of dialect segment interference specifically includes the following steps:
[0058] Step S31: Segmentation, specifically, the speech to be recognized is processed by sliding window to obtain speech segments, the dialect category probability distribution of each speech segment is calculated and output by a neural network, a probability threshold is set, and the speech segment type is determined. If the maximum value of the dialect category probability of the speech segment exceeds the probability threshold, it is determined to be a pure segment; otherwise, it is determined to be a mixed segment.
[0059] Step S32: Dynamic decoding, adjusting the decoding strategy according to the type of the speech segment.
[0060] Example 4, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S32, the dynamic decoding specifically includes the following steps:
[0061] Step S321: If the speech segment type is a pure segment, call the corresponding dialect speech recognition model package for decoding;
[0062] Step S322: If the speech segment type is a mixed segment, the specific operation is as follows:
[0063] Step S3221: Semantic verification and rearrangement, specifically, using weighted fusion decoding technology, the weights are determined based on the probability distributions of the R dialects with the highest probability distributions in the dialect speech recognition model package that has been loaded locally, and a dialect confusion correction table is constructed and used to replace the syllables of the speech segments to optimize the decoding results, resulting in R candidate texts after optimization;
[0064] Step S3222: Construct a context window, specifically by concatenating the R candidate texts of the current mixed segment with the identified context text segments before and after it to form R complete candidate segments;
[0065] Step S3223: Semantic consistency scoring, specifically, using semantic understanding technology to analyze the semantic representation of all candidate segments to obtain a semantic coherence score, and selecting the candidate segment with the highest semantic coherence score as the final recognition result, i.e., the text of the mixed segments;
[0066] Step S3224: Add the dialect corresponding to the candidate segment with the highest semantic coherence score to the dialect feature library of the current speaker, and load the corresponding dialect speech recognition model package on the cloud platform to the local machine.
[0067] Example 5, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S4, eliminating dialect accent interference specifically includes the following steps:
[0068] Step S41: Divide the domains, specifically, take standard Mandarin as the source domain and dialect accents as the target domain, define standard Mandarin speech data as source domain data, and define speech data with dialect accents as target domain data;
[0069] Step S42: Domain adversarial training, used to achieve speech recognition with accents, specifically involves constructing three sub-networks of the adversarial neural network, with the following specific steps:
[0070] Step S421: A convolutional neural network is used as a feature extractor to learn domain-invariant features. The feature extractor extracts domain-invariant features from the two-dimensional Mel spectrum features of the speech to be recognized.
[0071] Step S422: Use a multilayer perceptron as a domain classifier. The domain classifier and the feature extractor form an adversarial relationship. The domain classifier performs binary classification based on the state classification result and the domain-invariant features to determine whether the speech to be recognized belongs to the source domain or the target domain.
[0072] Step S423: Using a bidirectional gated recurrent unit network as a label predictor, the domain-invariant features are analyzed according to the speech time series, and the text corresponding to the speech to be recognized is finally predicted and output. The character error rate and word error rate are used as evaluation indicators to correct the feature extractor.
[0073] Example 6, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S11, the selection of at least two pre-trained speech recognition networks specifically refers to DeepSpeech2 and the WeChat Neural Network Speech Recognition System (WeNet).
[0074] Example 7, see Figures 1 to 4 This embodiment is based on the above embodiment. In steps S13-S14, the construction and classification training of the recurrent neural network of the long short-term memory layer specifically refers to: configuring the number of long short-term memory network layers and neurons, setting the fully connected layers, activation functions and dropout ratios, using batch normalization and recursive suppression of overfitting, selecting the SGD optimizer with momentum, adjusting the learning rate through the exponential decay mechanism, performing zero-padding on variable-length speech features during training to ensure consistent batch data dimensions, using leave-one-out cross-validation to evaluate the performance of the training set, and using unweighted average recall as the core evaluation index to train the recurrent neural network of the long short-term memory network.
[0075] Example 8, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S31, the probability threshold is 0.7.
[0076] Example 9, see Figures 1 to 4 This embodiment is based on the above embodiment, and in step S3221, R=5.
[0077] Example 10, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S3221, R=2 in this embodiment. The two dialects specifically refer to Mandarin and Minnan dialect. The probability distributions of the two dialects are 0.4 and 0.6.
[0078] Example 11, see Figures 1 to 4 This embodiment is based on the above embodiment, and in the formula involved in step S12, , , , , Only retain The elements, namely and ,calculate That is, the result of the low-dimensional projection function outputting the second element.
[0079] Example 12, see Figures 1 to 4 This embodiment is based on the above embodiment. In step S3221, the dialect confusion correction table specifically refers to: .
[0080] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0082] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A method for optimizing speech recognition in complex scenes based on transfer learning, characterized in that: The method includes the following steps: Step S1: Abnormal state recognition, used to indirectly infer the speaker's state by utilizing changes in the speech signal; Step S2: Cloud-based construction, specifically, constructing dialect speech recognition model packages and storing all types of dialect speech recognition model packages on the cloud platform, and setting up a dialect feature library composed of multiple dialect types for each speaker; Step S3: Eliminate dialect word interference to achieve complex speech recognition tasks that mix dialects and Mandarin; Step S4: Eliminate dialect accent interference, used to analyze the meaning of Mandarin speech with dialect accents.
2. The method for optimizing speech recognition in complex scenes based on transfer learning according to claim 1, characterized in that: In step S1, the abnormal state identification specifically includes the following steps: Step S11: Pre-trained feature extraction, specifically, select at least two pre-trained speech recognition networks, and input the speech to be recognized into each speech recognition network in turn to obtain the feature outputs of different layers of different speech recognition networks; Step S12: Feature projection and integration, specifically, using compact bilinear pooling technology and tensor sketch projection technology, the features of different layers of the different speech recognition networks are fused. By randomly initializing the mapping vector and symbol vector, the features are projected into a low-dimensional space and the convolution is calculated by Fourier transform to generate a fused compact feature set. Step S13: Classifier construction and optimization to achieve state awareness, specifically, constructing a recurrent neural network based on a long short-term memory layer, and collecting and labeling a dataset of special states; Step S14: State classification, used to achieve special state detection. Specifically, the recurrent neural network is used to perform special state classification training on the special state dataset. The trained recurrent neural network takes the compact feature set as input and outputs the state classification result.
3. The method for optimizing speech recognition in complex scenes based on transfer learning according to claim 2, characterized in that: In step S3, the elimination of dialect segment interference specifically includes the following steps: Step S31: Segmentation, specifically, the speech to be recognized is processed by sliding window to obtain speech segments, the dialect category probability distribution of each speech segment is calculated and output by a neural network, a probability threshold is set, and the speech segment type is determined. If the maximum value of the dialect category probability of the speech segment exceeds the probability threshold, it is determined to be a pure segment; otherwise, it is determined to be a mixed segment. Step S32: Dynamic decoding, adjusting the decoding strategy according to the type of the speech segment.
4. The method for optimizing speech recognition in complex scenes based on transfer learning according to claim 3, characterized in that: In step S32, the dynamic decoding specifically includes the following steps: Step S321: If the speech segment type is a pure segment, call the corresponding dialect speech recognition model package for decoding; Step S322: If the speech segment type is a mixed segment, the specific operation is as follows: Step S3221: Semantic verification and rearrangement, specifically, using weighted fusion decoding technology, the weights are determined based on the probability distributions of the R dialects with the highest probability distributions in the dialect speech recognition model package that has been loaded locally, and a dialect confusion correction table is constructed and used to replace the syllables of the speech segments to optimize the decoding results, resulting in R candidate texts after optimization; Step S3222: Construct a context window, specifically by concatenating the R candidate texts of the current mixed segment with the identified context text segments before and after it to form R complete candidate segments; Step S3223: Semantic consistency scoring, specifically, using semantic understanding technology to analyze the semantic representation of all candidate segments to obtain a semantic coherence score, and selecting the candidate segment with the highest semantic coherence score as the final recognition result, i.e., the text of the mixed segments; Step S3224: Add the dialect corresponding to the candidate segment with the highest semantic coherence score to the dialect feature library of the current speaker, and load the corresponding dialect speech recognition model package on the cloud platform to the local machine.
5. The method for optimizing speech recognition in complex scenes based on transfer learning according to claim 4, characterized in that: In step S4, eliminating dialect accent interference specifically includes the following steps: Step S41: Divide the domains, specifically, take standard Mandarin as the source domain and dialect accents as the target domain, define standard Mandarin speech data as source domain data, and define speech data with dialect accents as target domain data; Step S42: Domain adversarial training, used to achieve speech recognition with accents, specifically involves constructing three sub-networks of the adversarial neural network, with the following specific steps: Step S421: A convolutional neural network is used as a feature extractor to learn domain-invariant features. The feature extractor extracts domain-invariant features from the two-dimensional Mel spectrum features of the speech to be recognized. Step S422: Use a multilayer perceptron as a domain classifier. The domain classifier and the feature extractor form an adversarial relationship. The domain classifier performs binary classification based on the state classification result and the domain-invariant features to determine whether the speech to be recognized belongs to the source domain or the target domain. Step S423: Using a bidirectional gated recurrent unit network as a label predictor, the domain-invariant features are analyzed according to the speech time series, and the text corresponding to the speech to be recognized is finally predicted and output. The character error rate and word error rate are used as evaluation indicators to correct the feature extractor.